Having made the move to DX11 and SharpDX I can now use Tesellation Shaders to make my landscape more interesting without having enormous Vertex Buffers. The principles of tessellation shaders are;
- Your vertexes are passed into a traditional VertexShader. This VS simply passes through the vertexes to the next stage, and typically does no transformations. This is becasue the vertexes you submit might not be rendered.
- The vertexes are passed through to a Hull Shader, where you can decide to discard the set of vertexes that are passed in. This stage is the first time you see the new features of using tessellation – instead of operating on a single vertex at a time, you get passed an array of vertexes which form a triangle. Unless you want to do some culling here, you typically return everything you are passed, unaltered.
- The fun starts when your vertexes patches (array of 3 vertexes) are passed into the Domain Shader. Here the DS is also passed a set of barycentric coordinates. The job of the domain shader is to output a new Vertex based on the three passed in, using the barycentric cordinates passed. This is where the majority of the work is done.
In the Domain Shader you end up doing all the work typically done in the Vertex Shader; matrix calculations etc. Traditionally once the vertex leaves the VS is values are interoplated across the triangle and the resultant data structure is passed into the Pixel Shader. Using the DS you are in charge of interpolating the data from the three patch points in any way you see fit. The patch points themselves are never acutally included as vertexes for rendering, and are simply used as control points.
A landcape made of large uniform triangles can be tessellated within the DS to include lots of new sub-triangles, and using noise sampling or other techniques you can make the new vectors have different heights to those that would have been interpolated across the larger triangle. In fact you can change everything – the texture, coordinates etc.
The key to tessellation shaders is to use the extra triangles to generate some useful content and not just to subdivide triangles for the sake of it. The problem is that the introduction of new detail needs to be done in a way which wont affect other rendered object which might not use tesellation. For example a tree will be placed on a landscape based on a sample of a height map. If the landscape introduces tessellation and generates new bumps and other intersting features, that tree placement will be wrong and may sink into the ground or hover above it.
This is nice on a landscape but really comes into its own when painting water surfaces. These have a large appetite for vertexes, and a need to vary the vertexes over a period of time to provide animation. The DS can general a dense mat of sub-triangles and use a height map for the water surface to generate ripples and normals.
Careful selection of the kinds of deformation introduced into the DS is important to getting visually good results. Luckily this doesnt apply to water surfaces since nothing is dependent on their height. The image below uses a linear tessellation algorithm so closer triangles are subdivided more heavily than distant ones. The bump, height and texture samplers are all linked to the game time and so change their sample coordinate to give the illusion of a flowing river.