Shawn Hargreaves Blog
Alpha cutouts (where you use the alpha channel of a texture to determine the shape of a sprite or billboard) are the cause of many an aliasing problem. There are basically two ways to do alpha cutouts, both tragically flawed.
As long as you start with high quality source textures, and draw them using bilinear filtering and mipmaps, alpha blended cutout textures can be pleasingly free of aliasing artifacts.
But alpha blending does not play nice with the depth buffer, and it can be a PITA to manually depth sort every piece of alpha cutout geometry!
In an attempt to make the hardware depth buffer work with alpha cutouts, many developers switch from full alpha blending to alpha testing (using the XNA AlphaTestEffect, or the clip() HLSL pixel shader intrinsic). This does indeed avoid the complexity of manually sorting alpha blended objects, but at the cost of aliasing.
The problem is that alpha testing only provides a boolean on/off decision, with no ability to draw fractional alpha values.
In mathematical terms, when we quantize a smoothly varying signal such as the alpha channel of a texture into a binary on/off decision, we are increasing the amount of high frequency data in our signal, which moves the Nyquist threshold in the wrong direction and hence increases aliasing.
Or I can avoid the math and try a common sense explanation:
We would normally antialias our source textures by including fractional values around shape borders, but alpha testing does not support fractional alpha values, so this is no longer possible.
We would normally avoid aliasing when resampling textures via bilinear filtering and mipmapping. But even if the original texture uses only on/off boolean alpha, when we filter between multiple texels, or scale it down to create 1/2, 1/4, and 1/8 size mip levels, these will inevitably end up with fractional alpha values. To use alpha testing, we must round all such values to either fully transparent or fully opaque, which entirely defeats the point of filtering and mipmapping.
We would normally avoid aliasing along the edges of triangle geometry by multisampling, but remember this only runs the pixel shader one per final output pixel. Since each group of multisamples gets a single alpha value from this single shader invocation, each multisample inevitably also has the same alpha test result, so multisampling is useless when it comes to smoothing our alpha cutout silhouette.
Yowser! Our entire arsenal of antialiasing techniques just vanished in a puff of smoke...
The only common antialiasing technique that works with alpha tested cutouts is supersampling. Which is great if you can afford it, but most games cannot.
Example time. Consider this image, taken from the XNA billboard sample, which uses full alpha blending with a two pass depth sorting technique:
If I change the code to use alpha testing instead of blending, standard depth buffer sorting works with no more need for that expensive two pass technique, but at the cost of nasty aliasing on the thinner parts of the grass fronds:
How to fix this?
You don't. There are really just three choices for alpha cutouts:
In MotoGP we used a mixture of #2 and #3, and let the artists choose which was the lesser evil for each alpha cutout texture.
4. Pre sort chunks of geometry for a given number of directions(eg multiple index ranges) then render the chunks sorted and choose the direction which is most closly aligned with the view direction. A similar technique can be used for hair(or other approx spherical objects) by pre sorting from centre outwards.
This avoids most of the articacts and alpha testing can be used with alpha blending to try and reduce the worst of the depth buffer/fill rate problems.
5. Use a two-pass system, where you alpha-test on the first Z-only pass and alpha-blend on the second pass. The PS2 could do this for free in hardware using just one pass, so most PS2 games tended to use it. If you're using a Z-prepass, which a lot of games do, you can get the same effect there for no cost.
You could use distance fields to reduce the jagged edges when using alpha testing:
Isn't alpha-to-coverage made to allow some level of anti-aliasing for alpha-tested surfaces? I've never used it and I don't know if it's even doable in XNA, but I know it's a thing.
Alpha to coverage can be a valuable technique, but it only enables as many levels of alpha as you have multisamples, so is still a long way short of full blending. It's also a D3D10 feature, not available at all in D3D9.
"It's also a D3D10 feature, not available at all in D3D9." According to Microsoft:-) In practice it is mostly available with vendor "hacks" XNA could have easily exposed it...
I use Kayamon's "5. Use a two-pass system" for trees (even willows) and it works well.
"4. Pre sort chunks of geometry for a given number of directions(eg multiple index ranges) then render the chunks sorted and choose the direction which is most closly aligned with the view direction."
This is the technique we used for the alpha blended Grass rendering on the game "Pure".
We still had the distance sort the chunks of course, but there were far less chunks than blades of grass.
"It's also a D3D10 feature, not available at all in D3D9." Alpha-to-coverage is not a standard D3D9 feature, but many DX9 cards have it. It's also a feature that the XBOX supports -- so I wish that XNA exposed it. Alpha-to-coverage is also a much faster and easier solution than trying to alpha blend the grass blades when you are using a light pre-pass deferred approach (which I currently am). This is currently a tough problem to solve in that case, when developing for the XBOX 360.
Actually you probably can solve this using transparent edges (which will be unlit, or drawn with forward rendering) with a light pre-pass pipeline. However alpha-to-coverage is still likely the best compromise between speed and quality on the XBOX 360 in many situations, if you have many blades of grass.