DXT compression for normalmaps
If you apply DXT compression to a normal map texture, the results are usually pretty awful. But there are some simple things you can do to improve the quality.
First off, notice how in a tangent space normalmap, the Z component always faces roughly outwards (which is why tangent space normalmaps have a bluish tint). And because the texture contains normal vectors, we know the combined length of the X, Y, and Z channels must always be 1.
This means there is no need to explicitly store the Z value. We can throw this away, replacing the blue channel of our texture with zeros, then reconstruct the missing Z value in the pixel shader.
The normal is a unit vector, therefore:
x*x + y*y + z*z = 1
After looking up the values of x and y from the texture, our shader can rearrange this to compute:
z = sqrt(1 - (x*x + y*y))
But how can throwing away and then reconstructing the blue channel improve compression quality?
Remember that DXT compression works by choosing just two base colors for each 4x4 block of the image. If all the colors in a block lie along a line between these two end points, compression quality will be good. The worst artifacts occur when a single block contains colors that are scattered through RGB space, so a single line cannot be fit through them.
By discarding the blue channel of the texture, we collapse the three dimensional RGB colorspace into a two dimensional red/green space. This increases the odds of a single line being a good fit for all the colors in a block, and thus reduces compression artifacts.
For even better quality, you can discard the blue channel as described above, move the red data into the alpha channel (replacing red with zeros), then compress using DXT5. This leaves only a single color dimension for the compression to worry about, which guarantees every block will fit along a single line.
When using only one of the color channels, it is better to choose green rather than red or blue, because DXT uses a 5.6.5 format for the end point colors, and so has slightly better precision in the green channel.
