Shawn Hargreaves Blog

Premultiplied alpha content processor

2009-11-11T16:10:01Z

As mentioned in my previous post, a Content Processor that converts textures into premultiplied alpha format:

using Microsoft.Xna.Framework;
using Microsoft.Xna.Framework.Content.Pipeline;
using Microsoft.Xna.Framework.Content.Pipeline.Graphics;
using Microsoft.Xna.Framework.Content.Pipeline.Processors;

namespace PremultipliedAlpha
{
    [ContentProcessor]
    class PremultipliedAlphaTextureProcessor : TextureProcessor
    {
        public override TextureContent Process(TextureContent input, ContentProcessorContext context)
        {
            input.ConvertBitmapType(typeof(PixelBitmapContent<Color>));

            foreach (MipmapChain mipChain in input.Faces)
            {
                foreach (PixelBitmapContent<Color> bitmap in mipChain)
                {
                    for (int y = 0; y < bitmap.Height; y++)
                    {
                        for (int x = 0; x < bitmap.Width; x++)
                        {
                            Color c = bitmap.GetPixel(x, y);

                            c.R = (byte)(c.R * c.A / 255);
                            c.G = (byte)(c.G * c.A / 255);
                            c.B = (byte)(c.B * c.A / 255);

                            bitmap.SetPixel(x, y, c);
                        }
                    }
                }
            }

            return base.Process(input, context);
        }
    }
}

Premultiplied alpha in XNA Game Studio

2009-11-11T03:24:36Z

It is possible to use premultiplied alpha with XNA Game Studio, but the bad news is we don't do much to help you with it.

Why not?

Yeah. Our bad. In fact one of my biggest regrets about the design of the XNA Framework is that we didn't do more to make this easier!

To use premultiplied alpha, you must do three things:

1 – Set The Blend State

Premultiplied alpha blending is configured like this:

    graphicsDevice.RenderState.AlphaBlendEnable = true;
    graphicsDevice.RenderState.SourceBlend = Blend.One; 
    graphicsDevice.RenderState.DestinationBlend = Blend.InverseSourceAlpha;

Or if you are using SpriteBatch:

    spriteBatch.Begin(SpriteBlendMode.AlphaBlend, SpriteSortMode.Immediate, SaveStateMode.None);
    graphicsDevice.RenderState.SourceBlend = Blend.One;

2 – Premultiply Your Colors

Premultiplied blending only works if all your color values are in premultiplied format. But most paint programs and image file formats do not use premultiplied alpha! So we must find a good place to apply this conversion.

For most games, both 2D and 3D, the flow of color data goes something like this:

Color values are edited in a paint program, then saved into a .bmp or .png file, which is built into .xnb format by the Content Pipeline, then loaded into a Texture2D object.
Other colors may be used to tint the texture. These can be specified as part of the vertex data, or passed as the color parameter to SpriteBatch.Draw.
Still more colors may be computed on the fly (eg. realtime lighting).
All these colors are passed into the shader, which could do all kinds of interesting things with them, but most often just multiplies them together.
The shader produces a final combined color, which is passed to the alpha blending operation.

There are several options for where in this process we choose to convert from conventional to premultiplied format. Two in particular I think can be sensible depending on the situation:

Convert colors at the end of the pixel shader Convert colors in a custom Content Processor

Add "result.rgb *= result.a" right before the end of all your shaders Write a PremultipliedAlphaTextureProcessor, which automatically converts all your textures to premultiplied format while they are being built

Requires custom pixel shaders Does not require custom shaders, so works with SpriteBatch, BasicEffect, etc.

Premultiplication happens after the shader has processed any tint colors, so tints are still specified in conventional, non-premultiplied format All runtime colors, including tint values, are specified in premultiplied format

Alpha blending is done with premultiplied colors, so image composition works properly, but texture filtering happens before the premultiply conversion, so alpha cutouts remain a problem All rendering uses premultiplied colors, so both image composition and alpha cutouts work nicely

Reasonably easy retrofit to existing code, or even just specific parts of that code, as long as you have custom shaders Affects everywhere you do color math, which may be a lot of places, so it can be a pain to retrofit if you don't plan this from the start

3 – Use The Right Math

Any time you do computations on color values, you need to use the right math for the type of colors you are dealing with. Some things to bear in mind:

Keep track of which data is in which format. Mixing premultiplied with conventional colors is a recipe for chaos!
This distinction is only important for colors that have fractional or zero alpha. Opaque colors are the same in both formats, so nothing changes for computations that use RGB color without alpha.
Conventional colors change opacity by leaving RGB alone while decreasing alpha. For instance to draw a half transparent orange sprite we would specify a tint of (255, 128, 0, 128). But with premultiplied colors, we must also decrease the RGB values, so that same tint would be specified as (128, 64, 0, 128).
By far the most common color operation is tinting, which is done by multiplying two colors together (for instance this is how SpriteBatch.Draw combines its texture and color parameter). Color multiplication works the same for premultiplied and conventional colors, as long as both operands are in the same format.
This means that SpriteBatch works as-is with premultiplied data.
BasicEffect diffuse lighting and vertex coloring also use color multiplication, and therefore work correctly with premultiplied data.
The BasicEffect specular lighting and fog computations will not work correctly with premultiplied data. If you want specular lighting or fog while using premultiplied alpha, you will have to write a custom shader.

Premultiplied alpha and image composition

2009-11-07T20:34:28Z

From Wikipedia:

"Alpha compositing is the process of combining an image with a background to create the appearance of partial transparency. It is often useful to render image elements in separate passes, and then combine the resulting multiple 2D images into a single, final image in a process called compositing."

Example

I am making a Pong game. I start by clearing my background to CornflowerBlue (100, 149, 237). I then draw a translucent grey box (128, 128, 128, 128) at the top of the screen, which will form the background of my score display. With conventional alpha blending:

((128, 128, 128) * 0.5) + ((100, 149, 237) * (1 - 0.5)) = (114, 138, 182)

Well and good. But over time, my overlays get more complex, including a map, radar, waypoint indicators, mission objectives, leaderboard rankings, etc. (what can I say, this is a complex Pong game :-) To avoid redrawing so much stuff every frame, wouldn't it be cool if I could draw the overlays just once to a rendertarget, then copy the resulting static image over the top of my animating gameplay?

A Problem

Unfortunately, this doesn't work using conventional alpha blending:

Clear rendertarget to (0, 0, 0, 0)

Blend 50% grey into the rendertarget:

((128, 128, 128, 128) * 0.5) + ((0, 0, 0, 0) * (1 - 0.5)) = (64, 64, 64, 64)

Clear backbuffer to CornflowerBlue (100, 149, 237)

Blend rendertarget over the backbuffer:

((64, 64, 64) * 0.25) + ((100, 149, 237) * (1 - 0.25)) = (91, 127, 194)

Whoah! That's not even remotely the same result as before. The blend operation is happening twice: first when we draw into the rendertarget, then again when we draw the rendertarget into the backbuffer. Our alpha value ends up getting squared, so 0.5 becomes 0.25.

We could fix this by removing one of the blend operations:

Disable blending while drawing to the rendertarget
Or disable blending while drawing the rendertarget over the backbuffer

But these are rarely good solutions, since you most likely need blending in both places (the overlays in my Pong game are built from many layers of alpha blended graphics, and the gameplay scene should be visible behind them).

Some Math

Given a series of drawing operations which blend over the top of each other:

result = a -> b -> c -> d

we would like to be able to group a set of calls from the middle of this sequence, storing their result in a rendertarget, then later replace that set of calls with the contents of the rendertarget:

tmp = b -> c

result = a -> tmp -> d

The problem is that this changes the order of the blend operations. What was:

result = blend(blend(blend(a, b), c), d)

becomes:

result = blend(blend(a, blend(b, c)), d)

Because conventional alpha blending is not associative, changing the order of evaluation changes the result.

The Solution

Premultiplied alpha blending is associative, so any number of blending operations can be grouped and reordered without affecting the final result.

To use premultiplied alpha, we make two changes:

Convert our source colors into premultiplied format, so translucent grey becomes (64, 64, 64, 128) rather than (128, 128, 128, 128)
Change RenderState.SourceBlend from Blend.SourceAlpha to Blend.One

Working through the same example as before:

Clear rendertarget to (0, 0, 0, 0)

Blend 50% grey into the rendertarget:

(64, 64, 64, 128) + ((0, 0, 0, 0) * (1 - 0.5)) = (64, 64, 64, 128)

Clear backbuffer to CornflowerBlue (100, 149, 237)

Blend rendertarget over the backbuffer:

(64, 64, 64) + ((100, 149, 237) * (1 - 0.5)) = (114, 138, 182)

Tada! That's the same result as when we were drawing everything directly to the backbuffer.

Premultiplied alpha r0x0rz.

Historical Aside

Alvy Ray Smith writes about the invention of the alpha channel and image composition. I find it interesting that this was invented in the 1970s, and fully grokked as of the classic Porter-Duff paper in 1984, yet here we are a quarter of a century later, still having a hard time making composition work right because for some reason premultiplied alpha never became as widely understood as it deserves to be.

Premultiplied alpha

2009-11-06T19:39:49Z

Remember when you first figured out Santa Claus wasn't real? The growing doubt, tempered by the fact that all your friends believed in him, and surely they can't ALL be wrong, then the gradual realization that everybody was in fact wrong...

Well, I've got another one for you: the way most people do alpha blending is bogus!

At a fundamental level, alpha has no special meaning at all. Graphics cards manipulate numeric values, but what they do with these values is entirely determined by what shader code you write and which render states you set. The hardware works with Vector4 data types. Because these often represent colors, it is natural to use the first three components for red, green, and blue. That leaves a spare component, called alpha, which can be used for absolutely anything we like.

Some people use alpha to store shininess, or ambient occlusion, or a collision material ID. In the MotoGP particle system I used it to pass a per-particle random number into my vertex shader physics. But most often, alpha is used to represent transparency. It is important to understand that this is just a convention, and there are several different ways it can be done.

Conventional Alpha Blending

The majority of programmers, programs, programming APIs, file formats, etc, define transparency as:

RGB specifies the color of the object
Alpha specifies how solid it is

In math:

blend(source, dest) = (source.rgb * source.a) + (dest.rgb * (1 - source.a))

In code:

    RenderState.SourceBlend = Blend.SourceAlpha; 
    RenderState.DestinationBlend = Blend.InverseSourceAlpha;

In this world, RGB and alpha are independent. You can change one without affecting the other. Even when an object is fully transparent it still has the same RGB as if it was opaque. Thus 100% transparency can be represented by many different color values.

There isn't really a direct physical analogy for this. I guess it's similar to how a magic cloak of invisibility works in a fantasy universe, where I can be wearing the same red sweater as I am right now, and it remains red even though it currently happens to be invisible.

Premultiplied Alpha Blending

Here's a different way to think about transparency:

RGB specifies how much color the object contributes to the scene
Alpha specifies how much it obscures whatever is behind it

In math:

blend(source, dest) = source.rgb + (dest.rgb * (1 - source.a))

In code:

    RenderState.SourceBlend = Blend.One; 
    RenderState.DestinationBlend = Blend.InverseSourceAlpha;

In this world, RGB and alpha are linked. To make an object transparent you must reduce both its RGB (to contribute less color) and also its alpha (to obscure less of whatever is behind it). Fully transparent objects no longer have any RGB color, so there is only one value that represents 100% transparency (RGB and alpha all zero).

This is more like how light behaves in the real world. What is the RGB of my car windscreen? None: it is transparent, so has no color. How about my sunglasses? These have a fractional alpha value (letting some light some through, while blocking some) and also contribute some RGB for that nice rose-tinted glow.

To use premultiplied alpha, in addition to setting the appropriate renderstates, you must also convert your source graphics into premultiplied format. Drawing a non premultiplied color with premultiplied blending will not give sensible results!

To convert a non premultiplied color into premultiplied format:

color.rgb *= color.a

(hence why this is called "premultiplied" format)

Look at the blend equations for conventional vs. premultiplied alpha. If you substitute this color format conversion into the premultiplied blend function, you get the conventional blend function, so either way produces the same end result. The difference is that premultiplied alpha applies the (source.rgb * source.a) computation as a preprocess rather than inside the blending hardware.

I will write more about how to convert graphics into premultiplied format in a later post.

Why premultiplied r0x0rz

Premultiplied alpha is better than conventional blending for several reasons:

It works properly when filtering alpha cutouts (see below)
It works properly when doing image composition (stay tuned for my next post)
It is a superset of both conventional and additive blending. If you set alpha to zero while RGB is non zero, you get an additive blend. This can be handy for particle systems that want to smoothly transition from additive glowing sparks to dark pieces of soot as the particles age.
It plays nice with DXT compression, which only supports transparent pixels with an RGB of zero.

Premultiplied alpha cutouts

Remember how filtering can produce ugly fringes around the edges of alpha cutouts? Not a problem when using premultiplied alpha! Revisiting the example from my previous post:

tree = (0, 255, 0, 255)

border = (0, 0, 0, 0)

background = (0, 0, 255)

filtered = (tree + border) / 2 = (0, 128, 0, 128)

With conventional alpha blending, the result is darker than we wanted:

result = lerp(background, filtered.rgb, filtered.a) = (0, 64, 128)

But premultiplied blending produces the right answer:

result = filtered.rgb + (background * filtered.a) = (0, 128, 128)

This works because texture filtering and premultiplied blending are both linear transforms, and are therefore associative:

blend(filter(a, b), c) == filter(blend(a, c), blend(b, c))

Texture filtering: alpha cutouts

2009-11-02T15:21:26Z

Consider a cutout texture that contains a solid shape surrounded by transparency. Let's say this is a tree, although it could equally well be a cat or an overweight Italian plumber. Our tree is opaque and colored green:

tree = (0, 255, 0, 255)

The surrounding pixels are transparent:

border = (0, 0, 0, 0)

We are drawing over the top of a blue sky:

background = (0, 0, 255)

If we draw without filtering, everything works as expected. Opaque pixels will replace the background, leaving solid green, while the transparent pixels have no effect, leaving solid blue.

Problem

What if the texture needs to be filtered? For instance our tree could be positioned in such a way that some destination pixels are covered half by opaque green and half by the transparent border. First the filtering hardware interpolates between these two parts of the texture:

filtered = (tree + border) / 2 = (0, 128, 0, 128)

Now the alpha blending hardware combines this filtered color with our blue background:

result = lerp(background, filtered.rgb, filtered.a) = (0, 64, 128)

Huh? Halfway between a green tree and blue background should be (0, 128, 128), not (0, 64, 128). The output is darker than we wanted.

It seems logical that if a pixel has zero alpha, its RGB value should be irrelevant, right? Not so when filtering is enabled...

Filtering applies equally to the RGB and alpha channels. When used on the alpha channel of a cutout texture it will produce new fractional alpha values around the edge of the shape, which makes things look nice and antialiased. But filtering also produces new RGB colors, part way in between the RGB of the solid and transparent parts of the texture. Thus the RGB values of supposedly transparent pixels can bleed into our final image.

This most often results in dark borders around alpha cutouts, since the RGB of transparent pixels is often black. Depending on the texture, the bleeding could alternatively be white, pink, etc.

Alternative Explanation For The Mathematically Inclined

What we really wanted was:

filter( blend(tree, background), blend(border, background) )

But instead we got:

blend( filter(tree, border), background )

Because texture filtering and alpha blending are not associative, these do not produce the same result.

Workaround

What if we change our tree texture to have green in the RGB channels of its transparent areas, replacing (0, 0, 0, 0) with (0, 255, 0, 0)? We will still get color bleeding around the edges, but because the transparent RGB now matches the color of the main image, this will not look so ugly.

This workaround can be useful, but is far from perfect:

Not every paint program is able to edit the RGB values of transparent pixels.
Even if your paint program supports this, beware of codecs that may discard these colors when saving out the image, incorrectly figuring that since these pixels are transparent, their RGB values must be irrelevant (for some reason .png codecs seem especially bad at this).
Can't use DXT1 compression, which only supports transparent pixels with an RGB of zero.

If you get bored of having to manually fix-up alpha cutout textures, you could automate it using a custom content processor. This could check your textures as part of the build process, automatically replacing the RGB of transparent pixels with a copy of the RGB from the closest opaque pixel.

Solution

The best way to fix this is by using premultiplied alpha. Stay tuned for my next post...

New Game Studio samples

2009-10-22T17:31:43Z

Hot off the press, this sample provides handy classes for generating cubes, spheres, and teapots (great for debug rendering!) while this one shows how to do various hopefully useful things with skinned character models.

Notice how Skinned Model Extensions reuses code from the Primitives3D sample for drawing the collision spheres? You'd almost think Primitives3D was created because halfway through writing the skinning stuff I realized I had no way to display that data :-)

Texture filtering: mipmapped sprite sheets

2009-10-22T17:23:40Z

Beringela comments on my previous post:

"Why don't spritesheets (with an edge around each sprite) and mipmaps go well together and are there any workarounds other than not using spritesheets or not using mipmaps?"

Let's work through an example, which I shall simplify down to one dimension for clarity. Let's say we have two textures, sized 4x1:

A B C D

and 2x1:

E F

With gutters, we could pack these into a 10x1 sprite sheet:

a A B C D d e E F f

Now let's compute the first mip level:

A (B+C)/2 D E F

Hmm. That does somewhat resemble the original sheet, but it no longer has proper gutters, so filtering will bleed from one sprite to the other.

Real trouble starts with the second mip level:

(A*2 + B + C + D) / 5 (D + E*2 + F*2) / 5

Our second sprite has disappeared entirely! That rightmost pixel includes colors E and F from sprite #2, but also some amount of color D from sprite #1.

And of course the final mip is entirely meaningless:

(A*2 + B + C + D*2 + E*2 + F*2) / 10

There are two ways you can try to make sprite sheets and mipmaps play together:

Wishful thinking: include gutters wider than a single pixel, cross your fingers, and hope it turns out ok. Any time you see color bleeding problems, increase the gutter size until they go away.
The scientific approach: restrict all sprites in a sheet to be the same size, so you can terminate your mip chain at the point where an individual sprite has reached 1x1. Calculate the gutter size so you will still have an entire pixel gutter at this smallest mip level, with corresponding larger gutters for larger mips. Generate mip images separately for each sprite, rather than for the sheet as a whole, and regenerate the gutter with different colors for each mip. This is a considerable pain, and wastes a lot of memory (you end up with significantly more gutter pixels than actual data!) but it does work.

Personally, I would avoid using mipmaps and sprite sheets at the same time. Sprite sheets are never necessary: they're just an optimization to reduce texture switching. When an optimization ends up causing this amount of trouble, you have to wonder if it's really worth it?

Texture filtering: sprite sheets

2009-10-21T18:25:36Z

You can control what happens when texture filtering reads from past the edge of a texture using the SamplerState.AddressU and AddressV properties. But what if you are using a sprite sheet? The sampler address modes know nothing about how your sprites have been packed together, so when you filter a sprite from a sheet, the GPU will happily sample from outside the area you told it to draw. This causes colors from one sprite to show up along the edges of whatever happens to be packed next to it, which is unlikely to be what you wanted!

Possible solutions:

Don't use sprite sheets. This is especially a good idea if you want mipmaps, as it is near impossible to make sprite sheets and mipmaps play nice together.
Don't use filtering. Point sampling will never read past the edge of the region you specify, so it has no problem with sprite sheets. But then, point sampling tends to look rather ugly, and who wants that?
Roll your own filtering in the pixel shader. If you set the GPU to point sampling mode, you could write a shader that implements bilinear filtering by averaging several point samples, and this shader could know about your sprite sheet and thus avoid reading from outside the current sprite. This is likely to be slower than not using sprite sheets at all, though, so not usually a good idea.
Add a gutter region around each sprite. Given this 3x3 source image:

A B C

D E F

G H I

Expand it to 5x5:

a a b c c

a A B C c

d D E F f

g G H I i

g g h i i

Pack the 5x5 version into your sprite sheet, but just draw the inner 3x3 (shown in bold). Now the filtering hardware can happily read past the edge of the specified region, and will pick up sensible color values from the gutter pixels (shown in italic).

Adding gutters by hand soon gets boring, but this is easy to automate. Our Sprite Sheet sample provides a content processor that will pack any number of textures into a sprite sheet, automatically adding the necessary gutters to make filtering work correctly.

Texture filtering: the edge of the world

2009-10-20T23:20:38Z

Long time no post. But don't worry, I'm not dead yet...

One of the more surprising things about texture filtering is how easily it can read past the edge of the texture. Consider a 4x1 texture containing four color values:

A B C D

If we draw this using SpriteBatch with an integer destination location and no scaling or rotation, these values will be copied directly to the screen. But if we draw with scaling, rotation, or a fractional destination location, the output colors will be computed by filtering between the four input values. As long as our source texture coordinates lie in the range 0-1, these filtering computations should never read past the edge of the texture, right?

Right?

(you can probably guess this isn't right, on account of how much of this article is still to read :-)

Consider what happens if we scale up to cover an 8x1 destination region. The output pixels will be:

A (A+B)/2 B (B+C)/2 C (C+D)/2 D ???

What should go in that last cell?

We could just repeat D again, but that seems kind of odd. If we are drawing a tile map with the same sprite repeated many times to cover a grid, it would cause an ugly discontinuity where one tile joins the next.
We could continue the interpolation pattern, wrapping around to fill this last cell with (D+A)/2. This would look good along the joins between tiles, but not so good if we were drawing a character where it could cause colors from one side of the sprite to wrap around and appear on the other!

The right solution depends on what you are drawing, so there is no universally correct behavior. You can choose which you want by setting the SamplerState.AddressU and AddressV properties to TextureAddressMode.Clamp or TextureAddressMode.Wrap.

Many people assume these addressing modes are only important if your texture coordinates go outside the range 0-1, but as we see here, filtering can read past the edge of the texture even when the source texture coordinates lie inside it.

AddressU and AddressV are usually set to the same value, but they do not have to be. For instance the sky.fx shader from our Generated Geometry sample sets AddressU to Wrap and AddressV to Clamp, because the sky is a cylinder which should wrap from left to right, but we do not want stray pixels from the bottom of the ground wrapping around and appearing directly overhead!

Scaling is not the only way to run into these issues. Consider what happens if we slide our sprite half a pixel sideways. Now the four output colors are:

(A+B)/2 (B+C)/2 (C+D)/2 ???

Same dilemma. Should that last cell contain D, or (D+A)/2?

Drawing at fractional positions is especially confusing because we intuitively expect the result to be something like:

A (half alpha) (A+B)/2 (B+C)/2 (C+D)/2 D (half alpha)

But that's not how it works. Exercise for the reader: what would go wrong if it did?

I think the confusing part is that there isn't actually any such thing as drawing at a fractional screen position. Screen pixels can either be written or left alone: you can't change just half a pixel (at least not without alpha blending, which introduces its own set of problems). So when we ask the GPU to draw at a fractional location, it rounds to the nearest integer, then slides the texture in the opposite direction to make the output appear fractionally positioned. This produces coordinate values outside the original range.

"But I want my sprite to be rendered like in your last example, with half alpha on both edges!"

Easy: just make sure the outermost pixels of your texture have zero alpha (give it a single pixel transparent border). Now when you filter using TextureAddressMode.Clamp, the edges will smoothly fade to zero no matter how things are positioned.

"Couldn't I do that automatically using TextureAddressMode.Border?"

Why yes, indeed you could. But border address mode is not supported by all graphics cards.

"What about rotation?"

Hopefully by now it will be obvious that filtered rotation can read from outside the source texture region, just like scaling and fractional positioning.

Texture filtering: anisotropy

2009-09-25T02:00:18Z

Mipmapping allows textures to be scaled down by any amount while maintaining high quality filtering and cache-friendly memory access patterns. But mipmaps don't work so well when different axes are scaled by different amounts:

How do we choose which mip level to use for the elongated cat?

By default the GPU will select mipmaps according to whichever axis has the smallest scale. This is efficient and avoids aliasing artifacts, but can lead to unwanted blurriness along the larger axis.
You can use SamplerState.MipMapLevelOfDetailBias to force selection of a larger mip level, which reduces blurriness at the cost of aliasing artifacts and poor cache efficiency. Basically heading back toward the bad old days when we had no mipmaps at all.

Non uniform scaling is rare in 2D (although my cat has been eating so much lately, he is well on the way to horizontally scaling up his belly exactly like the above image :-) but it happens all the time in 3D:

Any time you apply a perspective transform to a textured triangle which is not exactly facing the camera, non uniform scaling will result.

Anisotropic filtering improves the quality of these situations by recognizing when textures are being scaled differently in different directions, and taking a larger number of samples to compensate.

To enable it:

    device.SamplerStates[0].MinFilter = TextureFilter.Anisotropic;
    device.SamplerStates[0].MagFilter = TextureFilter.Linear;
    device.SamplerStates[0].MipFilter = TextureFilter.Linear;
    device.SamplerStates[0].MaxAnisotropy = <n>;

MaxAnisotropy is typically set to 2 or 4. Some recent cards support as high as 16, so check the caps to see how high yours can go.

How it works:

Compute the aspect ratio between the larger and smaller scaling axes
If this is greater than MaxAnisotropy, clamp to MaxAnisotropy
Choose mip level according to this value (which will usually be whichever axis is larger)
To avoid aliasing, sample the texture multiple times along a line determined by the axis which is being shrunk the most, averaging the results
The number of samples depends on the ratio between the larger and smaller axes
GPUs are smart: if the scaling happens to be uniform, they will only take a single sample even when anisotropic filtering is enabled
Thanks to the aspect ratio clamping, there will never be more than MaxAnisotropy samples per destination pixel

How does it look? Here is our favorite landscape with mipmaps but no anisotropic filtering:

And now with mipmaps plus anisotropic filtering:

Both images are free from noisy aliasing, but the anisotropic version has better texture detail on surfaces which are angled relative to the camera, and is less blurry.

So why doesn't everyone use anisotropic filtering all the time?

Because this quality improvement comes at a price! Sometimes a high price. All these extra samples take time for the texture unit to compute, and cost extra memory bandwidth too.

Or it might be free. If your game is CPU bound, or bottlenecked by a different part of the GPU, anisotropic texture fetches might not actually slow things down at all. So try turning it on and see what happens. If it is too slow with a high MaxAnisotropy setting, try a smaller value to find the right balance between quality and performance.

Back when I was working on MotoGP, we wanted anisotropic filtering for quality reasons, but couldn't afford to use it everywhere. We ended up tweaking MaxAnisotropy differently for every texture, setting it to 4 for billboard adverts where the quality improvement was especially noticeable, to 3 for most of the road surface and grass, and 2 for less important things like trees and crowds. We sometimes even used different settings for each layer of a multitexturing shader, for instance a high anisotropy for the base texture but none at all for the detail texture.

Texture filtering: mipmaps

2009-09-14T23:10:52Z

In my previous article about texture filtering I mentioned that scaling down images using point sampling, or scaling to less than half size using linear filtering, looks bad because some source pixels are discarded.

Why exactly is this a problem?

Try this thought experiment:

We have a 256x256 texture, which is being scaled down by a factor of 1/256, so the entire image covers just a single screen pixel. What color should this pixel be?

If we use point sampling, only one pixel of the 256x256 image will be used, and the others discarded. This can give pretty random results depending on which specific pixel happens to be chosen!

Consider what happens if we tile our texture, so the scaled down image is repeated 100 times. If each repetition lines up exactly with one screen pixel, they will all choose the same source pixel, but if repetitions and screen pixels do not line up (maybe you are scaling down by 1/255 or 1/257 rather than 1/256, or using a 3D perspective projection), each repetition will end up choosing a different random location in the source texture. This is basically the same thing as:

    for each destination pixel:
        destination = sourceTexture.GetPixel(random.GetNext(), random.GetNext());

which is not going to give the best looking results :-)

It would obviously be better if we could average all the pixels from the source texture, rather than having to choose just one of them. But averaging 256x256 color values is too slow to do in realtime.

When you can't afford to compute something on the fly, it is time to precalculate...

A mipmap is a precalculated copy of an image that has been shrunk to a lower resolution using a more accurate, higher quality filtering algorithm than would be possible in realtime on the GPU. Mipmaps are arranged in a chain where each image is half the size of the previous one, for instance:

Original = 256x256
Mip 1 = 128x128
Mip 2 = 64x64
Mip 3 = 32x32
Mip 4 = 16x16
Mip 5 = 8x8
Mip 6 = 4x4
Mip 7 = 2x2
Mip 8 = 1x1

The GPU will automatically choose the appropriate image depending on how much the texture is being shrunk down.

To draw with mipmaps, you need two things:

You must include mipmap data when creating your textures (for instance by setting the Generate Mipmaps content processor parameter)
You must set SamplerState.MipFilter
- TextureFilter.None = do not use mipmaps
- TextureFilter.Point = low quality (but sometimes faster) mipmapping
- TextureFilter.Linear = higher quality mipmapping
- If you draw using BasicEffect, this will automatically set MipFilter = Linear

The difference between Point and Linear mip filtering is how the mipmap level is chosen:

MipFilter = Point
- Chooses the appropriate mip level depending on how much the texture is being shrunk
- If this is a fractional value, rounds up to the next larger mip level
- This means the mip image may still need to be shrunk by a small amount
- But linear filtering is good for shrinking as long as we don't go below half size
- To shrink below half size, we'd just choose a smaller mip level instead
- There can sometimes be visible artifacts when switching between mip levels
MipFilter = Linear
- Samples from both the next larger and next smaller mip levels
- Interpolates between them based on how close the current image scale is to the two mip levels
- Avoids artifacts along boundaries between mip levels
- But we must now sample 8 rather than 4 source pixels for each destination, so this can cost more if your app is texture fetch limited

Let's see this in action. Here is the linear filtered terrain from my previous post:

And now using mipmaps:

Note how the distant hills appear smooth and free from noisy aliasing.

Popular urban myth #1

"Mipmaps take up too much memory"

Actually, mipmaps use little extra memory, thanks to the power of powers:

Mip 1 = 1/4 the original size
Mip 2 = 1/16 the original size
Mip 3 = 1/64 the original size
Mip 4 = 1/256 the original size

If you continue this sequence, you'll see that an entire mipmap chain all the way down to 1x1 takes up just 1/3 more memory than the original texture. So it's a negligible overhead, especially once you get past that first mip.

Popular urban myth #2

"Mipmaps are slower for the GPU to render"

In fact, they are often much faster! If you aren't scaling down a texture, having mipmaps won't cost anything. But when you are scaling down, mipmaps can save crazy amounts of memory bandwidth.

Remember our example of a tiled 256x256 texture, where each repeat is being scaled down to 1x1 (a common situation in things like terrain rendering). Without mipmaps, every destination pixel will sample a radically different location in the source texture, so the GPU must jump around fetching colors from different areas of memory. GPUs typically have very small texture caches, relying on the fact that textures tend to be accessed sequentially, so this access pattern will thrash the cache and can bring even a high end card to its knees. But with mipmaps, the GPU can simply load a small mip level which will easily fit in the cache, and can then render many destination pixels without having to go back to main memory.

So mipmaps are not just for reducing aliasing: they can actually speed up rendering, too.

Shawn's Recommendation™: if you are going to draw a texture in 3D, or planning to scale it down to less than half size, you should always use mipmaps.

Texture filtering

2009-09-08T19:33:12Z

Any time we draw graphics using textures, we must figure out how to map one grid of pixels (the texture) onto a different grid of pixels (the screen). This process is called filtering.

The Identity Transform

The pixel mapping is trivial if:

The texture is not being scaled
The texture is not rotated, or only rotated in 90 degree increments
The texture is being drawn at an integer destination location
The texture may be flipped horizontally or vertically
There are no more complex transforms (shears, perspective, etc)

In this case the source and destination pixel grids line up exactly, so pixel values can be copied directly across.

Filtering In Theory

When the source and destination pixel grids do not line up, we must somehow compute new destination pixel values from the source data. In mathematical terms, you can think of this as a two step process:

Based on the source pixel values, try to guess what the original continuously varying analog image might have been
Quantize this imaginary analog image back down to digital format, sampling at the destination pixel locations based on our chosen definition of what a pixel is

Of course this isn't actually implemented as two stages: both parts are combined in a single computation.

It can sometimes also be useful to think of filtering as a choice between two different operations:

When scaling up an image, we must invent new destination pixel values by trying to guess what should go in between the source pixels
When scaling down an image, we must combine several source pixels to produce a single destination value

There are many ways this can be done. For instance when I resize an image in Paint.NET I can choose between three filtering modes: nearest neighbor, bilinear, or bicubic. Many algorithms are tuned to work with specific types of image, for instance hqx filtering gives great results when scaling up retro game graphics, but would be a terrible choice for digital photographs.

Filtering In Practice

Realtime graphics hardware supports just four filtering modes:

Point (also called nearest neighbor)
Linear (also called bilinear)
Mipmapping (also called trilinear)
Anisotropic

These are specified by three SamplerState properties:

MinFilter controls how textures are minified (scaled down)
MagFilter controls how textures are magnified (scaled up)
MipFilter controls whether mipmapping is used

The attentive reader may wonder why the TextureFilter enum also includes GaussianQuad and PyramidalQuad options. Ignore these: they're a historical legacy. I guess someone once thought they might be useful, but I don't know any graphics cards that actually bother to implement them!

Note that these are just the built-in filtering modes. Any time a shader performs a texture lookup, the hardware automatically applies these filtering computations and returns a filtered value. If you want some other kind of filtering, you can implement it yourself in a shader by doing several texture lookups at slightly different locations to read the values of multiple source pixels, then applying your own filtering computation. This is more expensive than the built-in hardware filtering, but necessary if you want to use more advanced filters such as Gaussian blur or edge detection.

Point Sampling

Point sampling uses a trivially simple filter function. For each destination pixel, it rounds to the closest matching location in the source pixel grid, then takes the value of that single source pixel. Implications:

When magnifying an image, pixels become large and blocky
When shrinking, many source pixel values are discarded, which causes aliasing and shimmery speckled noise
When drawing sprites at fractional destination positions, point sampling has the same effect as if you rounded to an integer location before drawing
When rotating sprites, some pixel locations will round up while others round down, which causes shimmery aliasing

For example, here is a textured terrain using point sampling:

Note how the closest pixels (where the texture is magnified) are large and blocky, while the distant hills (where the texture is minified) look like they're covered in random green and brown noise as opposed to a proper texture! This noise looks especially ugly when the camera moves.

Linear Filtering

For each destination pixel, linear filtering works out the closest matching location in the source pixel grid. It then reads four pixel values from around this location, and interpolates between them. Implications:

When magnifying an image, it produces smooth gradients in between pixels
When shrinking, four source pixels are averaged to produce each destination value, so images can be shrunk to half size before any source pixels are discarded
When rotating or drawing sprites at fractional destination positions, averaging between adjacent pixels creates the illusion of smooth movement with subpixel accuracy, but at the cost of a slight blurring

The same terrain as above, using linear filtering:

Note how the closest pixels are smoothed out, and the mid ground is less noisy than with point sampling, but the distant hills still look pretty nasty. This is because with a maximum of four source samples per destination (two vertical and two horizontal), we can only shrink to half size before we have to start skipping some source pixels. Scaling to less than half size still causes the same aliasing as with point sampling.

We could fix this by averaging more than four source pixels, right? But sampling pixel values is expensive, and graphics cards have to balance the conflicting goals of high visual quality but also high performance. Linear filtering is a classic compromise: good enough to look great when used wisely, but still far from perfect.

Consider, for instance, this 3x3 image, scaled up using point sampling:

If I scale it up on my GPU using linear filtering, I get:

That's not horrible, but also not exactly beautiful. If I scale up the same image in Paint.NET using a bicubic filter, I get a much smoother result:

Paint.NET produces better results than my GPU because it uses a more sophisticated filter algorithm that examines more than four source pixels when computing each destination value.

Coming up: mipmaps, anisotropic filtering, and the challenges presented by alpha cutouts and sprite sheet borders...

What is a pixel?

2009-08-25T21:17:12Z

I wanted to write about premultiplied alpha, but realized I need to explain some things about filtering first. And then I realized that to properly explain filtering, I need to start with an even more fundamental theoretical question: what is a pixel?

From Wikipedia:

"A pixel is the smallest item of information in an image. Pixels are normally arranged in a 2-dimensional grid, and are often represented using dots or squares. Each pixel is a sample of an original image, where more samples typically provide more-accurate representations of the original."

That's not a bad definition, but it's a little vague, especially the "often represented using dots or squares" part. Trouble is, when we try to make this more precise, we realize there are actually several different kinds of pixels in common use!

A pixel is a tiny square. Images are grids of tightly packed pixels, with no gaps or overlap between adjacent squares.
A pixel is a geometric point of zero size. When a continuously varying analog source image is converted to digital format, its color is sampled at each pixel location. Color values from in between these locations are not recorded.
A pixel is a circular dot with soft edges. These dots may or may not be larger than the pixel grid spacing: if they are larger, adjacent pixels will overlap. When digitizing an image, all colors within the pixel region are combined, using a falloff curve that weights their contribution according to distance from the pixel center.

Which version is true?

This matches what we see in programs like Photoshop, where we can zoom into an image and see the pixels scale up to large squares. It also matches what we are used to from early computers and game machines, where low resolutions had large and blocky pixels. But it's not really true in most cases today.
Matches how mathematicians like to view the world, but no actual hardware works like this.
Digital cameras, scanners, monitors, and printers all work this way, or some variant of it. Unfortunately there is no consistency about how much pixels overlap or what falloff curve is used. Some devices vary the concept even further, for instance an LCD uses separate dots with different center locations for the red, green, and blue color components.

Why does this matter?

Much of the time it does not. Many people write code assuming the mathematically elegant zero sized pixels of #2, and this often works fine even though it doesn't truly match their hardware.

One time it does matter is when quantizing an image, by converting analog to digital, or shrinking a digital image to a lower resolution. Any time we do such things, we must choose how the new pixel values will be calculated, which requires a specific definition of exactly what a pixel is.

Another time this matters is when scaling a digital image up to a higher resolution. In some ways this is impossible. We are trying to add more pixels, but these pixel values aren't stored in the source image, and we can't recreate missing data out of thin air! But this is too common and useful an operation to just throw our hands in the air and give up because of a minor technical issue like the problem being fundamentally unsolvable :-) So, we guess. The better we guess, the better the resulting scaled image will look. The more we know about how the source image was created, the more accurate a guess we can make, but in practice we usually know very little, as images don't typically include data describing what pixel falloff curves they were created from.

So we muddle through with no real standard for what a pixel actually is, and everybody just makes the best guesses they can. Or more often, ignores the question entirely...

Combining shaders

2009-08-18T01:37:00Z

"So, I have this shader that does normalmapping, and this other shader that does skinned animation. How can I use them both to render an animated normalmapped character?"

Welcome, my friend, to one of the fundamental unsolved problems of graphics programming...

It seems a reasonable expectation that if you have two effects which work separately, it should be easy to combine them, no? After all, that's exactly what happens in a program like Photoshop. I can add a drop shadow, then a blur, then apply a contrast adjustment, and everything Just Works™. Why not the same for GPU shaders?

To understand the problem, we need to understand the underlying programming model. An app like Photoshop has a very simple model:

Data is a bitmap image (a 2D array of color values)
Filters are functions which take in one bitmap and output a different bitmap
Any number of filters can be stacked by passing the output from one filter as the input of another

This programming model is conceptually simple, mostly because it only deals with one data type. It is trivial to chain filters together when their input and output are the same type.

The GPU shader programming model is more flexible, and thus more complex. A slightly simplified diagram:

The blue boxes represent input data. Red are customizable processing operations, and yellow is the final output. Let's walk through what happens each time you draw something, starting at the left of the diagram and working to the right:

The GPU reads vertex data and indices
These values are combined to form one or more triangles
Your vertex shader program runs once for each vertex
- Inputs: vertex data + effect parameter values
- Outputs: position + colors + texture coordinate values
The GPU takes the position values output by the vertex shader, and works out what screen pixels are covered by each triangle
It interpolates color and texture coordinate values over the surface of the triangle, generating smooth gradients between the three corner vertices
Your pixel shader program runs once for each pixel covered by the triangle
- Inputs: colors and texture coordinates (produced by interpolation of the vertex shader output values) + effect parameter values + textures
- Outputs: color
The color produced by the pixel shader is combined with the previous color at that location in the rendertarget, by applying a user specified blend function
The resulting color is stored into the output rendertarget

Yikes! Note that although the final output is a 2D bitmap (same as for a Photoshop filter), the input is a combination of vertex data, indices, effect parameters, and textures. The input and output types are not the same, which means there is no generalized way to pass the output from one shader as the input of another, and thus no way to automatically combine multiple shaders.

In fact the only universal way to combine two shaders is to understand how they work individually, then write a new shader that contains all the functionality you are interested in. This is the price we pay for flexibility. Because shader programs can do so many different things in so many different ways, the right way to merge them is different for every pair of shaders. For instance to use animation alongside normalmapping, it is necessary to animate the tangent vectors used by the normalmap computation, which requires changes to both the animation and normalmapping shader code.

However, there are specific cases in which Photoshop style layering is possible, if you impose extra constraints on the programming model by restricting all your shaders to work in a similar way:

When processing rectangular 2D regions (most often fullscreen), you can feed the output rendertarget from one drawing operation as an input texture to another. This is only possible when all the interesting work is done in the pixel shader, with SpriteBatch often used to provide the vertex data, indices, and vertex shader. Restricting the geometry pipeline to 2D quads enables Photoshop style composition using a separate shader per layer. Check out this sample for an example.
If you have several pixel shaders which produce different color values for the same model, and the desired way to combine these colors is a simple arithmetic operation (typically addition, multiplication, or interpolation), you can draw several times directly to the backbuffer, one pass per shader, and use alpha blending to combine the shader outputs. The first pass will typically use opaque blending with standard depth buffer states, while subsequent passes use some other blend function, depth compare set to equal, and depth writes disabled. This can be a good solution for scenes with large numbers of lights, where each pass adds the contribution of a single light to the color already in the backbuffer. See this sample.

In a previous job I designed a system for automatically combining shader fragments in more flexible ways than are possible using rendertargets or alpha blending. This worked well, but was complex both to implement and when adding new shader fragments, and there were still many things it could not handle.

So there you have it. Combining shaders turns out to be harder than you might expect, and is usually a manual process. But on the plus side, I guess this makes good job security for us shader programmers :-)

Why does my content rebuild every time?

2009-08-14T20:03:10Z

The XNA Framework Content Pipeline tries to use incremental rebuilds wherever possible. This means if you add a hundred content files, build your project, change just one of those files, then build again, only that specific file needs to be rebuilt.

In a perfect world this would work 100% of the time, but in reality we sometimes rebuild more content than is strictly necessary. Most often this happens if you make a change to a custom processor assembly. The pipeline does not keep track of which assemblies were involved in building which pieces of content, so if any custom assembly changes, all content will be rebuilt. Unwanted rebuilds may also occur when changing project settings, if you have circular project references, or if the Content Pipeline cache file is somehow getting deleted.

If you think your content is rebuilding more often than it should, you can find out why by turning up the MSBuild output verbosity. If you rebuild-all with the Skinned Model sample, the Visual Studio output pane will normally show something like:

    Building dude.fbx -> SkinningSample\bin\x86\Debug\Content\dude.xnb 
    Building head.tga -> SkinningSample\bin\x86\Debug\Content\head_0.xnb 
    Building SkinnedModel.fx -> SkinningSample\bin\x86\Debug\Content\SkinnedModel_0.xnb 
    Building jacket.tga -> SkinningSample\bin\x86\Debug\Content\jacket_0.xnb 
    Building pants.tga -> SkinningSample\bin\x86\Debug\Content\pants_0.xnb 
    Building upBodyC.tga -> SkinningSample\bin\x86\Debug\Content\upBodyC_0.xnb

Open the Tools / Options menu in Visual Studio. If you are using C# Express, make sure the Show all settings box is checked. Expand Projects and Solutions, select the Build and Run node, and change the MSBuild project build output verbosity setting from Minimal to Normal.

Now when we rebuild-all, the output pane shows:

    Target CoreCompile: 
        Building dude.fbx -> SkinningSample\bin\x86\Debug\Content\dude.xnb 
        Rebuilding because asset is new 
        Importing dude.fbx with Microsoft.Xna.Framework.Content.Pipeline.FbxImporter 
        Serializing obj\x86\Debug\dude_0.xml 
        Processing dude.fbx with SkinnedModelPipeline.SkinnedModelProcessor 
        Compiling SkinningSample\bin\x86\Debug\Content\dude.xnb 
        (etc)

If we make a change to SkinningData.cs, which changes the custom pipeline assembly, the next build will show:

    Target CoreCompile: 
        Rebuilding all content because pipeline assembly SkinnedModelPipeline.dll has changed 
        Building dude.fbx -> SkinningSample\bin\x86\Debug\Content\dude.xnb 
        Importing dude.fbx with Microsoft.Xna.Framework.Content.Pipeline.FbxImporter 
        Serializing obj\x86\Debug\dude_0.xml 
        Processing dude.fbx with SkinnedModelPipeline.SkinnedModelProcessor 
        Compiling SkinningSample\bin\x86\Debug\Content\dude.xnb 
        (etc)

If you find that content keeps rebuilding because assemblies are changing, consider splitting these assembles to separate game logic (which tends to change often) from content data types (which change less often). If your content project can avoid referencing your game logic assembly, it will no longer need to rebuild each time that code changes.

Convert colors at the end of the pixel shader	Convert colors in a custom Content Processor
Add "result.rgb = result.a*" right before the end of all your shaders	Write a PremultipliedAlphaTextureProcessor, which automatically converts all your textures to premultiplied format while they are being built
Requires custom pixel shaders	Does not require custom shaders, so works with SpriteBatch, BasicEffect, etc.
Premultiplication happens after the shader has processed any tint colors, so tints are still specified in conventional, non-premultiplied format	All runtime colors, including tint values, are specified in premultiplied format
Alpha blending is done with premultiplied colors, so image composition works properly, but texture filtering happens before the premultiply conversion, so alpha cutouts remain a problem	All rendering uses premultiplied colors, so both image composition and alpha cutouts work nicely
Reasonably easy retrofit to existing code, or even just specific parts of that code, as long as you have custom shaders	Affects everywhere you do color math, which may be a lot of places, so it can be a pain to retrofit if you don't plan this from the start