Greg Schechter's Blog

Effect Library posted

Greg Schechter — Wed, 22 Oct 2008 23:13:50 GMT

Troy Jefferson interned this summer with the WPF team and gathered a bunch of Effects that had been written by members of the WPF team. These are now available for downloading on CodePlex. At that same site, you can also browse to a video of these effects in action.

The single input effects in this library include:

BandedSwirl
Bloom
BrightExtract
ColorKeyAlpha
ColorTone
ContrastAdjust
DirectionalBlur
Embossed
Gloom
GrowablePoissonDiskEffect
InvertColor
LightStreak
Magnify
Monochrome
Pinch
Pixelate
Ripple SharpenSmoothMagnify
Swirl
Tone
Toon
ZoomBlur

There are also a set of two-input effects package up as "transitions" for doing interesting visual transitions between elements. These include:

BandedSwirl
Blinds
Blood
CircleReveal
CircleStretch
CircularBlur
CloudReveral
Cloudy
Crumble
Dissolve
DropFade
Fade
LeastBright
LineReveal
MostBright
PixelateIn
PixelateOut
Pixelate
RadialBlur
RadialWiggle
RandomCircleReveal
Ripple
Rotate
Saturate
Shrink
SlideIn
SmoothSwirl
Swirl
Water
Wave.

Thanks, Troy!

A More Useful Multi-Input Effect

Greg Schechter — Sun, 28 Sep 2008 02:00:00 GMT

In my last post, I introduced multi-input effects, where you can send in arbitrary WPF brushes that are interpreted as samplers in the shader. The example I did just showed simple image combination via linear combination of pixel values -- illustrative, but not very useful.

In this post, I'm going to demonstrate a more useful application of multi-input effects. One of the things that TV networks and local stations do to identify themselves as programs are airing is display a little image that looks like a bit like a watermark in the lower portion of the screen. This is subtle enough to not be distracting, but also clearly identifies the source. This sort of subtle branding can be quite powerful.

Some do this watermarking just via a translucent icon (trivial to do in WPF without Effects). Others use the icon in a more sophisticated way. We're going to show a more sophisticated way here.

Here is a screenshot of the sample app that's included in this post. Note the rose logo in the lower right hand corner of the image.

If you look closely at this close up, you may be able to see that the rose icon is really shifting the underlying pixels.

This is known as "displacement mapping", and makes for a subtle yet distinctive effect as opposed to simply alpha blending the icon on top. Note that here we use it on top of a static image. However, this can be used on live video, on UI, atop DirectX content, etc.

Using the LogoDisplacer effect

The attached project includes a "LogoDisplacer" Effect. The above is modeled as so in XAML:

<Image Source="desert.jpg" Stretch="None" Loaded="Image_Loaded">
    <Image.Effect>
        <lds:LogoDisplacer Logo="{StaticResource logoImage}" 
               Displacement="20" AdditionalLogoOpacity="0.4" />
    </Image.Effect>
</Image>

(For simplicitly, I've elided the databinding syntax that hooks the Displacement and AdditionalLogoOpacity properties to sliders.) The "Logo" property itself is the secondary sampler input, and comes in as a reference to an ImageBrush resource in this case, defined as:

<Window.Resources>
    <ImageBrush x:Key="logoImage" ImageSource="LogoImage.png" />
</Window.Resources>

Here, LogoImage.png looks like this, but yours could be anything:

This is a snapshot from Photoshop where the checkerboard represents fully transparent background pixels. (I created this in Photoshop just by using a character from the Zapf Dingbats font, and then having the Layer Blending options set to Bevel & Emboss with Contouring.)

The LogoDisplacer shader effect uses this logo image to displace the content under it. It does so by, for each pixel, seeing how far the red channel is from 0.5, and offsetting in x the lookup of the pixel from the input image by that fraction of the "Displacement" property (in pixels). It will do the same with the green channel and the y offset. Thus if the image is grayscale (as in this example), red and green are the same, and the x and y displacement will be the same. This is all modulated by the alpha channel of the logo, so where there are transparent pixels in the logo, no displacement occurs.

After this potentially displaced lookup into the original image happens, the corresponding pixel from the logo image itself is blended in with "AdditionalLogoOpacity" alpha, to make the logo more apparent. AdditionalLogoOpacity should be small enough to not overshadow the displacement, but big enough to allow the overlay to be discernible.

When I apply the LogoDisplacer effect as constructed above, my result looks like this:

Not quite the result we were hoping for, since we want the rose just in the lower right hand corner, and at it's original pixel size.

The reason the rose takes up the whole image was discussed in the last post. The sampler inputs are always sized to match the render size of the UIElement that the effect is being applied to. So even though our logo image is 100x100, it gets scaled up to the size of the desert image (1024x768), and that's why it's pixellated.

We need to do something that prevents this resize/pixellation. Since we know that whatever input we provide will be scaled to the size of the UIElement, we can't do a predetermined scaling -- that is, we can't create a 1024x768 image and just put the rose in the lower right hand corner, because if I apply to a UIElement of a different size, the scaling will be wrong again.

Using the Viewbox property

What we do instead is rely on the Viewbox property of ImageBrush. By default, Viewbox is the unit-rect ([0,0,1,1]) that the image fills. If we make the Viewbox larger, we can in effect position the unit-rectangle (and thus the logo) anywhere within that larger viewbox. If I make the Viewbox be [0,0,2,2], then the logo will fill the upper left quadrant. If I make it (-1,-1,2,2), then it will fill the lower right quadrant.

A little math then lets us figure out how to size the Viewbox so that it fits in the lower right 100x100 pixels (or whatever the logo's pixel size is), based on the actual width and height of the UIElement. The following is the event handler invoked when the desert image above completes loading:

private void Image_Loaded(object sender, RoutedEventArgs e)
{
    ImageBrush logo = (ImageBrush)this.Resources["logoImage"];

    // Create a viewbox so that the original logo image is in the 
    // lower right hand corner at its original pixel size (relative 
    // to the image element).
    Image thisImage = (Image)sender;
    BitmapSource bms = logo.ImageSource as BitmapSource;
    double scaleX = (double)thisImage.ActualWidth / bms.PixelWidth;
    double scaleY = (double)thisImage.ActualHeight / bms.PixelHeight;

    // Choose viewbox to move it into the lower right.
    logo.Viewbox = new Rect(1.0 - scaleX, 1.0 - scaleY, scaleX, scaleY);
}

Now, when we run the app, we get the first image shown in this post, with the logo at its correct pixel size in the lower right hand corner of the image.

Note also that since the logo is just an ImageBrush, it can be made dynamic. The attached solution has the logo slowly rotating about its center.

A note about performance

The performance impact of a sample like this can be deceiving, since we're only showing the lower right hand corner of the element being affected. However, remember that all pixels in the element are being examined..., but only those within the logo are being modified. Thus, any modifications to the element itself will cause the entire effect to be re-run over the element. So, while the GPU is incredibly fast, there are still limits, and one should be aware of the implications.

Writing the LogoDisplacer effect

Let's move on to how the LogoDisplacer effect gets written.

The C# file

The C# file, LogoDisplacer.cs, doesn't really have anything very interesting. It just sets up the DependencyProperties that are bound to shader registers, as we've seen with other shaders. In this case, there's Input (Sampler Register 0), Logo (Sampler Register 1), Displacement (Constant Register 0), and AdditionalLogoOpacity (Constant Register 1). We also include this mellifluous property setting, which we discuss in the Appendix of this posting:

this.DdxUvDdyUvRegisterIndex = 6;

The FX file

This is where things get more interesting. Here's the entire .fx file:

//-----------------------------------------------------------------------------------------
// Shader constant register mappings (scalars - float, double, Point, Color, Point3D, etc.)
//-----------------------------------------------------------------------------------------

float  displacement : register(C0);
float  additionalLogoOpacity : register(C1);
float4 ddxDdy : register(C6);

//--------------------------------------------------------------------------------------
// Sampler Inputs (Brushes, including ImplicitInput)
//--------------------------------------------------------------------------------------

sampler2D implicitInputSampler : register(S0);
sampler2D logoSampler : register(S1);


//--------------------------------------------------------------------------------------
// Pixel Shader
//--------------------------------------------------------------------------------------

float4 main(float2 uv : TEXCOORD) : COLOR
{
   // Pull the sample from the logo.
   float4 logoSample = tex2D(logoSampler, uv);
   
   // See how far away the the red and green channels are from "gray".  Use this
   // value to shift.
   float2 fracShift = float2(0.5,0.5) - logoSample.rg;
   
   // But first modulate the shift by the alpha in the logo.  Most of the logo is
   // transparent, so this will zero out fracShift anywhere other than the logo.
   fracShift *= logoSample.a;
   
   // Calculate coordinate to sample main image at, by displacing by the logo's
   // "distance from gray".  ddxDdy is used to ensure that "displacement" is treated
   // in pixel units (it maps from pixel units to [0,1] units).
   float2 displacedCoord = uv + fracShift * displacement 
                              * float2(length(ddxDdy.xy), length(ddxDdy.zw));
    
   // Now get the main image's color at that displaced coordinate.
   float4 color = tex2D(implicitInputSampler, displacedCoord);
    
   // And mix in that color with a portion of the logo, to make the logo more clearly
   // visible.  Modulate it with "additionalLogoOpacity".
   float4 finalColor = color + logoSample * logoSample.a * additionalLogoOpacity;
   
   return finalColor;
}

The first sections (up until "main") are just setting up the register associations that we showed in the C# file. Then, the main() entry point:

Pulls out the color value from the logo and sees how far red and green are from 0.5 (modulating by the alpha value so it will be zero for transparent pixels).
Figures out where to sample the main input image based upon the distances from 0.5, multiplied by the incoming "displacement" value that was in a DependencyProperty. (The DdxDdy thing is mentioned here... we'll get there soon...)
Now samples the main input sampler at this potentially displaced coordinate, arriving at a color.
Finally adds in some portion of the logo pixel itself on the displaced color from the main input sampler, based upon the value of the AdditionalLogoOpacity DependencyProperty.

This results in the subtle but visible displacement effect we see in the above images. A running solution is attached to this post (note that it requires the ShaderBuildTask from here.)

That's all for now!

Appendix: Advanced - a little about DdxUvDdyUvRegisterIndex

(Note that this is a fairly complicated topic that I'm not going to do full justice to in this post.)

Recall that the PixelShader system samples textures within a unit square, [0,0,1,1]. However, our Displacement value is provided in pixels (which makes sense, since we typically want to display the logo at its original pixel size). We need a way of converting from these pixel values into the unit square. That's where DdxUvDdyUvRegisterIndex comes in. You set it in the C# file to a constant register index, then in your shader, you access that register constant. This constant is a float4, representing the increment in unit coordinates that the GPU needs to move when sampling to get to the next pixel to process. The components are as follows:

ddxDdy.x - the unit coord distance in x to move one horizontal pixel in the destination sampler.
ddxDdy.y - the unit coord distance in y to move one horizontal pixel in the destination sampler.
ddxDdy.z - the unit coord distance in x to move one vertical pixel in the destination sampler.
ddxDdy.w - the unit coord distance in y to move one vertical pixel in the destination sampler.

Thus, in the above code, we multiply the Displacement by "float2(length(ddxDdy.xy), length(ddxDdy.zw))". We take the length of x & y combined to tell us about the horizontal, the z & w combined to tell us about the vertical. It's necessary to use all of these because if, for instance, I just used the primary components (x & w), but then rotated my source 90 degrees, these values would be 0, since horizontal movement no longer translated to x-coord movement in unit space.

Introducing Multi-Input Shader Effects

Greg Schechter — Tue, 16 Sep 2008 07:16:00 GMT

Thus far in this series on Effects, we've discussed building and using Effects that have a single "texture" or "sampler" as input to them. In the Beta of .NET 3.5 SP1, that's all that was available. With the RTM release, we've added the ability to provide multiple samplers as input to the pixel shaders that drive Effects, thus substantially increasing the flexibility of what can be done with Effects.

In this post, I'll show a very simple example of multi-input effects. The result isn't anything particularly useful from a UI construction perspective, but it shows the general technique that can be used in more UI-relevant ways. In fact, in the next post I do, I'll show something that's quite a bit more interesting from a UI perspective.

The result

I'll demonstrate a simple shader that just combines, via pixel addition, two images. This lets me take these two images:

and create a little app that can produce these sorts of combinations of the two:

The code

Let's first look at the XAML needed to build this. This example, in fact, has no code at all... it's all XAML. (At least the program that consumes the Effect... the Effect definition itself requires code.)

<Window ... >

  <Window.Resources>
      <ImageBrush x:Key="treeKey" ImageSource="tree.jpg" />
  </Window.Resources>

    <StackPanel>

     <Image Source="toucan.jpg" Width="1024" Height="768">
         <Image.Effect>
             <eff:SimpleMultiInputEffect Input2="{StaticResource treeKey}" 
                             MixInAmount="{Binding ElementName=slider1, Path=Value}" />
         </Image.Effect>
     </Image>

        <StackPanel Margin="10" Orientation="Horizontal">
            <Slider Minimum="-0.5" Maximum="1" SmallChange="1" LargeChange="1" Value="0.5" Name="slider1" Width="289" />
            <Label HorizontalAlignment="Left"  Name="label1" VerticalAlignment="Bottom" Width="120" Content="{Binding ElementName=slider1, Path=Value}" />
        </StackPanel>
    </StackPanel>
</Window>

The large text is the stuff that's more relevant here. First note that we reference our tree.jpg image in an ImageBrush that's defined as a resource. That's because all sampler inputs come in through WPF as Brushes. In this case, it's an ImageBrush, but it could just as easily have been a VisualBrush, and thus anything that can be constructed in a WPF tree.

Next, the <Image> displays the Toucan image, and we apply our custom "SimpleMultiInputEffect" to the <Image>. Here, we set a property, Input2, as a reference to our ImageBrush holding the tree. We also set a double-valued property MixInAmount, which is bound to the slider value. The Effect itself lets MixInAmount control the amount of blending that Input2 will do against the primary input.

Moving to the Effect definition itself. In the C# for the Effect, the only thing notable is that we define another Brush-valued DependencyProperty:

public Brush Input2
{
    get { return (Brush)GetValue(Input2Property); }
    set { SetValue(Input2Property, value); }
}

public static readonly DependencyProperty Input2Property =
    ShaderEffect.RegisterPixelShaderSamplerProperty("Input2", typeof(SimpleMultiInputEffect), 1);

The "1" provided as the last argument to RegisterPixelShaderSamplerProperty() indicates that this Brush's realization will be available in sampler register "1" in the pixel shader.

The HLSL gets a little more interesting:

//-----------------------------------------------------------------------------------------
// Shader constant register mappings (scalars - float, double, Point, Color, Point3D, etc.)
//-----------------------------------------------------------------------------------------

float mixInAmount : register(C0);

//--------------------------------------------------------------------------------------
// Sampler Inputs (Brushes, including ImplicitInput)
//--------------------------------------------------------------------------------------

sampler2D input1 : register(S0);
sampler2D input2 : register(S1);


//--------------------------------------------------------------------------------------
// Pixel Shader
//--------------------------------------------------------------------------------------

float4 main(float2 uv : TEXCOORD) : COLOR
{
   float4 color1 = tex2D(input1, uv);
   float4 color2 = tex2D(input2, uv);
   return color1 + mixInAmount * color2;
}

This is also very simple. All that's happening here is that color1 and color2 are being sampled from the two input samplers (the Toucan and the Tree images, in our application), and combined according to the value of mixInAmount. When that value is 0, we end up with pure Toucan . When it's positive, we're adding Tree in. When negative, we're subtracting Tree out. All on a per-pixel basis.

More explanation

There are a few more details worth expanding on here:

The secondary inputs get sized to the rendered size of the UIElement that the effect is being applied to. Thus, when they make it into the shader, all the sampler inputs are the same pixel size. You can use transforms on the incoming brushes, or viewport/viewbox, to manipulate different portions of the brush into place for finer control. (In the above example, both the toucan image and the tree image are the same size, so it didn't matter here.)
You might be wondering how the initial sampler input gets its value, since we never assigned in Brush-valued resources through XAML or code in previous examples. That's because the default for all samplers (registered via RegisterPixelShaderSamplerProperty) is a special Brush that you get from accessing Effect.ImplicitInput. ImplicitInput just means "construct a brush from the image resulting from rendering the UIElement that this Effect is being applied to". So, in the case above, this is the <Image> element that we have the Toucan in. If we were applying the Effect to a Button, it'd be the rasterization of the Button.

That's it

That's it for now. I've attached the solution that demonstrates this. To build it, it requires the Shader Build Task that I had previously written about. Next time I'll show a use of multiple effects that has a more practical use for UIs than the simple blending demonstrated above.

A VisualStudio BuildTask and project and item templates for writing ShaderEffects

Greg Schechter — Tue, 12 Aug 2008 04:50:43 GMT

In a number of places in this series on Effects, I've alluded to a VisualStudio BuildTask and project templates that ease the authoring of shader-based Effects. We now have one up and available for you to download, install, and use. This was written by Gerhard Schneider and me. (Gerhard is the development lead who's team is responsible for implementing the new GPU-based Effects, among other things.) We're distributing it as a CodePlex project under the Microsoft Public License. You can find it at the WPF Futures site, which is under the umbrella www.codeplex.com/wpf. At that site, you'll find both a .zip file to install from, and the source code that you can peruse/modify yourself if you need to.

Once you have this installed, you can edit .fx files and the the corresponding C# ShaderEffect subclasses all in VisualStudio, and just hit F5 or rebuild from VS to compile the whole project, including compiling the .fx file and inserting the bytecode as a resource into the resultant assembly.

The build task and project templates can be found in "Shader Effects BuildTask and Templates.zip", and in there you'll find a README that documents what you need to do to install.

Basically what the installer does is add a BuildTask assembly to the GAC. This BuildTask gets run for "Effect" labeled project files, and basically runs the HLSL compiler on them, turning a .fx file into a .ps file containing the shader bytecode. That .ps file is then included in the assembly as a resource, which will get loaded into a PixelShader object.

Note that the machine that this is installed on does not need to have the DirectX SDK installed. In this case, it will use a statically linked shader compiler to compile the .fx file into bytecode. If there is a DirectX SDK installed on the machine, it will try to use the latest one as the compiler.

The VS project template lets you create a new Effect Library, with the right build directive in it's .csproj file, and an initial Effect subclass and .fx file. The VS item template lets you add in new Effect subclasses and .fx files to the Effect Library assembly you're building.

One thing to keep in mind: the created C# project references a build task that runs trusted code. This will cause a dialog like the following to pop up when you open a project. You should expect this and click Yes (unless of course you think my code is going to reformat your hard drive :-) ):

Important Note: the software referenced here is provided as-is, with no guarantees of updates/improvements/fixes/etc. You're more than welcome, of course, to grab the code and modify it to suit your purposes.

.NET Framework 3.5SP1 and Visual Studio 2008 SP1 released!

Greg Schechter — Tue, 12 Aug 2008 04:28:11 GMT

Alright! .NET 3.5 SP1 and VS 2008 SP1 are live and available on the web today. There are lots of places to read about all the great stuff there, but from the perspective of this blog, the two big additions over the Beta of .NET 3.5 SP1 are:

Support for multiple texture inputs to shaders used in ShaderEffects. The makes ShaderEffects quite a bit more expressive and powerful.
The addition of D3DImage, allowing composition-level interop between DirectX surfaces and WPF. This feature opens up many, many possibilities of meaningful integration between code that already uses DirectX, or code that is more appropriately written in DirectX than in WPF.

That's just a teaser... I'll describe these in more detail (or link to other resources that do) in upcoming posts.

Interested in working on the WPF team?

Greg Schechter — Thu, 22 May 2008 21:02:46 GMT

As you can see from this and other blogs, there's lots of exciting stuff going on in WPF these days, and we're hiring! Ivo Manolov, our Test Manager has a post about job openings in Quality Assurance for WPF, including a posting for a Test Architect. Also, here's a site that contains job listings for both Software Development positions and Quality Assurance positions.

Interested? Check it out!

Writing custom Effects - adding parameters to Effects

Greg Schechter — Thu, 15 May 2008 23:15:11 GMT

A couple of posts ago, I wrote about writing custom Effects. The example that I dove into was ColorComplementEffect, an Effect that has no parameters other than the incoming "sampler" that it inverts the color on.

This post is going to go into what it takes to add parameters to your Effects, which allows for much, much more powerful Effects.

Shader Constants and Dependency Properties

The first thing to understand is that HLSL shaders expose "shader constants" bound to "shader registers". We saw a shader constant in the form of a "sampler2D" called "implicitInput" in the previous post on writing Effects:

sampler2D implicitInput : register(s0);

The parameters we discuss here will be shader constants of type float, float2, float3, or float4 in HLSL. These shader constants maintain their value for an entire "frame" of the pixel shader executing across every pixel in the frame. This is why they're called "shader constants", since they're constant per-frame, though they can and often do change between frames.

From an Effects point of view, they can be thought of as a property or parameter of the shader/effect. And we already have a great way of dealing with properties in WPF -- we use DependencyProperties, which provides for change notification, databinding, animation, etc.

So... the next step is kind of obvious... we expose HLSL shader constants through custom Dependency Properties on the corresponding Effect.

An example - ThresholdEffect

Let's move to a simple example... the ThresholdEffect used here:

        <eff:ThresholdEffect Threshold="0.25" BlankColor="Orange" />

This Effect just turns pixels that are below the specified Threshold intensity (0.25 in the case above) into the BlankColor (orange, in the case above).

Here's the HLSL for the ThresholdEffect:

sampler2D implicitInput : register(s0);
float threshold : register(c0);
float4 blankColor : register(c1);

float4 main(float2 uv : TEXCOORD) : COLOR
{
    float4 color = tex2D(implicitInput, uv);
    float intensity = (color.r + color.g + color.b) / 3;
    
    float4 result;
    if (intensity > threshold)
    {
        result = color;
    }
    else
    {
        result = blankColor;
    }
    
    return result;
}

It's a very straightforward effect. It samples the texture, figures out the intensity by averaging RGB, and if above the threshold returns the sampled color, otherwise the blankColor.

I'll list the entire C# for the ThresholdEffect class below, but the most important part is to understand how these properties are defined. Here's one of them, Threshold:

public double Threshold
{
    get { return (double)GetValue(ThresholdProperty); }
    set { SetValue(ThresholdProperty, value); }
}

public static readonly DependencyProperty ThresholdProperty = 
    DependencyProperty.Register("Threshold", typeof(double), typeof(ThresholdEffect), 
            new UIPropertyMetadata(0.5, PixelShaderConstantCallback(0)));

The CLR getter/setter Threshold is identical to all CLR getter/setters for DPs... it just does a GetValue/SetValue. Then the definition for ThresholdProperty is also the same as all DPs... the one difference is that the PropertyChangedCallback is created via "PixelShaderConstantCallback(registerNumber)", which generates a callback to be invoked when the property changes. In this case, we pass 0 as the parameter to PixelShaderConstantCallback, since that matches the "threshold" shader constant in the HLSL that's assigned register "c0".

Once we've set this up, ThresholdProperty is just like any other DP in the system. Bind to it, bind from it, animate it, etc.

The only other thing we need to do is call "UpdateShaderValue(ThresholdProperty)" in the constructor of the Effect. This is necessary to inform the system about this value the first time, since the PropertyChangedCallback doesn't execute when the default value is set. Don't forget to call UpdateShaderValue() on each of the properties you define, including InputProperty!!

What Types are supported?

The DependencyProperties that are bound to floating point shader constant registers can be any of the following types:

Double
Single ('float' in C#)
Color
Size
Point
Vector
Point3D
Vector3D
Point4D

They each will go into their shader register filling up whatever number of components of that register are appropriate. For instance, Double and Single go into one component, Color into 4, Size, Point and Vector into 2, etc. Unfilled components are set to '1'.

Some minutiae

Register Limit: There is a limit of 32 floating point registers that can be used in PS 2.0. In the unlikely event that you have more values than that that you want to pack in, you might consider tricks like packing, for instance, two Points into a single Point4D, etc.

What about int and bool registers?: PS 2.0 doesn't deal particularly well with int and bool registers. We decided to support only float registers. If for some reason, you really need int or bool in your HLSL, you can cast a float register as appropriate.

Complete ThresholdEffect listing

Finally, here's the entire listing for the ThresholdEffect class, which includes the Input sampler property, Threshold that we saw above, and the BlankColor property, that's managed in the exact same way that we did Threshold:

public class ThresholdEffect : ShaderEffect
{
    public ThresholdEffect()
    {
        PixelShader = _pixelShader;

        UpdateShaderValue(InputProperty);
        UpdateShaderValue(ThresholdProperty);
        UpdateShaderValue(BlankColorProperty);
    }

    public Brush Input
    {
        get { return (Brush)GetValue(InputProperty); }
        set { SetValue(InputProperty, value); }
    }

    public static readonly DependencyProperty InputProperty =
        ShaderEffect.RegisterPixelShaderSamplerProperty("Input", typeof(ThresholdEffect), 0);


    public double Threshold
    {
        get { return (double)GetValue(ThresholdProperty); }
        set { SetValue(ThresholdProperty, value); }
    }

    public static readonly DependencyProperty ThresholdProperty = 
        DependencyProperty.Register("Threshold", typeof(double), typeof(ThresholdEffect), 
                new UIPropertyMetadata(0.5, PixelShaderConstantCallback(0)));


    public Color BlankColor
    {
        get { return (Color)GetValue(BlankColorProperty); }
        set { SetValue(BlankColorProperty, value); }
    }

    public static readonly DependencyProperty BlankColorProperty =
        DependencyProperty.Register("BlankColor", typeof(Color), typeof(ThresholdEffect), 
                new UIPropertyMetadata(Colors.Transparent, PixelShaderConstantCallback(1)));


    private static PixelShader _pixelShader =
        new PixelShader() { UriSource = Global.MakePackUri("ThresholdEffect.ps") };
}

A simple Effect sample project and ClickOnce application

Greg Schechter — Wed, 14 May 2008 06:55:00 GMT

The previous post in this series began to talk about writing custom Effects for WPF. We looked specifically at ColorComplementEffect, an effect with no parameters. In upcoming posts, we'll get into the details of multi-parameter effects and other specifics. But in the meantime, I thought it would be useful to post a sample VS project and a ClickOnce application from that project for folks just itching to get going.

The app looks like this:

It's nothing fancy, mostly XAML and, in fact, the way the code and structure and UI are constructed are not conducive to hosting all effects now and forever (a more sophisticated hoster of effects would make a lot more use of metadata and reflection to allow arbitrary effects to be applied and parameters to be tweaked). However, this definitely does the job in terms of showing how Effects are used and written. There are two projects in the solution -- one is a MyEffects library, and the other just an EffectTest application that uses Effects from MyEffects.

Note that this code and app are specific to the .NET 3.5 SP1 Beta release. We already know of at least one breaking change in the RTM version that will require this code to be modified (just slightly).

Without further ado:

Here's the ClickOnce application to run directly.
Here's the .zip file of the solution that's attached to this post. All but about 15K of it is taken up by the JPEG image in the background. (The other 15K is the good stuf :-) )

One thing to be aware of in the solution. The compiled pixel shaders are included as resources in the MyEffects library. I did the fxc.exe compilation outside of the project system. As mentioned in the last post, we have a VS BuildTask in the works that will let this be part of the project system, but until then, I included the compiled bytecode directly.

Have fun! More to come...

Writing custom GPU-based Effects for WPF

Greg Schechter — Mon, 12 May 2008 17:08:00 GMT

The last few posts in this series on Effects have focused on the perspective of those using Effects. These Effects have to come into being somehow, and that's where we turn to now.

In this post we showed how to apply a ColorComplementEffect to go from this to this:

ColorComplementEffect, which very much looks like a photographic negative, is just about the simplest Effect one can imagine. Conceptually, all it does is take the RGB color components and create a new color with each component subtracted from 1.0. Let's first show what it takes to write it and add it to your app, then we'll expand and generalize from there in the next post.

Creating the HLSL for color complement

Here's some simple HLSL for doing the color complement. Note that this series is not in the least intended to be an HLSL tutorial. There are a bunch out on the Net, and there is also a Programmer's Guide and Reference Guide on MSDN that can be a good place to start. Also starting from existing examples (like those included here) and tweaking and experimenting is always a good approach.

sampler2D implicitInput : register(s0);

float4 main(float2 uv : TEXCOORD) : COLOR
{
    float4 color = tex2D(implicitInput, uv);

    float4 complement;
    complement.r = color.a - color.r;
    complement.g = color.a - color.g;
    complement.b = color.a - color.b;
    complement.a = color.a;

    return complement;
}

There are a few initial things worth noting here:

The "implicitInput" shader constant is what's in shader constant sampler register S0. That's going to be the "original" composition of the textbox and button over the image, all converted to a "sampler". We'll see in a bit how the association to register s0 happens.
"main" is the entrypoint to the pixel shader. It receives as input the texture coordinate (of type 'float2') of where we're currently outputting in the texture coordinate space of the element being applied to. This varies from 0-1 in u and v. Note again that while HLSL in general provides more flexibility on input types to shaders, for WPF currently, this is always a float2-valued texture coordinate.
Only a pixel shader is being provided here. No vertex shader.

Given these, we now get to the body of the function. The first line "tex2D" samples our input texture at the current sampling position uv, and receives a float4 (a 4-component vector with each value a float) that represents the color. After declaring a local 'complement' float4, we proceed to assign it's R, G, and B values to A minus the corresponding sampled value. The A value (alpha) receives the corresponding sampled value's alpha directly. Finally, we return that value.

[May 1, 2009: My esteemed colleagues Ashraf and Don pointed out that my earlier incarnation of this of (1 - component) was wrong, and it should be (A - component). That's because color RGB has their alpha value premultiplied through already. (1 - component) would work for non-premultiplied. The reason it hasn't come up when I used it was the images I supplied were all opaque (A = 1) images, and in this case (1 - component) == (A - component).]

When the Effect runs, this main() function is executed on every pixel affected. It's run very fast, and quite a bit in parallel, but it is run.

The above isn't the best way to express this HLSL. The thing is that HLSL drives a SIMD vector processor, so rather than have these three separate RGB calculations, we can use HLSL to combine them:

float4 main(float2 uv : TEXCOORD) : COLOR
{
    float4 color = tex2D(implicitInput, uv);

    float4 complement;
    complement.rgb = color.a - color.rgb;
    complement.a = color.a;

    return complement;
}

The syntax "complement.rgb = color.a - color.rgb" will treat the "color.a" as replicated into each element of a 3-vector, and then do the subtraction on each element, all with a single PS 2.0 instruction. (Actually, the HLSL compiler deduces this without me being explicit so the two above shaders generate the same number of instructions, but I never like to count on understanding exactly when compiler optimizations will or won't kick in in super performance sensitive code.)

Compiling the HLSL

WPF Effects do not take the HLSL text directly. You need to compile it into the binary bytecode that DirectX natively accepts, which is how WPF passes it along. To do this, you run fxc.exe on your shader file. Fxc.exe is the shader compiler that comes with the Microsoft DirectX SDK. You can find it at Utilities\Bin\x86\fxc.exe in the SDK.

Say your HLSL was in 'cc.fx', the following command line would generate the compiled bytecode int 'cc.ps' in the same directory:

> fxc /T ps_2_0 /E main /Focc.ps cc.fx

This says to compile to the PS 2.0 profile, and to look for the entrypoint named "main".

(We have a rough prototype that adds the fxc.exe compilation as a MSBuild build task so that it can be incorporated directly into projects without having to break out to a command line shader compiler. When that's further along, we'll post this out there for all to use.)

Writing your managed code Effect subclass

Now it's time to write our managed code that will expose this Effect out to WPF developers. We'll use C# here.

using System;
using System.Windows;
using System.Windows.Media;
using System.Windows.Media.Effects;

namespace MyEffects
{
    public class ColorComplementEffect : ShaderEffect
    {
        public ColorComplementEffect()
        {
            PixelShader = _shader;
            UpdateShaderValue(InputProperty);
        }

        public Brush Input
        {
            get { return (Brush)GetValue(InputProperty); }
            set { SetValue(InputProperty, value); }
        }

        public static readonly DependencyProperty InputProperty =
            ShaderEffect.RegisterPixelShaderSamplerProperty(
                    "Input", 
                    typeof(ColorComplementEffect), 
                    0);

        private static PixelShader _shader =
            new PixelShader() { UriSource = new Uri(@"pack://application:,,,/MyEffects;component/ColorComplementEffect.ps") };

    }
}

Here's basically how this works:

We derive from ShaderEffect, itself a subclass of Effect. Most importantly, ShaderEffect exposes a PixelShader property of type PixelShader.
We define a static PixelShader instance which references the compiled bytecode. It's static because the same PixelShader object can be shared amongst all instances of the Effect.
We define a Brush-valued DependencyProperty called InputProperty, and the corresponding Input CLR property. This is almost identical to how we define other DPs in WPF. The difference is that we use a helper method called ShaderEffect.RegisterPixelShaderSamplerProperty. As in other DP definitions, both the name and the owning type are specified. But the third parameter here (0, in this case) represents the sampler register that the Input property will be able to be accessed from in the shader. Note that this 0 matches the s0 in the HLSL above:
```
sampler2D implicitInput : register(s0);
```

The instance constructor just assigns in the static _shader to the per-instance PixelShader property, and calls UpdateShaderValue() on any DPs that are associated with shader registers. In this case, just the InputProperty. This latter is necessary to ensure it's set for the first time, since DPs don't call their PropertyChangedCallbacks on the setting of their default values.

The other thing worth mentioning here is the gibberish in the "pack://" URI when we reference our pixel shader bytecode file, ColorComponentEffect.ps. Since we don't want to reference a loose file on disk, and we'd like the shader bytecode to live in whatever packaging the C# ColorComponentEffect class goes into (since they should travel together), we use the WPF "pack://" URI syntax. In this case, we're building a library called "MyEffects", which is why that appears in the pack URI.

In order for this "pack://" URI to work, the bytecode needs to make its way into the built component. You do this by adding the shader bytecode file into your project, and ensuring that its Build Action is set to "Resource" as this snip from the Solution Explorer shows:

I have a little helper function that lets me express this URI string without hardwiring in "MyEffects", and without re-generating this same gibberish each time:

new PixelShader() { UriSource = Global.MakePackUri("ColorComplementEffect.ps") }

I include the code for the Global helper class at the bottom of this post.

(Note also that PixelShader objects can also be defined via a Stream, which enables, for instance, authoring tools to construct shader bytecode on the fly and pass it to WPF without ever having to persist it to the file system.)

<Geek-Out-Alert--Somewhat-Esoteric-Stuff-Here> (as if this whole topic isn't geeky enough to start with)

Recall that we considered ColorComplementEffect to be a zero-parameter effect. So what's with this "Input" DP? Note that the Effect is invoked via this XAML:

  <Grid >
    <Grid.Effect>
        <eff:ColorComplementEffect />
    </Grid.Effect>

    ...
  </Grid>

The "sampler" that the shader works on is the rasterization of the Grid itself into a bitmap. You can think of the implementation as creating a VisualBrush of the Grid, then providing that Brush as the value for the Input DP. It doesn't quite work like that, and we expose a new type of Brush called ImplicitInput to represent this usage. ImplicitInput is the default value for DPs that defined using "RegisterPixelShaderSamplerProperty", meaning that they automatically receive the "brush" of the element that they're applied to as their shader sampler (register 0 in this case, since that's what we specified).

We'll see in later posts why exposing these as Brushes, and having the ability to control whether ImplicitInput is used is important. Hint: it has to do with multi-input effects.

</Geek-Out-Alert--Somewhat-Esoteric-Stuff-Here>

OK, that's it for this post... in the next one we'll get into somewhat more complicated Effect definition. In particular, parameterized effects.

Appendix: Global.MakePackUri

This is the MakePackUri helper I referenced above. It's nice in that it's not verbose (like the pack:// URI itself), and, more importantly, it doesn't hardwire in a module name.

internal static class Global
{
    /// <summary>
    /// Helper method for generating a "pack://" URI for a given relative file based on the
    /// assembly that this class is in.
    /// </summary>
    public static Uri MakePackUri(string relativeFile)
    {
        string uriString = "pack://application:,,,/" + AssemblyShortName + ";component/" + relativeFile;
        return new Uri(uriString);
    }

    private static string AssemblyShortName
    {
        get
        {
            if (_assemblyShortName == null)
            {
                Assembly a = typeof(Global).Assembly;

                // Pull out the short name.
                _assemblyShortName = a.ToString().Split(',')[0];
            }

            return _assemblyShortName;
        }
    }

    private static string _assemblyShortName;
}

More details on how Effects work

Greg Schechter — Mon, 12 May 2008 17:06:00 GMT

The last couple posts in this series have dug into features and example usage of Effects in WPF. Let’s go into some other aspects of the feature that are important to understand.

Software Rendering

When we discuss Effects, we typically talk about them being GPU-accelerated. This is typically the case. However, there are three important situations where the Effects cannot be GPU-accelerated:

When the graphics card being run on does not support PixelShader 2.0 or above. This is becoming more and more rare, but is definitely still out there.
When the WPF application is being remoted over Remote Desktop or Terminal Server or some other kind of mirror driver.
When the WPF application is in a mode where software rendering is required – such as rendering to a RenderTargetBitmap, or printing.

In all of these cases, WPF will render Effects via software. What does that mean? As mentioned in an earlier post, Effects are written using HLSL for programming the GPU. WPF incorporates a “JITter” (Just-in-Time compiler) that takes the compiled HLSL bytecode, validates it, and dynamically generates x86 SSE instructions that can execute the HLSL in software the same way it would be executed in the GPU. SSE stands for “Streaming SIMD instructions”, which is an indicator that it’s a decent alternative when there isn’t a GPU available. It’s certainly considerably slower than the GPU (and considerably less parallel), and obviously taxes the CPU more, but given that it’s fundamentally a SIMD processor as well, it’s quite well suited to executing HLSL. All told, we’ve been very pleased, and very pleasantly surprised with how well the CPU does execute shaders in the absence of a GPU.

Interactivity

The Effects shown thus far in this series are all “in-place” effects, in the sense that they output some modification of the pixel that’s at the same location in the input texture as its destination in the output texture. That is, they just manipulate color, and not position. However, distortion types of effects, like swirls, bulges, pinches, etc., are all very powerful, important and achievable Effects. In those cases, you may get a result like this:

It so happens that the button and the textbox in the above image remain fully interactive and operable, and you can click on their onscreen position to give them focus. How does this work? The HLSL shaders know nothing about input? The way this works is that the Effect author has an opportunity to define the “EffectMapping” that the Effect subjects its points to. The author of the Effect is responsible for returning a GeneralTransform that takes a position and returns the post-Effect corresponding position. The author also needs to provide the inverse mapping. This is then used for input testing as well as for functionality like TransformTo{Ancestor,Descendant,Visual}.

(Note that GeneralTransform is a generalization of the Transform class that, while not heavily used, has existed since the first version of WPF, and represents transforms that cannot be expressed via an affine 3x2 matrix, and is thus necessary for representing the flexibility that Effects possess. They were introduced for exactly this sort of purpose.)

Full Trust Requirement

WPF will only run Effects within fully-trusted applications. There is no support currently for running Effects in partially-trusted or untrusted apps. This is something we want to move towards in a subsequent release (particurly for the software rendering implementation).

What sort of shaders?

GPUs expose multiple types of shaders. In shader model 3.0 and prior, there are both Vertex Shaders and Pixel Shaders. PS 4.0 adds Geometry Shaders. For the WPF Effects feature, we only support the use of Pixel Shaders. Furthermore, the only value that varies on per-pixel invocation of the shader is the incoming texture coordinate (uv), representing where on the output surface the shader is running. There are a number of powerful techniques that one can potentially achieve by introducing Vertex Shaders and more flexible input data for the Pixel Shader. However, for our first time getting into the GPU programming arena, WPF is only exposing the feature via Pixel Shaders as described here.

What about BitmapEffects?

The initial release of WPF introduced a class called BitmapEffect, and a property on UIElement also called BitmapEffect. These played much the same role, conceptually, as Effects do. However, they were riddled with performance problems. The main problems with them was that a) they executed in software, and b) they executed on the UI thread. The first issue means they didn’t take advantage of the GPU. The second issue is worse, and means that the entire rendering of everything in the tree leading up to the BitmapEffect needed to be rendered in software as well. Thus, the performance impact on an application was quite considerable. For certain cases (BitmapEffects on small pixel areas that aren’t updating often) things were OK. However, one quickly fell of the performance cliff into the problems I describe above.

In WPF 3.5 SP1, we still have BitmapEffects, though their use is discouraged and, in fact, they are marked [Obsolete] in the WPF assemblies, meaning that their use will generate compiler warnings. This is done so that they can be removed from the system in a future version (not sure just when that would happen), but not have existing apps that depend upon them break.

Of the built-in BitmapEffects, there are two that are far and away the most commonly used: BlurBitmapEffect and DropShadowBitmapEffect. For these, we’ve built equivalent (but hardware accelerated) Effect-based versions that you can attach to UIElement.Effect now. Also, we have an “emulation” layer so that apps that are already out there that hook up UIElement.BitmapEffect to the BlurBitmapEffect or DropShadowBitmapEffect will now result in use of the Effect-based ones. (Note that this won’t work in all cases – for instance, if you use a BitmapEffectGroup, this won’t get emulated.)

Additional RenderCapability APIs

With the addition of Effects, the System.Windows.Media.RenderCapability class gets two new members to allow applications to fine-tune their use of shader-based Effects:

IsPixelShaderVersionSupported(major, minor) will tell you whether the system being run on supports the specified Pixel Shader version on the GPU. This is useful for understanding whether effects that you want to run will run in hardware, as you may potentially want to exclude them if they don't run in hardware.
IsShaderEffectSoftwareRenderingSupported is a Boolean property that says whether the system being run on supports SSE2, which is the determinant for whether the software JITter described above will work.

That’s it for this post, and that pretty much wraps up the general information on using Effects and some of the information surrounding their use. If you’re interested in authoring Effects, you may well be clamoring for more details there. That will be the topic of the next set of posts in this series.

Using Effects in WPF (Part 2)

Greg Schechter — Mon, 12 May 2008 17:04:00 GMT

My previous post on Effects (part of this series) gives a simple example of instantiating an parameterless Effect (it just takes the complement of a color) through XAML. Here we discuss more about the use of Effects.

First, lest you think that Effects are somehow a XAML-only feature, let’s write some code to use Effects. Recall that the invocation of the Effect through XAML looked like this:

  <Grid >
    <Grid.Effect>
        <eff:ColorComplementEffect />
    </Grid.Effect>

    <Image ...  />
    <Button ... />
    <TextBox ... />
  </Grid>

In code, presuming I already have the Grid and it’s called “myGrid”, we’d just do the following:

myGrid.Effect = new MyEffects.ColorComplementEffect();

Like everything else in WPF, Effects are readily creatable and manipulable through code. Given that, most of the below will focus on usage through XAML just because it’s the most succinct way of expressing the usage.

The ColorComplementEffect is very limited in what it can do, as it takes no parameters. Let’s look at using an Effect that does take parameters. This is an intensity-thresholding effect – for each pixel, it determines if the pixel is above a certain intensity and, if it is, outputs that pixel. Otherwise it outputs a constant color (transparent by default). The XAML below applies the ThresholdEffect with a threshold value of 0.2, and yellow as the color to fill in those pixels that are below an average 0.2 intensity.

  <Grid >
    <Grid.Effect>
        <eff:ThresholdEffect Threshold="0.25" BlankColor="Orange" />
    </Grid.Effect>

    <Image ...  />
    <Button ... />
    <TextBox ... />
  </Grid>

The result looks like this:

Because the ThresholdEffect class exposes Threshold and BlankColor as DependencyProperties, we can use our standard WPF techniques to databind and animate these properties. For instance, the XAML below introduces a slider and wires the Threshold value of the ThresholdEffect to the value of the slider. As you move the slider, more or less of the grid content “thresholds” in and out:

   <Grid >
       <Grid.Effect>
           <eff:ThresholdEffect Threshold="{Binding ElementName=thresholdSlider, Path=Value}" BlankColor="Orange" />
       </Grid.Effect>

       <Image ... />
       <Button ... />
       <TextBox .../>
    </Grid>
    <Slider Name="thresholdSlider" Minimum="0" Maximum="1" ... />

Chaining Effects

In the world of GPU programming, there’s a concept known as “multi-pass”, where the programmer sets up multiple pixel shaders and cycles through them, with the output of one becoming the input of the next. In this release of WPF, we do not have explicit support for building multi-pass effects. However, WPF provides a very natural way to express what is typically desired – namely “containment”. Say I want to apply my ThresholdEffect as above, but after that I want to apply the ColorComplementEffect. A very simple way to do this is to take the Grid shown above, and wrap a Border around it. On this Border, add the ColorComplementEffect. Here’s the XAML:

<Border>
    <Border.Effect>
        <eff:ColorComplementEffect />
    </Border.Effect>
    <Grid >
        <Grid.Effect>
            <eff:ThresholdEffect Threshold="0.25" BlankColor="Orange" />
        </Grid.Effect>

        <Image ... /> 
        <Button ... />
        <TextBox .../> 
    </Grid>
</Border>

Here’s what this looks like:

Multi-input Effects

The Effects described here receive a single input bitmap from the UIElement that they are attached to, and they can get an arbitrary number of “scalar” properties through DependencyProperties that the Effect author provides for his users, and that they can be chained via element containment. These “scalar” properties are things like double, Color, Point, Point3D, Vector, Vector3D, Size, etc.

One thing that all the Effects thus far have in common is that they have exactly one bitmap input... the rasterization of the UIElement that they're being applied to. WPF also will support "multi-input" effects where multiple UIElements (or other sources of bitmap data) can be provided to and manipulated by an Effect. (We don't consider effects like ThresholdEffect to be multi-input even though they have multiple properties/parameters... we consider "multi-input" to mean receiving more than one bitmap input.)

Multi-input effects will be part of the .NET 3.5 SP1 release, but they're not in the Beta, so we won't discuss them further here.

What have we seen?

Thus far in these posts we’ve looked at invoking GPU-accelerated Effects from your WPF applications and components.

Effects actually do have more features from a “using Effects” point of view, but for now, this is a good stopping point, as what’s been described above represents the bulk of the power and flexibility of Effects. The next post, rather than giving more examples and features, will talk about some of the operational aspects that are important to know about for the stuff described above.

GPU-accelerated custom effects for WPF

Greg Schechter — Mon, 12 May 2008 17:02:00 GMT

With WPF 3.5 SP1 on the horizon (and the Beta available now), I plan to discuss some of the new graphics features that are coming into WPF in this release. There are a number of great new additions as well as improvements on existing features, but I will say that the one that I’m the most excited about is GPU-accelerated custom effects. I’m going to go into a good amount of depth in a series of upcoming blog posts, so I’ll dedicate this one to talking about the basics of what we’re offering and the motivation behind the feature.

One of the hallmarks of WPF is the ability to mix and match media types, to compose visual elements, and to give the developer a substantial amount of freedom in the way they construct the interfaces for their applications. However, they are typically restricted to using the building blocks that WPF provides, such as rectangles, text, video, paths, gradients, images, etc. As rich as that set of primitives is, it is still a fixed set of primitives that only has the ability to grow as new releases of WPF come out. At the same time, the graphics processing unit (GPU) is becoming ever more powerful and ever more flexible. These two facts are somewhat at odds with each other, and this post talks about ways we bridge the gap by providing for GPU-accelerated custom effects (which I’ll just shorten to “Effects” from this point on).

With Effects, you can harness the programmability of the GPU and still benefit from all the power of animation, data binding, element composition and media integration that WPF offers. Moreover, 3^rd parties can offer very powerful libraries of Effects that other developers using WPF can just use without needing to understand anything about GPU programming.

Applying motion blur to a scrolling list

As an initial example to see the sorts of power and flexibility one has with Effects, consider this UI:

The "film strip" at the top can be scrolled through. Effects have been used as part of this film strip control so that when you move quickly through them, a "motion blur" effect gets applied, slowing down as you come to rest. Here's a resultant snapshot of that:

The rest of this post will look at a simpler example for explanatory purposes.

A simple example of using a built-in Effect

As a first simple example, we'll use the built in DropShadow Effect to make this Button:

look like this:

To do this, all we do is add an Effect property to our button:

<Button ... >
    <Button.Effect>
        <DropShadowEffect />
    </Button.Effect>
    Hello
</Button>

This Effect uses the built-in "DropShadowEffect".

A simple example of using a custom Effect

Using a custom Effect is really identical to using a built-in Effect.

The following is a simple semi-transparent Button and TextBox and an Image placed inside of a Grid.

The XAML for the above is roughly this (I’ve removed properties of Image and Button and TextBox that just aren’t relevant to this discussion):

  <Grid >
    <Image ...  />
    <Button ... />
    <TextBox ... />
  </Grid>

What I’ll do now is apply a very simple Effect called ColorComplementEffect that’s part of a custom assembly I’ve created. All we do is this:

  <Grid >
    <Grid.Effect>
        <eff:ColorComplementEffect />
    </Grid.Effect>

    <Image ...  />
    <Button ... />
    <TextBox ... />
  </Grid>

What we’ve done is established the Effect property on the Grid, and set it to an instance of ColorComplementEffect. Note the “eff:” namespace on this… this indicates my custom assembly, and I need to add the following xmlns declaration to ensure this is found in the namespace MyEffects in the custom assembly MyEffects.dll:

xmlns:eff="clr-namespace:MyEffects;assembly=MyEffects"

ColorComplementEffect, for every pixel in the element it’s being attached to, outputs the complement of that pixel (that is, 1.0 - val, for each of the RGB components of the element). The result looks like this:

Note how the button and text box are still present and remain interactive, just with the colors complemented (which makes it look like a photographic negative).

(A later post will show how to create Effects like ColorComplementEffect, here we just discuss using them.)

What’s happening here?

So what’s a good way to think about what we just did above? The Effects that can be written will operate on pixels, and (roughly) will take bitmaps in (which are just arrays of pixels) and for each pixel that it needs to output, the Effect will generate a color. Yet we applied the Effect to, in the above case, a Grid with multiple children, which we know isn’t a bitmap.

What’s happening is that WPF sees that an Effect is applied to a UIElement, and rasterizes the UIElement at the proper size into a bitmap, and then applies the Effect. Thus, Effects can be applied to any UIElement (or Visual, for that matter). This includes video, and some of the more impactful uses apply Effects to video.

What can I do with these?

It’s the possibilities that GPU-accelerated Effects unlock that have me so excited about this addition. Basically, at GPU processing speeds, applications can apply effects that do, for instance:

Color modification: channel separation, tinting, saturation, contrast, monochrome, toning, thresholding, pixellation, bloom, chromakey, etc.
Displacement effects: swirl, motion blur, ripples, pixellation, sharpen, pinch, bulge, etc.
Generative effects: algorithmic generation of interesting texture patterns, fractals, etc.
Multiple-input effects: blending, masking, storing computational results in textures, etc.

But, the most exciting thing about all of this is that the above are just a small sampling of possibilities we’ve discussed in the team. What we’re certain of is that folks in the WPF community can and are going to be creating Effects that we would have never anticipated and will just blow everyone away.

What are these Effects?

While we don’t discuss the specifics of writing Effects in this posting (later posts will do so at length), we do describe what they are. Let me first describe how one programs a GPU, independent of WPF:

GPU’s are fundamentally SIMD processors, able to operate on a very large number of pixels in parallel. A class of GPU “program” called a “pixel shader” (or a “fragment shader”) is basically a program that will execute on each and every pixel it outputs.
In the Microsoft ecosystem, GPUs are programmed and controlled through DirectX. You download a GPU “program” into the GPU through DirectX, and you use DirectX to invoke that program.
There is a GPU assembly language that the GPU’s operate on. There is also a higher-level language called HLSL that is part of DirectX. DirectX provides a compiler to compile the HLSL down into GPU assembly code.

Given this, an Effect is two things:

A compiled HLSL (or assembly) pixel shader with shader constants used to represent variable inputs from the application.
A .NET class that declares contains the pixel shader, and exposes WPF DependencyProperties that are bound to the shader constants used in the pixel shader. This allows modification of properties and parameters of shaders in the exact same way that properties of other WPF entities are manipulating (including support for animation and databinding of these properties and parameters).

But isn’t writing HLSL hard?

Aspects of writing HLSL can be tricky, particularly adapting to the SIMD mindset and the limit to the number of instructions that can be executed.

In a way, though, the question is missing the point. Our expectation is that there will be many, many consumers of Effects (app developers as well as component developers who just want to use an Effect). Consuming an Effect, as the example above shows, is identical to consuming any other custom-written WPF control or library component. There will be fewer creators of Effects, just because the bar is somewhat higher, requiring an understanding of HLSL. However, we do expect/hope even that number to be pretty high, just because the Effects written can be very highly leveraged by users of the WPF system.

That’s it for this initial post on Effects. Lots more goodness to come in subsequent posts.

A Series on GPU-based Effects for WPF

Greg Schechter — Mon, 12 May 2008 17:00:00 GMT

GPU-based Effects are a hot new feature in WPF for .NET 3.5 SP1. I'm going to be blogging a series of articles with progressively more detailed information in them, and am using this post as a place to store a table of contents of these posts. Here's what's out there so far:

VB.NET version of Planerator posted

Greg Schechter — Mon, 11 Feb 2008 05:08:14 GMT

Sean Cullinan, of blendblog.net, recently posted a VB.NET port of the Planerator.

Some implementation details of the Planerator

Greg Schechter — Wed, 28 Nov 2007 09:31:28 GMT

My previous two posts (here and here) discuss usage of the Planerator control. There are some unique issues that needed to be resolved in the implementation that WPF geeks might be interested in. If you just are interested in using it, and not finding out how the sausage is made, you might want to just skip this.

Actually, though, the sausage-making in this case is pretty nice, and the Planerator implementation is quite clean. It's contained in about 280 lines of C# code in Planerator.cs. There are some tricky aspects, though. Below are the salient points.

Interactive Content on 3D

One of the new features in WPF 3.5 is Viewport2DVisual3D, which is a way of getting a UIElement in as a material in 3D, and yet have it remain interactive. This is used in the Planerator for the front face of the plane. The back face of the plane just uses a VisualBrush for its material, and thus isn't interactive.

Caching of brush realizations

One critical performance optimization that the developer must opt into is to allow caching of the bitmaps that result from rasterizing a VisualBrush or Viewport2DVisual3D. Without doing this, each frame renders the material anew, which can be quite expensive. There are static methods off of RenderOptions that control caching in WPF. In Planerator.cs, these are called from the "SetCachingForObject" private method.

Layout invalidation

When the contents of the Planerator change internally resulting in layout changes to those contents (for instance, a Label in a horizontal StackPanel gets longer, making the entire content wider), then the Planerator itself should be remeasured. This, though, poses a problem because the contents are embedded in a Viewport2DVisual3D, and layout doesn't flow in and out of that.

The solution pursued here is that when we set up the Planerator, we wrap the child in a custom Decorator defined in the project and called LayoutInvalidationCatcher. LayoutInvalidationCatcher overrides MeasureOverride and ArrangeOverride, and results in invalidating the Planerator instance's measure and arrange, respectively, thus forcing the Planerator to be re-laid out, and getting the results we want.

Databinding passthrough

A problem similar to the Layout Invalidation problem above is with databinding. Because the children are embedded in Viewport2DVisual3D, databinding processing doesn't flow into it. Therefore, we make the logical child of the Planerator be the content itself, while the visual child is the 3D construct that get built up, including the use of Viewport2DVisual3D. By doing this, we allow databinding to flow to the Planerator's child properly, through the logical tree. Then everything is set up to render it as part of the visual tree.

Proper sizing

Finally, a key aspect to making Planerator meaningful and friction-free is that it sizes itself precisely to the size of the underlying content, as if the Planerator just weren't there (at least until you start changing the rotation angles). There are two relevant pieces to this. One is camera setup, and this is described in detail in this previous post of mine.

The second is in determining the proper size of the child element itself. It's not sufficient to use the results of Measure(), since those aren't the "ink bounds"... that is, it includes whitespace that doesn't get rendered to because, for instance, a Canvas is made explicitly wider than all the stuff that renders into it. This is problematic because when such a visual is applied as a VisualBrush (or as Viewport2DVisual3D) it just uses the rendered bounds of the Visual, and thus the result would be distorted if we used Measure() results. Thus, in order to get the correct "ink bounds", we use VisualTreeHelper.GetDescendantBounds().

Put all those together, and you get a pretty cool Planerator control.