Greg Schechter's Blog

Introducing the new Bing Maps beta

2009-12-07T05:19:51Z

Well, I’ve been off the air for quite some time, and now have the opportunity to pop up and talk about what I’ve been working on lately. I made the switch from focusing on building platforms like Silverlight, WPF and the Desktop Window Manager to building an app built on such platforms – namely the Bing Maps site. It’s great way to be able to see what you’ve been working on from a different perspective, and exercise a different set of muscles.

Last week was an exciting one: we launched the beta of the new Bing Maps site, the first to use Silverlight. We don’t use Silverlight just for the sake of using it, but because it enables fantastic user-experiences and superior performance and fluidity. Chris Pendleton’s blog entry give a lot more detail on what we’ve released. Rather than repeating it here, I recommend just giving his entry a read. Meanwhile, I will provide some links to some cool examples here – check ‘em out:

Bird’s Eye view in Manhattan looking over Central Park West
Seattle’s Fremont neighborhood highlighting establishments via “What’s nearby”
A Photosynth, displayed in Bing Maps, of a famous sculpture of socialist workers in Tiananmen Square that I took back in May
Today’s Front Pages – from around the world
Current Tweets from downtown Chicago
Locally relevant blogging content from San Francisco

We’ve learned a lot in building the site, and we have lots of surprises and improvements up our sleeves for upcoming releases, functionality, and perf improvements. So stay tuned, and enjoy using Bing Maps!

Effect Library posted

2008-10-23T02:13:50Z

Troy Jefferson interned this summer with the WPF team and gathered a bunch of Effects that had been written by members of the WPF team. These are now available for downloading on CodePlex. At that same site, you can also browse to a video of these effects in action.

The single input effects in this library include:

BandedSwirl
Bloom
BrightExtract
ColorKeyAlpha
ColorTone
ContrastAdjust
DirectionalBlur
Embossed
Gloom
GrowablePoissonDiskEffect
InvertColor
LightStreak
Magnify
Monochrome
Pinch
Pixelate
Ripple SharpenSmoothMagnify
Swirl
Tone
Toon
ZoomBlur

There are also a set of two-input effects package up as "transitions" for doing interesting visual transitions between elements. These include:

BandedSwirl
Blinds
Blood
CircleReveal
CircleStretch
CircularBlur
CloudReveral
Cloudy
Crumble
Dissolve
DropFade
Fade
LeastBright
LineReveal
MostBright
PixelateIn
PixelateOut
Pixelate
RadialBlur
RadialWiggle
RandomCircleReveal
Ripple
Rotate
Saturate
Shrink
SlideIn
SmoothSwirl
Swirl
Water
Wave.

Thanks, Troy!

A More Useful Multi-Input Effect

2008-09-28T05:00:00Z

In my last post, I introduced multi-input effects, where you can send in arbitrary WPF brushes that are interpreted as samplers in the shader. The example I did just showed simple image combination via linear combination of pixel values -- illustrative, but not very useful.

In this post, I'm going to demonstrate a more useful application of multi-input effects. One of the things that TV networks and local stations do to identify themselves as programs are airing is display a little image that looks like a bit like a watermark in the lower portion of the screen. This is subtle enough to not be distracting, but also clearly identifies the source. This sort of subtle branding can be quite powerful.

Some do this watermarking just via a translucent icon (trivial to do in WPF without Effects). Others use the icon in a more sophisticated way. We're going to show a more sophisticated way here.

Here is a screenshot of the sample app that's included in this post. Note the rose logo in the lower right hand corner of the image.

If you look closely at this close up, you may be able to see that the rose icon is really shifting the underlying pixels.

This is known as "displacement mapping", and makes for a subtle yet distinctive effect as opposed to simply alpha blending the icon on top. Note that here we use it on top of a static image. However, this can be used on live video, on UI, atop DirectX content, etc.

Using the LogoDisplacer effect

The attached project includes a "LogoDisplacer" Effect. The above is modeled as so in XAML:

<Image Source="desert.jpg" Stretch="None" Loaded="Image_Loaded">
    <Image.Effect>
        <lds:LogoDisplacer Logo="{StaticResource logoImage}" 
               Displacement="20" AdditionalLogoOpacity="0.4" />
    </Image.Effect>
</Image>

(For simplicitly, I've elided the databinding syntax that hooks the Displacement and AdditionalLogoOpacity properties to sliders.) The "Logo" property itself is the secondary sampler input, and comes in as a reference to an ImageBrush resource in this case, defined as:

<Window.Resources>
    <ImageBrush x:Key="logoImage" ImageSource="LogoImage.png" />
</Window.Resources>

Here, LogoImage.png looks like this, but yours could be anything:

This is a snapshot from Photoshop where the checkerboard represents fully transparent background pixels. (I created this in Photoshop just by using a character from the Zapf Dingbats font, and then having the Layer Blending options set to Bevel & Emboss with Contouring.)

The LogoDisplacer shader effect uses this logo image to displace the content under it. It does so by, for each pixel, seeing how far the red channel is from 0.5, and offsetting in x the lookup of the pixel from the input image by that fraction of the "Displacement" property (in pixels). It will do the same with the green channel and the y offset. Thus if the image is grayscale (as in this example), red and green are the same, and the x and y displacement will be the same. This is all modulated by the alpha channel of the logo, so where there are transparent pixels in the logo, no displacement occurs.

After this potentially displaced lookup into the original image happens, the corresponding pixel from the logo image itself is blended in with "AdditionalLogoOpacity" alpha, to make the logo more apparent. AdditionalLogoOpacity should be small enough to not overshadow the displacement, but big enough to allow the overlay to be discernible.

When I apply the LogoDisplacer effect as constructed above, my result looks like this:

Not quite the result we were hoping for, since we want the rose just in the lower right hand corner, and at it's original pixel size.

The reason the rose takes up the whole image was discussed in the last post. The sampler inputs are always sized to match the render size of the UIElement that the effect is being applied to. So even though our logo image is 100x100, it gets scaled up to the size of the desert image (1024x768), and that's why it's pixellated.

We need to do something that prevents this resize/pixellation. Since we know that whatever input we provide will be scaled to the size of the UIElement, we can't do a predetermined scaling -- that is, we can't create a 1024x768 image and just put the rose in the lower right hand corner, because if I apply to a UIElement of a different size, the scaling will be wrong again.

Using the Viewbox property

What we do instead is rely on the Viewbox property of ImageBrush. By default, Viewbox is the unit-rect ([0,0,1,1]) that the image fills. If we make the Viewbox larger, we can in effect position the unit-rectangle (and thus the logo) anywhere within that larger viewbox. If I make the Viewbox be [0,0,2,2], then the logo will fill the upper left quadrant. If I make it (-1,-1,2,2), then it will fill the lower right quadrant.

A little math then lets us figure out how to size the Viewbox so that it fits in the lower right 100x100 pixels (or whatever the logo's pixel size is), based on the actual width and height of the UIElement. The following is the event handler invoked when the desert image above completes loading:

private void Image_Loaded(object sender, RoutedEventArgs e)
{
    ImageBrush logo = (ImageBrush)this.Resources["logoImage"];

    // Create a viewbox so that the original logo image is in the 
    // lower right hand corner at its original pixel size (relative 
    // to the image element).
    Image thisImage = (Image)sender;
    BitmapSource bms = logo.ImageSource as BitmapSource;
    double scaleX = (double)thisImage.ActualWidth / bms.PixelWidth;
    double scaleY = (double)thisImage.ActualHeight / bms.PixelHeight;

    // Choose viewbox to move it into the lower right.
    logo.Viewbox = new Rect(1.0 - scaleX, 1.0 - scaleY, scaleX, scaleY);
}

Now, when we run the app, we get the first image shown in this post, with the logo at its correct pixel size in the lower right hand corner of the image.

Note also that since the logo is just an ImageBrush, it can be made dynamic. The attached solution has the logo slowly rotating about its center.

A note about performance

The performance impact of a sample like this can be deceiving, since we're only showing the lower right hand corner of the element being affected. However, remember that all pixels in the element are being examined..., but only those within the logo are being modified. Thus, any modifications to the element itself will cause the entire effect to be re-run over the element. So, while the GPU is incredibly fast, there are still limits, and one should be aware of the implications.

Writing the LogoDisplacer effect

Let's move on to how the LogoDisplacer effect gets written.

The C# file

The C# file, LogoDisplacer.cs, doesn't really have anything very interesting. It just sets up the DependencyProperties that are bound to shader registers, as we've seen with other shaders. In this case, there's Input (Sampler Register 0), Logo (Sampler Register 1), Displacement (Constant Register 0), and AdditionalLogoOpacity (Constant Register 1). We also include this mellifluous property setting, which we discuss in the Appendix of this posting:

this.DdxUvDdyUvRegisterIndex = 6;

The FX file

This is where things get more interesting. Here's the entire .fx file:

//-----------------------------------------------------------------------------------------
// Shader constant register mappings (scalars - float, double, Point, Color, Point3D, etc.)
//-----------------------------------------------------------------------------------------

float  displacement : register(C0);
float  additionalLogoOpacity : register(C1);
float4 ddxDdy : register(C6);

//--------------------------------------------------------------------------------------
// Sampler Inputs (Brushes, including ImplicitInput)
//--------------------------------------------------------------------------------------

sampler2D implicitInputSampler : register(S0);
sampler2D logoSampler : register(S1);


//--------------------------------------------------------------------------------------
// Pixel Shader
//--------------------------------------------------------------------------------------

float4 main(float2 uv : TEXCOORD) : COLOR
{
   // Pull the sample from the logo.
   float4 logoSample = tex2D(logoSampler, uv);
   
   // See how far away the the red and green channels are from "gray".  Use this
   // value to shift.
   float2 fracShift = float2(0.5,0.5) - logoSample.rg;
   
   // But first modulate the shift by the alpha in the logo.  Most of the logo is
   // transparent, so this will zero out fracShift anywhere other than the logo.
   fracShift *= logoSample.a;
   
   // Calculate coordinate to sample main image at, by displacing by the logo's
   // "distance from gray".  ddxDdy is used to ensure that "displacement" is treated
   // in pixel units (it maps from pixel units to [0,1] units).
   float2 displacedCoord = uv + fracShift * displacement 
                              * float2(length(ddxDdy.xy), length(ddxDdy.zw));
    
   // Now get the main image's color at that displaced coordinate.
   float4 color = tex2D(implicitInputSampler, displacedCoord);
    
   // And mix in that color with a portion of the logo, to make the logo more clearly
   // visible.  Modulate it with "additionalLogoOpacity".
   float4 finalColor = color + logoSample * logoSample.a * additionalLogoOpacity;
   
   return finalColor;
}

The first sections (up until "main") are just setting up the register associations that we showed in the C# file. Then, the main() entry point:

Pulls out the color value from the logo and sees how far red and green are from 0.5 (modulating by the alpha value so it will be zero for transparent pixels).
Figures out where to sample the main input image based upon the distances from 0.5, multiplied by the incoming "displacement" value that was in a DependencyProperty. (The DdxDdy thing is mentioned here... we'll get there soon...)
Now samples the main input sampler at this potentially displaced coordinate, arriving at a color.
Finally adds in some portion of the logo pixel itself on the displaced color from the main input sampler, based upon the value of the AdditionalLogoOpacity DependencyProperty.

This results in the subtle but visible displacement effect we see in the above images. A running solution is attached to this post (note that it requires the ShaderBuildTask from here.)

That's all for now!

Appendix: Advanced - a little about DdxUvDdyUvRegisterIndex

(Note that this is a fairly complicated topic that I'm not going to do full justice to in this post.)

Recall that the PixelShader system samples textures within a unit square, [0,0,1,1]. However, our Displacement value is provided in pixels (which makes sense, since we typically want to display the logo at its original pixel size). We need a way of converting from these pixel values into the unit square. That's where DdxUvDdyUvRegisterIndex comes in. You set it in the C# file to a constant register index, then in your shader, you access that register constant. This constant is a float4, representing the increment in unit coordinates that the GPU needs to move when sampling to get to the next pixel to process. The components are as follows:

ddxDdy.x - the unit coord distance in x to move one horizontal pixel in the destination sampler.
ddxDdy.y - the unit coord distance in y to move one horizontal pixel in the destination sampler.
ddxDdy.z - the unit coord distance in x to move one vertical pixel in the destination sampler.
ddxDdy.w - the unit coord distance in y to move one vertical pixel in the destination sampler.

Thus, in the above code, we multiply the Displacement by "float2(length(ddxDdy.xy), length(ddxDdy.zw))". We take the length of x & y combined to tell us about the horizontal, the z & w combined to tell us about the vertical. It's necessary to use all of these because if, for instance, I just used the primary components (x & w), but then rotated my source 90 degrees, these values would be 0, since horizontal movement no longer translated to x-coord movement in unit space.

Introducing Multi-Input Shader Effects

2008-09-16T10:16:00Z

Thus far in this series on Effects, we've discussed building and using Effects that have a single "texture" or "sampler" as input to them. In the Beta of .NET 3.5 SP1, that's all that was available. With the RTM release, we've added the ability to provide multiple samplers as input to the pixel shaders that drive Effects, thus substantially increasing the flexibility of what can be done with Effects.

In this post, I'll show a very simple example of multi-input effects. The result isn't anything particularly useful from a UI construction perspective, but it shows the general technique that can be used in more UI-relevant ways. In fact, in the next post I do, I'll show something that's quite a bit more interesting from a UI perspective.

The result

I'll demonstrate a simple shader that just combines, via pixel addition, two images. This lets me take these two images:

and create a little app that can produce these sorts of combinations of the two:

The code

Let's first look at the XAML needed to build this. This example, in fact, has no code at all... it's all XAML. (At least the program that consumes the Effect... the Effect definition itself requires code.)

<Window ... >

  <Window.Resources>
      <ImageBrush x:Key="treeKey" ImageSource="tree.jpg" />
  </Window.Resources>

    <StackPanel>

     <Image Source="toucan.jpg" Width="1024" Height="768">
         <Image.Effect>
             <eff:SimpleMultiInputEffect Input2="{StaticResource treeKey}" 
                             MixInAmount="{Binding ElementName=slider1, Path=Value}" />
         </Image.Effect>
     </Image>

        <StackPanel Margin="10" Orientation="Horizontal">
            <Slider Minimum="-0.5" Maximum="1" SmallChange="1" LargeChange="1" Value="0.5" Name="slider1" Width="289" />
            <Label HorizontalAlignment="Left"  Name="label1" VerticalAlignment="Bottom" Width="120" Content="{Binding ElementName=slider1, Path=Value}" />
        </StackPanel>
    </StackPanel>
</Window>

The large text is the stuff that's more relevant here. First note that we reference our tree.jpg image in an ImageBrush that's defined as a resource. That's because all sampler inputs come in through WPF as Brushes. In this case, it's an ImageBrush, but it could just as easily have been a VisualBrush, and thus anything that can be constructed in a WPF tree.

Next, the <Image> displays the Toucan image, and we apply our custom "SimpleMultiInputEffect" to the <Image>. Here, we set a property, Input2, as a reference to our ImageBrush holding the tree. We also set a double-valued property MixInAmount, which is bound to the slider value. The Effect itself lets MixInAmount control the amount of blending that Input2 will do against the primary input.

Moving to the Effect definition itself. In the C# for the Effect, the only thing notable is that we define another Brush-valued DependencyProperty:

public Brush Input2
{
    get { return (Brush)GetValue(Input2Property); }
    set { SetValue(Input2Property, value); }
}

public static readonly DependencyProperty Input2Property =
    ShaderEffect.RegisterPixelShaderSamplerProperty("Input2", typeof(SimpleMultiInputEffect), 1);

The "1" provided as the last argument to RegisterPixelShaderSamplerProperty() indicates that this Brush's realization will be available in sampler register "1" in the pixel shader.

The HLSL gets a little more interesting:

//-----------------------------------------------------------------------------------------
// Shader constant register mappings (scalars - float, double, Point, Color, Point3D, etc.)
//-----------------------------------------------------------------------------------------

float mixInAmount : register(C0);

//--------------------------------------------------------------------------------------
// Sampler Inputs (Brushes, including ImplicitInput)
//--------------------------------------------------------------------------------------

sampler2D input1 : register(S0);
sampler2D input2 : register(S1);


//--------------------------------------------------------------------------------------
// Pixel Shader
//--------------------------------------------------------------------------------------

float4 main(float2 uv : TEXCOORD) : COLOR
{
   float4 color1 = tex2D(input1, uv);
   float4 color2 = tex2D(input2, uv);
   return color1 + mixInAmount * color2;
}

This is also very simple. All that's happening here is that color1 and color2 are being sampled from the two input samplers (the Toucan and the Tree images, in our application), and combined according to the value of mixInAmount. When that value is 0, we end up with pure Toucan . When it's positive, we're adding Tree in. When negative, we're subtracting Tree out. All on a per-pixel basis.

More explanation

There are a few more details worth expanding on here:

The secondary inputs get sized to the rendered size of the UIElement that the effect is being applied to. Thus, when they make it into the shader, all the sampler inputs are the same pixel size. You can use transforms on the incoming brushes, or viewport/viewbox, to manipulate different portions of the brush into place for finer control. (In the above example, both the toucan image and the tree image are the same size, so it didn't matter here.)
You might be wondering how the initial sampler input gets its value, since we never assigned in Brush-valued resources through XAML or code in previous examples. That's because the default for all samplers (registered via RegisterPixelShaderSamplerProperty) is a special Brush that you get from accessing Effect.ImplicitInput. ImplicitInput just means "construct a brush from the image resulting from rendering the UIElement that this Effect is being applied to". So, in the case above, this is the <Image> element that we have the Toucan in. If we were applying the Effect to a Button, it'd be the rasterization of the Button.

That's it

That's it for now. I've attached the solution that demonstrates this. To build it, it requires the Shader Build Task that I had previously written about. Next time I'll show a use of multiple effects that has a more practical use for UIs than the simple blending demonstrated above.

A VisualStudio BuildTask and project and item templates for writing ShaderEffects

2008-08-12T07:50:43Z

In a number of places in this series on Effects, I've alluded to a VisualStudio BuildTask and project templates that ease the authoring of shader-based Effects. We now have one up and available for you to download, install, and use. This was written by Gerhard Schneider and me. (Gerhard is the development lead who's team is responsible for implementing the new GPU-based Effects, among other things.) We're distributing it as a CodePlex project under the Microsoft Public License. You can find it at the WPF Futures site, which is under the umbrella www.codeplex.com/wpf. At that site, you'll find both a .zip file to install from, and the source code that you can peruse/modify yourself if you need to.

Once you have this installed, you can edit .fx files and the the corresponding C# ShaderEffect subclasses all in VisualStudio, and just hit F5 or rebuild from VS to compile the whole project, including compiling the .fx file and inserting the bytecode as a resource into the resultant assembly.

The build task and project templates can be found in "Shader Effects BuildTask and Templates.zip", and in there you'll find a README that documents what you need to do to install.

Basically what the installer does is add a BuildTask assembly to the GAC. This BuildTask gets run for "Effect" labeled project files, and basically runs the HLSL compiler on them, turning a .fx file into a .ps file containing the shader bytecode. That .ps file is then included in the assembly as a resource, which will get loaded into a PixelShader object.

Note that the machine that this is installed on does not need to have the DirectX SDK installed. In this case, it will use a statically linked shader compiler to compile the .fx file into bytecode. If there is a DirectX SDK installed on the machine, it will try to use the latest one as the compiler.

The VS project template lets you create a new Effect Library, with the right build directive in it's .csproj file, and an initial Effect subclass and .fx file. The VS item template lets you add in new Effect subclasses and .fx files to the Effect Library assembly you're building.

One thing to keep in mind: the created C# project references a build task that runs trusted code. This will cause a dialog like the following to pop up when you open a project. You should expect this and click Yes (unless of course you think my code is going to reformat your hard drive :-) ):

Important Note: the software referenced here is provided as-is, with no guarantees of updates/improvements/fixes/etc. You're more than welcome, of course, to grab the code and modify it to suit your purposes.

.NET Framework 3.5SP1 and Visual Studio 2008 SP1 released!

2008-08-12T07:28:11Z

Alright! .NET 3.5 SP1 and VS 2008 SP1 are live and available on the web today. There are lots of places to read about all the great stuff there, but from the perspective of this blog, the two big additions over the Beta of .NET 3.5 SP1 are:

Support for multiple texture inputs to shaders used in ShaderEffects. The makes ShaderEffects quite a bit more expressive and powerful.
The addition of D3DImage, allowing composition-level interop between DirectX surfaces and WPF. This feature opens up many, many possibilities of meaningful integration between code that already uses DirectX, or code that is more appropriately written in DirectX than in WPF.

That's just a teaser... I'll describe these in more detail (or link to other resources that do) in upcoming posts.

Interested in working on the WPF team?

2008-05-23T00:02:46Z

As you can see from this and other blogs, there's lots of exciting stuff going on in WPF these days, and we're hiring! Ivo Manolov, our Test Manager has a post about job openings in Quality Assurance for WPF, including a posting for a Test Architect. Also, here's a site that contains job listings for both Software Development positions and Quality Assurance positions.

Interested? Check it out!

Writing custom Effects - adding parameters to Effects

2008-05-16T02:15:11Z

A couple of posts ago, I wrote about writing custom Effects. The example that I dove into was ColorComplementEffect, an Effect that has no parameters other than the incoming "sampler" that it inverts the color on.

This post is going to go into what it takes to add parameters to your Effects, which allows for much, much more powerful Effects.

Shader Constants and Dependency Properties

The first thing to understand is that HLSL shaders expose "shader constants" bound to "shader registers". We saw a shader constant in the form of a "sampler2D" called "implicitInput" in the previous post on writing Effects:

sampler2D implicitInput : register(s0);

The parameters we discuss here will be shader constants of type float, float2, float3, or float4 in HLSL. These shader constants maintain their value for an entire "frame" of the pixel shader executing across every pixel in the frame. This is why they're called "shader constants", since they're constant per-frame, though they can and often do change between frames.

From an Effects point of view, they can be thought of as a property or parameter of the shader/effect. And we already have a great way of dealing with properties in WPF -- we use DependencyProperties, which provides for change notification, databinding, animation, etc.

So... the next step is kind of obvious... we expose HLSL shader constants through custom Dependency Properties on the corresponding Effect.

An example - ThresholdEffect

Let's move to a simple example... the ThresholdEffect used here:

        <eff:ThresholdEffect Threshold="0.25" BlankColor="Orange" />

This Effect just turns pixels that are below the specified Threshold intensity (0.25 in the case above) into the BlankColor (orange, in the case above).

Here's the HLSL for the ThresholdEffect:

sampler2D implicitInput : register(s0);
float threshold : register(c0);
float4 blankColor : register(c1);

float4 main(float2 uv : TEXCOORD) : COLOR
{
    float4 color = tex2D(implicitInput, uv);
    float intensity = (color.r + color.g + color.b) / 3;
    
    float4 result;
    if (intensity > threshold)
    {
        result = color;
    }
    else
    {
        result = blankColor;
    }
    
    return result;
}

It's a very straightforward effect. It samples the texture, figures out the intensity by averaging RGB, and if above the threshold returns the sampled color, otherwise the blankColor.

I'll list the entire C# for the ThresholdEffect class below, but the most important part is to understand how these properties are defined. Here's one of them, Threshold:

public double Threshold
{
    get { return (double)GetValue(ThresholdProperty); }
    set { SetValue(ThresholdProperty, value); }
}

public static readonly DependencyProperty ThresholdProperty = 
    DependencyProperty.Register("Threshold", typeof(double), typeof(ThresholdEffect), 
            new UIPropertyMetadata(0.5, PixelShaderConstantCallback(0)));

The CLR getter/setter Threshold is identical to all CLR getter/setters for DPs... it just does a GetValue/SetValue. Then the definition for ThresholdProperty is also the same as all DPs... the one difference is that the PropertyChangedCallback is created via "PixelShaderConstantCallback(registerNumber)", which generates a callback to be invoked when the property changes. In this case, we pass 0 as the parameter to PixelShaderConstantCallback, since that matches the "threshold" shader constant in the HLSL that's assigned register "c0".

Once we've set this up, ThresholdProperty is just like any other DP in the system. Bind to it, bind from it, animate it, etc.

The only other thing we need to do is call "UpdateShaderValue(ThresholdProperty)" in the constructor of the Effect. This is necessary to inform the system about this value the first time, since the PropertyChangedCallback doesn't execute when the default value is set. Don't forget to call UpdateShaderValue() on each of the properties you define, including InputProperty!!

What Types are supported?

The DependencyProperties that are bound to floating point shader constant registers can be any of the following types:

Double
Single ('float' in C#)
Color
Size
Point
Vector
Point3D
Vector3D
Point4D

They each will go into their shader register filling up whatever number of components of that register are appropriate. For instance, Double and Single go into one component, Color into 4, Size, Point and Vector into 2, etc. Unfilled components are set to '1'.

Some minutiae

Register Limit: There is a limit of 32 floating point registers that can be used in PS 2.0. In the unlikely event that you have more values than that that you want to pack in, you might consider tricks like packing, for instance, two Points into a single Point4D, etc.

What about int and bool registers?: PS 2.0 doesn't deal particularly well with int and bool registers. We decided to support only float registers. If for some reason, you really need int or bool in your HLSL, you can cast a float register as appropriate.

Complete ThresholdEffect listing

Finally, here's the entire listing for the ThresholdEffect class, which includes the Input sampler property, Threshold that we saw above, and the BlankColor property, that's managed in the exact same way that we did Threshold:

public class ThresholdEffect : ShaderEffect
{
    public ThresholdEffect()
    {
        PixelShader = _pixelShader;

        UpdateShaderValue(InputProperty);
        UpdateShaderValue(ThresholdProperty);
        UpdateShaderValue(BlankColorProperty);
    }

    public Brush Input
    {
        get { return (Brush)GetValue(InputProperty); }
        set { SetValue(InputProperty, value); }
    }

    public static readonly DependencyProperty InputProperty =
        ShaderEffect.RegisterPixelShaderSamplerProperty("Input", typeof(ThresholdEffect), 0);


    public double Threshold
    {
        get { return (double)GetValue(ThresholdProperty); }
        set { SetValue(ThresholdProperty, value); }
    }

    public static readonly DependencyProperty ThresholdProperty = 
        DependencyProperty.Register("Threshold", typeof(double), typeof(ThresholdEffect), 
                new UIPropertyMetadata(0.5, PixelShaderConstantCallback(0)));


    public Color BlankColor
    {
        get { return (Color)GetValue(BlankColorProperty); }
        set { SetValue(BlankColorProperty, value); }
    }

    public static readonly DependencyProperty BlankColorProperty =
        DependencyProperty.Register("BlankColor", typeof(Color), typeof(ThresholdEffect), 
                new UIPropertyMetadata(Colors.Transparent, PixelShaderConstantCallback(1)));


    private static PixelShader _pixelShader =
        new PixelShader() { UriSource = Global.MakePackUri("ThresholdEffect.ps") };
}

A simple Effect sample project and ClickOnce application

2008-05-14T09:55:00Z

The previous post in this series began to talk about writing custom Effects for WPF. We looked specifically at ColorComplementEffect, an effect with no parameters. In upcoming posts, we'll get into the details of multi-parameter effects and other specifics. But in the meantime, I thought it would be useful to post a sample VS project and a ClickOnce application from that project for folks just itching to get going.

The app looks like this:

It's nothing fancy, mostly XAML and, in fact, the way the code and structure and UI are constructed are not conducive to hosting all effects now and forever (a more sophisticated hoster of effects would make a lot more use of metadata and reflection to allow arbitrary effects to be applied and parameters to be tweaked). However, this definitely does the job in terms of showing how Effects are used and written. There are two projects in the solution -- one is a MyEffects library, and the other just an EffectTest application that uses Effects from MyEffects.

Note that this code and app are specific to the .NET 3.5 SP1 Beta release. We already know of at least one breaking change in the RTM version that will require this code to be modified (just slightly).

Without further ado:

Here's the ClickOnce application to run directly.
Here's the .zip file of the solution that's attached to this post. All but about 15K of it is taken up by the JPEG image in the background. (The other 15K is the good stuf :-) )

One thing to be aware of in the solution. The compiled pixel shaders are included as resources in the MyEffects library. I did the fxc.exe compilation outside of the project system. As mentioned in the last post, we have a VS BuildTask in the works that will let this be part of the project system, but until then, I included the compiled bytecode directly.

Have fun! More to come...

Writing custom GPU-based Effects for WPF

2008-05-12T20:08:00Z

The last few posts in this series on Effects have focused on the perspective of those using Effects. These Effects have to come into being somehow, and that's where we turn to now.

In this post we showed how to apply a ColorComplementEffect to go from this to this:

ColorComplementEffect, which very much looks like a photographic negative, is just about the simplest Effect one can imagine. Conceptually, all it does is take the RGB color components and create a new color with each component subtracted from 1.0. Let's first show what it takes to write it and add it to your app, then we'll expand and generalize from there in the next post.

Creating the HLSL for color complement

Here's some simple HLSL for doing the color complement. Note that this series is not in the least intended to be an HLSL tutorial. There are a bunch out on the Net, and there is also a Programmer's Guide and Reference Guide on MSDN that can be a good place to start. Also starting from existing examples (like those included here) and tweaking and experimenting is always a good approach.

sampler2D implicitInput : register(s0);

float4 main(float2 uv : TEXCOORD) : COLOR
{
    float4 color = tex2D(implicitInput, uv);

    float4 complement;
    complement.r = color.a - color.r;
    complement.g = color.a - color.g;
    complement.b = color.a - color.b;
    complement.a = color.a;

    return complement;
}

There are a few initial things worth noting here:

The "implicitInput" shader constant is what's in shader constant sampler register S0. That's going to be the "original" composition of the textbox and button over the image, all converted to a "sampler". We'll see in a bit how the association to register s0 happens.
"main" is the entrypoint to the pixel shader. It receives as input the texture coordinate (of type 'float2') of where we're currently outputting in the texture coordinate space of the element being applied to. This varies from 0-1 in u and v. Note again that while HLSL in general provides more flexibility on input types to shaders, for WPF currently, this is always a float2-valued texture coordinate.
Only a pixel shader is being provided here. No vertex shader.

Given these, we now get to the body of the function. The first line "tex2D" samples our input texture at the current sampling position uv, and receives a float4 (a 4-component vector with each value a float) that represents the color. After declaring a local 'complement' float4, we proceed to assign it's R, G, and B values to A minus the corresponding sampled value. The A value (alpha) receives the corresponding sampled value's alpha directly. Finally, we return that value.

[May 1, 2009: My esteemed colleagues Ashraf and Don pointed out that my earlier incarnation of this of (1 - component) was wrong, and it should be (A - component). That's because color RGB has their alpha value premultiplied through already. (1 - component) would work for non-premultiplied. The reason it hasn't come up when I used it was the images I supplied were all opaque (A = 1) images, and in this case (1 - component) == (A - component).]

When the Effect runs, this main() function is executed on every pixel affected. It's run very fast, and quite a bit in parallel, but it is run.

The above isn't the best way to express this HLSL. The thing is that HLSL drives a SIMD vector processor, so rather than have these three separate RGB calculations, we can use HLSL to combine them:

float4 main(float2 uv : TEXCOORD) : COLOR
{
    float4 color = tex2D(implicitInput, uv);

    float4 complement;
    complement.rgb = color.a - color.rgb;
    complement.a = color.a;

    return complement;
}

The syntax "complement.rgb = color.a - color.rgb" will treat the "color.a" as replicated into each element of a 3-vector, and then do the subtraction on each element, all with a single PS 2.0 instruction. (Actually, the HLSL compiler deduces this without me being explicit so the two above shaders generate the same number of instructions, but I never like to count on understanding exactly when compiler optimizations will or won't kick in in super performance sensitive code.)

Compiling the HLSL

WPF Effects do not take the HLSL text directly. You need to compile it into the binary bytecode that DirectX natively accepts, which is how WPF passes it along. To do this, you run fxc.exe on your shader file. Fxc.exe is the shader compiler that comes with the Microsoft DirectX SDK. You can find it at Utilities\Bin\x86\fxc.exe in the SDK.

Say your HLSL was in 'cc.fx', the following command line would generate the compiled bytecode int 'cc.ps' in the same directory:

> fxc /T ps_2_0 /E main /Focc.ps cc.fx

This says to compile to the PS 2.0 profile, and to look for the entrypoint named "main".

(We have a rough prototype that adds the fxc.exe compilation as a MSBuild build task so that it can be incorporated directly into projects without having to break out to a command line shader compiler. When that's further along, we'll post this out there for all to use.)

Writing your managed code Effect subclass

Now it's time to write our managed code that will expose this Effect out to WPF developers. We'll use C# here.

using System;
using System.Windows;
using System.Windows.Media;
using System.Windows.Media.Effects;

namespace MyEffects
{
    public class ColorComplementEffect : ShaderEffect
    {
        public ColorComplementEffect()
        {
            PixelShader = _shader;
            UpdateShaderValue(InputProperty);
        }

        public Brush Input
        {
            get { return (Brush)GetValue(InputProperty); }
            set { SetValue(InputProperty, value); }
        }

        public static readonly DependencyProperty InputProperty =
            ShaderEffect.RegisterPixelShaderSamplerProperty(
                    "Input", 
                    typeof(ColorComplementEffect), 
                    0);

        private static PixelShader _shader =
            new PixelShader() { UriSource = new Uri(@"pack://application:,,,/MyEffects;component/ColorComplementEffect.ps") };

    }
}

Here's basically how this works:

We derive from ShaderEffect, itself a subclass of Effect. Most importantly, ShaderEffect exposes a PixelShader property of type PixelShader.
We define a static PixelShader instance which references the compiled bytecode. It's static because the same PixelShader object can be shared amongst all instances of the Effect.
We define a Brush-valued DependencyProperty called InputProperty, and the corresponding Input CLR property. This is almost identical to how we define other DPs in WPF. The difference is that we use a helper method called ShaderEffect.RegisterPixelShaderSamplerProperty. As in other DP definitions, both the name and the owning type are specified. But the third parameter here (0, in this case) represents the sampler register that the Input property will be able to be accessed from in the shader. Note that this 0 matches the s0 in the HLSL above:
```
sampler2D implicitInput : register(s0);
```

The instance constructor just assigns in the static _shader to the per-instance PixelShader property, and calls UpdateShaderValue() on any DPs that are associated with shader registers. In this case, just the InputProperty. This latter is necessary to ensure it's set for the first time, since DPs don't call their PropertyChangedCallbacks on the setting of their default values.

The other thing worth mentioning here is the gibberish in the "pack://" URI when we reference our pixel shader bytecode file, ColorComponentEffect.ps. Since we don't want to reference a loose file on disk, and we'd like the shader bytecode to live in whatever packaging the C# ColorComponentEffect class goes into (since they should travel together), we use the WPF "pack://" URI syntax. In this case, we're building a library called "MyEffects", which is why that appears in the pack URI.

In order for this "pack://" URI to work, the bytecode needs to make its way into the built component. You do this by adding the shader bytecode file into your project, and ensuring that its Build Action is set to "Resource" as this snip from the Solution Explorer shows:

I have a little helper function that lets me express this URI string without hardwiring in "MyEffects", and without re-generating this same gibberish each time:

new PixelShader() { UriSource = Global.MakePackUri("ColorComplementEffect.ps") }

I include the code for the Global helper class at the bottom of this post.

(Note also that PixelShader objects can also be defined via a Stream, which enables, for instance, authoring tools to construct shader bytecode on the fly and pass it to WPF without ever having to persist it to the file system.)

<Geek-Out-Alert--Somewhat-Esoteric-Stuff-Here> (as if this whole topic isn't geeky enough to start with)

Recall that we considered ColorComplementEffect to be a zero-parameter effect. So what's with this "Input" DP? Note that the Effect is invoked via this XAML:

  <Grid >
    <Grid.Effect>
        <eff:ColorComplementEffect />
    </Grid.Effect>

    ...
  </Grid>

The "sampler" that the shader works on is the rasterization of the Grid itself into a bitmap. You can think of the implementation as creating a VisualBrush of the Grid, then providing that Brush as the value for the Input DP. It doesn't quite work like that, and we expose a new type of Brush called ImplicitInput to represent this usage. ImplicitInput is the default value for DPs that defined using "RegisterPixelShaderSamplerProperty", meaning that they automatically receive the "brush" of the element that they're applied to as their shader sampler (register 0 in this case, since that's what we specified).

We'll see in later posts why exposing these as Brushes, and having the ability to control whether ImplicitInput is used is important. Hint: it has to do with multi-input effects.

</Geek-Out-Alert--Somewhat-Esoteric-Stuff-Here>

OK, that's it for this post... in the next one we'll get into somewhat more complicated Effect definition. In particular, parameterized effects.

Appendix: Global.MakePackUri

This is the MakePackUri helper I referenced above. It's nice in that it's not verbose (like the pack:// URI itself), and, more importantly, it doesn't hardwire in a module name.

internal static class Global
{
    /// <summary>
    /// Helper method for generating a "pack://" URI for a given relative file based on the
    /// assembly that this class is in.
    /// </summary>
    public static Uri MakePackUri(string relativeFile)
    {
        string uriString = "pack://application:,,,/" + AssemblyShortName + ";component/" + relativeFile;
        return new Uri(uriString);
    }

    private static string AssemblyShortName
    {
        get
        {
            if (_assemblyShortName == null)
            {
                Assembly a = typeof(Global).Assembly;

                // Pull out the short name.
                _assemblyShortName = a.ToString().Split(',')[0];
            }

            return _assemblyShortName;
        }
    }

    private static string _assemblyShortName;
}