WPF3D Team Blog : Materials

WPF3D Lighting and Shading

wpf3d — Tue, 30 Jun 2009 04:54:00 GMT

We use the standard fixed-function Blinn-Phong model. You can read up on all of the equations here at MSDN.

If you have a Tier 2 card, we actually do our lighting in a vertex shader. If you don’t have a Tier 2 card, we do the lighting on the CPU. Why not just use D3D9’s fixed-function APIs you might ask? We wanted to be able to blend a texture (SpecularMaterial.Brush) only with the specular component and it turns out the D3D9 fixed-function APIs can’t do that.

An important thing to know is all Material Brushes are treated as textures even though they might not be implemented as such (e.g. SolidColorBrush). This means if the amount of light hitting a vertex is more than 1.0, it will be capped at 1.0 before being blended with the texture. This is an unfortunate behavior of the fixed function pipeline. I discussed this more in the color knobs post.

There is one thing that may be unique to out lighting model. Since we are a retained system, we factor in a Light’s transform to all of its properties. We approximate the scale being applied to the light by taking the cube root of the absolute value of the determinant of the upper 3x3 part of the Light’s transform. For a uniform scale this will be the actual scale but for non-uniform it’s good enough (the real solution is too costly for something uncommon). Internally, this number is factored into the range and attenuation values so things make sense. For example, if the Light undergoes a uniform scale of 2.0, we’ll multiple its range by 2.0.

-- Jordan

Transparent DiffuseMaterials and Depth Sorting

wpf3d — Fri, 19 Jun 2009 04:28:00 GMT

I hinted at this a long time ago and then forgot to follow up, whoops!

As the old post says, DiffuseMaterial writes to the depth buffer. This means if you draw one diffuse model and then draw another diffuse model behind it, the card knows not to draw the second one on top of the first because the first is closer. The depth buffer only knows about position and has no concept of transparency so if the first model is transparent, you won’t see the second model behind it because the depth buffer rejected it as being behind the first.

What does this mean? This means you have to sort your transparent DiffuseMaterials from back to front. If your scene is dynamic, you’ll have to do this any time the relationship between the transparent models and the camera changes. Peter gives an example of how to do this here.

-- Jordan

P.S. If you have a lot of overlapping models (a.k.a overdraw) you can sort your opaque models from front to back. This will prevent all of the obstructed models from being rasterized and will improve performance if rasterization is the bottleneck. If you have both opaque and transparent models, first draw the opaque ones then the transparent ones.

Cel Shading

wpf3d — Sat, 08 Sep 2007 07:02:00 GMT

Charles Petzold has been experimenting with cel shading on his blog at the request of Chris Cavanagh (whom has updated his 3D physics XBAP btw). Though we do use shaders internally, WPF3D's API is fixed function so you have to dig out the ol' fixed function playbook to achieve fancier effects. The plays usually boil down to abusing textures :)

First, make a small, one dimensional texture containing the colors you want. Here are a couple of examples I've zoomed in on and highlighted the pixels to make things clearer. Why I'm using grayscale will make sense in a minute:

Next, we need a function that maps to [0, 1] and takes into account the light position relative to a vertex. How about the cosine of the angle to the light? Turns out we can compute that fairly quickly because it's equal to the dot product of the normal and the vector to the light divided by their lengths.

    Int32Collection idxs = mesh.TriangleIndices;
    Point3DCollection pts = mesh.Positions;
    Vector3DCollection nrms = mesh.Normals;
    PointCollection txs = mesh.TextureCoordinates;
    mesh.TextureCoordinates = null;
    for (int i = 0, count = idxs.Count; i < count; ++i)
    { 
        int idx = idxs[i]; 
        Vector3D toLight = lightPos - pts[idx]; 
        toLight.Normalize(); 

        // The normals are pre-normalized, no need to do it again  
        double dp = Vector3D.DotProduct(toLight, nrms[idx]); 
        
        // A negative dot product means the vertex is facing away  
        // from the light so let's set that to the darkest color  
        if (dp < 0) 
        { 
            dp = 0; 
        } 
        
        txs[idx] = new Point(dp, 0); 
    } 
    mesh.TextureCoordinates = txs;

Naturally, you're going to need to redo this calculation any time the relationship between the light and the mesh changes so it needs to be fast. This is why I go through all of that collection nonsense (read this).

So why did I use a grayscale texture? If I baked the color in, I'd have to make a new texture any time I wanted to change the color. Instead, I can use the color knobs to set the color and use the grayscale texture like a mask. In these photos, I have a single AmbientLight with a DiffuseMaterial with AmbientColor="green" and Brush equal to the one dimensional texture. I don't actually have a real point light in the scene which helps performance.

Being that it's per-vertex, more vertices means a higher quality image and you're going to see some popping when it's animated. The teapot doesn't have too many vertices so the spout and black line on the body don't look great. On the highly tesselated torus, those issues are gone but it's still hard to get a clean, crisp edge because of the linear interpolation of the texture.

-- Jordan

P.S. If you use a texture with two colors and instead dot the normal with the direction to the camera, you'll get silhouette edges but they could look really bad because of the per-vertex nature again.

Material Behavior

wpf3d — Fri, 09 Mar 2007 03:37:00 GMT

What Material you choose will impact your GeometryModel3D in three ways:

Lighting model
Depth write
Blend with the back buffer

The first one is straightforward since it's right in the name of the Material plus it's documented. The last two aren't obvious at all and this is probably the first you've heard of them.

Type	Depth Write	Blend
DiffuseMaterial	Yes	SrcOver: S_αS+(1-S_α)D
"AmbientMaterial"	Yes	SrcOver: S_αS+(1-S_α)D
EmissiveMaterial	No	Additive: S_αS+D
SpecularMaterial	No	Additive: S_αS+D

Depth write and the depth buffer

If you haven't heard of the depth buffer (a.k.a. the z buffer) before, it's essentially how the card keeps track of the distance to an object from the camera per pixel. When a polygon is being rasterized at a pixel that has already been written to, do you keep the old value or take the new value? For opaque objects, you typically want the one closest to the camera to win because you won't be able to see the occluded one. This is what we tell the card to do (for you D3D programmers out there, we use D3DCMP_LESSEQUAL).

For example, say you draw a triangle with a DiffuseMaterial. If you then draw another triangle directly behind, it will be ignored in the area of overlap because the first triangle is in front of it. But if a polygon doesn't write to the depth buffer, then it's like the card doesn't know it's there. Say you draw a triangle with an EmissiveMaterial. If you then draw an overlapping triangle it doesn't matter if it's behind or in front -- the new triangle overwrites the old one because the card has no "record" of the old one.

Can you think of why the depth buffer might cause problems with transparent objects in a dynamic scene? That's a topic for another post :)

Blend

In the equation above, S means source (the GeometryModel3D you're drawing) and D means destination (what you've drawn to the back buffer already). From the equations you can make a few observations like a fully opaque DiffuseMaterial will never let you see what was previously written or even opaque additive Materials will appear transparent because they don't block out what is already there.

Why are there different behaviors?

The behaviors came out of how we think most people use Materials. Just about every Model3D will contain a DiffuseMaterial and will want to write to the depth buffer. If you then want your Model3D to glow or shine, you throw in an EmissiveMaterial or SpecularMaterial. Since you already wrote the depth, you don't need to do it again in the emissive and specular passes. Furthermore, you don't want emissive and specular to subtract away from the diffuse layer you just put down.

Extra Tips

"AmbientMaterial" is in quotes because it doesn't exist but you can make a Material that only responds to ambient light. See my earlier color knobs post.

If you'd like one of the additive Materials to write depth, you can do so by putting them in a MaterialGroup along with a transparent black DiffuseMaterial. That way the DiffuseMaterial doesn't affect the color but does affect the depth buffer. You can put this "depth pass" DiffuseMaterial before or after the other Materials but you will get different results between the two if you have any overlapping front faces (or back faces if we're talking about BackMaterial).

-- Jordan

Interacting with 2D on 3D in WPF

wpf3d — Wed, 13 Dec 2006 06:19:00 GMT

Interacting with 2D placed on 3D is now possible in v1 of the Windows Presentation Foundation!

In between shipping Vista and planning the next version of WPF, we realized that with a clever implementation it was possible to provide this feature today on v1 bits, and I’m happy to report that as of now, the binaries (and source code to them) needed to do this are available for download (here).

The download also contains two sample applications - InteractiveViewport3DSample and Channel9Demo - which illustrate how to use the code. So if you just want to dive right in to 2D on 3D, that's a good place to start.

The rest of this post will describe how to use this feature as a developer, as well as how it’s implemented behind the scenes. If you want the video version though, which includes a cool demo of an app we made with the 2D on 3D code (photo shown below – source here http://www.codeplex.com/3DTools/Release/ProjectReleases.aspx?ReleaseId=2058), check it out on Channel9 (here).

How do you use it?

There are two main classes needed to add 2D on 3D to you app: Interactive3DDecorator and InteractiveVisual3D. The Interactive3DDecorator handles the majority of the work that allows the 2D on 3D interaction to happen. To use it, you simply place it around the Viewport3D that you intend to make interactive. So in XAML, you have:

<local:Interactive3DDecorator>
    <Viewport3D>
        …
    </Viewport3D>
</local:Interactive3DDecorator>

Then, to actually add interactive 2D on 3D objects in to your scene, you use the InteractiveVisual3D class. InteractiveVisual3D is a subclass of ModelVisual3D and provides the following dependency properties which set up the interactive 2D on 3D:

Geometry - The Geometry3D that is to become the content of the InteractiveVisual3D .
Visual – The 2D you want to interact with on the 3D geometry.
Material – A user specified material to use on the 3D object. By default, a DiffuseMaterial is used.
IsBackVisible – Indicates whether the material used for the front face should also be mirrored on the back face.

The first two properties, Geometry and Visual, are the primary ones needed. They allow the geometry that should be used to bet set, as well as the Visual that should appear on that geometry. For example, the below XAML code demonstrates how to create an InteractiveVisual3D.

<local:InteractiveVisual3D Geometry="{StaticResource PlaneMesh}">
    <local:InteractiveVisual3D.Visual>
        <StackPanel>
            <Label Content="Sample UI" />
            <Button Content="Close Window"/>
            <TextBox />
        </StackPanel>
    </local:InteractiveVisual3D.Visual>
</local:InteractiveVisual3D>

The other properties allow for fine tuning of the object’s appearance. The Material property allows a user to create their own material for the object. To do so, a user can create a material as always, and then specify using the IsInteractiveMaterial attached property which material is intended to be “interactive” (i.e. the VisualBrush created using the passed in Visual is set as the Brush for that material). Note, this attached property must be set to true on at least one material – otherwise an exception will be thrown. If you ever want to disable interaction, you can set “IsHitTestVisible” on the Visual to be false. As an example of setting a custom material, the following code sets the material to be composed of a DiffuseMaterial, which will contain the visual brush, and a SpecularMaterial.

<local:InteractiveVisual3D.Material>
    <MaterialGroup>
        <DiffuseMaterial local:InteractiveVisual3D.IsInteractiveMaterial="True"/>
        <SpecularMaterial Brush="Red" />
    </MaterialGroup>
</local:InteractiveVisual3D.Material>

Finally, IsBackVisible sets the material used on the front face to also be used on the back face. Currently, the Interactive3DDecorator does not distinguish between front and back faces, so this allows for there to be interaction with the back face as well.

How does it work?

At a very high level, the interaction with 2D on 3D is achieved by really interacting with a hidden version of that 2D content in 2D. The 2D is positioned such that the point in 3D the mouse is over is the exact same point as the mouse is over on the hidden 2D version. Then, when the user clicks, etc… they are interacting with exactly the same location. If you want to see this for yourself, you can set the “Debug” property on Interactive3DDecorator to true, which makes the hidden layer partially visible.

The Interactive3DDecorator then consists of two elements: the 3D content that is displayed within it and a hidden layer that is used to position and display the 2D content that is being interacted with. Depending on what 2D content on 3D is being interacted with, the hidden layer changes to hold that 2D content. The images below give a visual representation of what is going on. The first photo is just of the 2D content we wish to place on 3D. The second is that 2D content on 3D. And finally, the third image shows how the 2D on 3D interaction takes place. When the mouse moves over the “B” in the button, the hidden layer (shown to be only partially transparent for this example) is moved such that the mouse is over the same point in the hidden layer as it is in the 3D scene.

To figure out where to position the hidden layer works as follows. When the mouse moves in the 3D scene, a ray is shot in to the 3D scene to see if it intersects any object. If it hit an object, and it is an InteractiveVisual3D, we can use the return parameters from the intersection to compute the texture coordinate that was hit. Then from these, we can map from the (u,v) value of the texture coordinate, on to an x,y point on the 2D visual, which is the point we need to place under the mouse. More specifically, the code assumes texture coordinates are all in the range (0,0) to (1,1) – i.e. upper left to lower right of the image (for those planning to use this, this is important to know, since your texture coordinates need to be within this range to enable interaction with the 2D content). Then the point on the 2D object that was hit is simply (u * Width, v * Height).

One of the great things about this method then is that the only event that needs to be tracked is when the mouse moves. There’s no need for catching and forwarding on every type of event: the hidden layer takes care of all of this work.

Positioning With Capture

There’s one very interesting “gotcha” though: what happens when one of the 2D objects grabs capture and then you move off the 3D mesh it is on? For example, you select some text, and then move the mouse above that selected text, or click and hold on a button, and then move away from it. Correct hidden content positioning becomes more complicated in this case. The problem becomes difficult for many reasons. In the normal 2D situation, both the mouse position and the 2D content exist in the same plane. The transformation that is applied to the 2D content can be used to transform the mouse position to the content’s local coordinate system. However in 3D, due to the projection of 3D on to a 2D plane, the mouse’s position actually corresponds to a line in 3D space. In addition, the element with capture could also be mapped to any arbitrary geometry. When the mouse is over the 3D object, hit testing tells us where it is relative to the 2D visual. When it is off the 3D object, due to the above issues, there is no longer a straight forward answer to this question: the 2D point corresponds to a 3D line and the 2D content could be on arbitrary 3D geometry. Also, because the element has capture, it wants to receive all events. Before, we only needed to be sure that the mouse was over the correct object at all times. Now we need to position the hidden visual such that it is in the proper position relative to the object that has capture.

To solve this issue, the code takes the following approach: The overall idea is to reduce the 3D problem back to 2D. In the normal 2D case, the transformations applied to the content can be used to convert the mouse position to the content’s local coordinate system. This transformed position then lets the content know where the mouse is relative to it. In 3D, due to the many orientations of the geometry and texture coordinate layouts, it’s difficult to say where a 3D point is in the relative coordinate system of the 2D content on 3D. To approximate this, the outline of the 2D content on 3D, after it has been projected to screen space, is computed, and then the mouse is positioned based on this projection.

For example, take the 3 images below. The first is the 2D content on 3D. In the second image, text is selected, and the mouse is moved to a point off the object. The third image is the outline of the text box (the object that has capture). This outline is then used to position the hidden visual.

After the outline is available, The closest point on this outline to the mouse position is computed, and then this point on the outline is considered what was “hit” and it is placed under the moue position (hence, highlighting up to the “T” in the middle image below). Since we place the mouse by the closest edge point, the interaction tends to behave as it would in 2D, since we position the hidden content based on what the mouse is closest to on the 2D content on 3D. By placing it at the closest edge point, we are explicitly stating about where we expect the mouse to be relative to the 2D on 3D element’s orientation.

This method helps provide an intuitive response to the interaction, since the interaction happens with the closest point on the object with capture to the mouse.

Enjoy using it! We’ve had a lot of fun building this, and can’t wait to see what kind of incredible applications will be built with it!

-Kurt

Material Color Knobs

wpf3d — Fri, 08 Dec 2006 23:40:00 GMT

We affectionately refer to DiffuseMaterial.Color, DiffuseMaterial.AmbientColor, SpecularMaterial.Color, and EmissiveMaterial.Color as the "color knobs." The color knobs are probably the least understood WPF3D feature because they were one of the last (if not the last) features we added and most of the things you can do with them are not obvious. Hopefully this article will give you insight into how the properties work and help you write better XAML exporters.

The full lighting equation used by WPF is definitely something we need to cover later, but essentially the final color for a single Material is going to be: Color * lighting result * Brush. DiffuseMaterial includes both ambient and diffuse lighting so it has two color properties making the equation: (AmbientColor * ambient lighting result + Color * diffuse lighting result) * Brush. From this equation, you can see that you can't just set Color and be done -- you also need a Brush. By default, all of the Color properties are white giving them no effect on the final appearance.

Now let's look at a few ways to use these properties. (By the way, if I refer to a color channel going from 0.0 -> 1.0 it's the same as 0 -> 255 if you're using bytes)

Color Knobs as a Filter

Let's say a GeometryModel3D with a DiffuseMaterial is being hit by a light containing red. If you set the red channel of the color knob to 0, the mesh won't reflect any red light. If you set it to 1, it'll reflect all of the red light the Brush reflects. By changing the value between 0 and 1 you can "dial in" how much you want to reflect back. This is how we started calling them "knobs."

The color knobs are full colors so they do have alpha. Setting the alpha value does what you would expect (e.g. alpha of 0.5 makes it half transparent). There is one exception to things and, surprise, it's DiffuseMaterial. Only DiffuseMaterial.Color.A affects the transparency of the Model. Turning down DiffuseMaterial.AmbientColor.A does make the ambient component transparent and increase the impact of the diffuse component, but it does not affect Model transparency.

AmbientMaterial

DiffuseMaterial, SpecularMaterial, EmissiveMaterial... where's AmbientMaterial? We don't have it. What do you if you want a Model to respond to only ambient light? Before color knobs, it couldn't be done. With color knobs, create a DiffuseMaterial, set Color to (0,0,0,0), and all diffuse contribution will be wiped out leaving you with just ambient!

Fully Illuminated Solid Colors

Say you wanted to make a Model look green. You have two good options: a white brush with a green color knob or a green brush with a white (the default value) color knob. Is there a difference between the two? It depends on how much light is hitting the Model. If the amount of light hitting isn't greater than 1.0 in each channel, then they're equivalent. If not, here's what happens:

Both pictures are of a sphere mesh with a DiffuseMaterial and two full power point lights shining on it. On the left, the the Color is set to #ff006000 and the Brush is set to white. On the right, the Color is set to white and the Brush is a #ff006000 SolidColorBrush. Why is the one on the left so vibrant and the one on the right so lackluster?

In the graphics pipeline, lighting is calculated in the vertex processing stage and textures are sampled later on in pixel processing. In WPF, all Brushes applied to 3D are done as textures -- even SolidColorBrushes. Unfortunately, the graphics card clamps colors coming out of the vertex stage to 1 so no matter how many hundreds of lights you have you will never be able to exceed the color of the SolidColorBrush. As you can see in the image on the right, the big solid color splotch in the middle has a value of #ff006000, the color of the SCB. The color knob value is incorporated during vertex processing so it is multiplied by the unclamped lighting result. For those of you used to per-vertex colors in other APIs, you can think of the color knob as a single vertex color for the entire mesh. This explanation is tricky and requires some knowledge of graphics hardware but hopefully it made some sense :)

In general, you may end up wanting to use the color knobs for all of your solid color needs*.

-- Jordan

* There is one minor perf issue to take note of if you don't have a Tier 2 graphics card: If you have a MaterialGroup with more than one DiffuseMaterial or more than one SpecularMaterial and those duplicate Materials have different color knob values, you will incur a performance penalty. For example, a MaterialGroup with two DiffuseMaterials one with Color (.5, .5, .5) and one with Color (1, 1, 1). Fortunately, this doesn't happen too often. See this MSDN article for information about tiering.