Presentation Source

Mitigating Airspace Issues In WPF Applications

Dwayne Need [MSFT] — Tue, 26 Feb 2013 22:58:09 GMT

Introduction

WPF offers a modern approach for building Windows applications, but it builds directly on Win32 – the traditional UI infrastructure in Windows. Because Win32 was developed in an era where CPU/GPU horsepower was much more limited than it is today, it utilizes a number of optimizations for rendering. These include using the “reverse painters algorithm”, tracking invalid regions for minimal updates, copying pixels around the screen to avoid unnecessary repainting, extensive clipping to avoid overdraw, limited support for alpha blending, and other such restrictions. Most of these optimizations were built for GDI, the standard rendering technology in Windows for many years. DirectX is an entirely different rendering technology, originally intended for PC gaming on Windows, but recently finding more of a presence in desktop applications (consider IE9, for example). DirectX provides access to the power of modern GPUs, and WPF embraced this technology – rather than GDI - in order to build the kind of rich composition framework we envisioned. As a consequence, WPF developed its own transformation, clipping, and composition implementation which is completely different than what Win32 implements.

In Win32, a “window” is the basic unit of user-interface composition. Programmatically, a window is referenced by its handle, or HWND. Windows come in two major flavors: top-level windows and child windows. Top-level windows are what you normally interact with, they float on the desktop and usually have the standard minimize/maximize/restore/close buttons, a title bar, etc. This is the primary scenario for WPF; where it is the exclusive content of the client area of a top-level window, which is what you get when you create a WPF Window object. On the other hand, child windows are a unit of composition for user-interface components within a top-level window. Child windows can be incredibly useful; they can run on different threads (or even processes), they can have their own state, they can respond to special messages, etc. A number of technologies build on child windows, such as ActiveX, so incorporating them into your WPF application is a common requirement. To properly include a child window in a WPF element tree you must derive from HwndHost and implement the logic to construct and destroy the child window, as well as hooking up keyboard processing and handling any special messages. While WPF does not use child windows for most of its user-interface elements because we have our own element tree, there are some exceptions; the WebBrowser control is probably the most widely known example of a WPF control that is actually a hosted child window. WPF also includes support for hosting Windows Forms components, which are implemented with child windows.

You can even embed WPF content within a child window by directly creating an HwndSource instance. This is a powerful technique that we will be exploring more in this article, and it is often used in scenarios where WPF is used to develop a plugin for another application. A note of caution: child windows have a number of exotic settings that WPF does not support well; such as tinkering with child/sibling clips or composited painting.

The kind of rich user-interface presentation that is enabled by WPF is possible in large part by the way we uniformly compose the element tree. Composition is what allows WPF to render elements in interesting ways: clipped against opacity masks, distorted via transforms, blended with effects like transparency, processed through pixel shaders, redirected through brushes, projecting onto 3D objects, etc. But due to the limitations of how child windows were designed to work, composing them is notoriously difficult. Indeed, when WPF hosts a child window, we skip the composition all together and simply position it over its layout slot and then let it do its own thing. As you might imagine, without composition many problems can be observed: missing content, incorrect clipping, lack of transparency, z-order issues, etc. We refer to these issues as “airspace” problems – a term coined by Nick Kramer.

In some instances, you may also encounter strange rendering artifacts on the screen; particularly when moving a child window around. These rendering artifacts are often due to conflicts between GDI and DirectX, along with some limitations or bugs in the DWM.

This article discusses in some detail all of these issues, and explores some ways of mitigating them.

Airspace Problems

The most common complaint related to airspace problems we hear about concerns clipping. To be visible on screen, a WPF element tree is always associated with some window; typically a top-level window but it could also be a child window. WPF composes the element tree and renders to that window. Child windows always render on top of, and are always clipped to the bounds of, their parent window. Recall that when hosting a child window within an element tree, all WPF will do is position the child window over its layout slot. It is important to realize the WPF is not rendering the child window. The child window is rendering itself – or, more accurately, the child window is painting to a GDI device context associated with its window without coordinating through WPF. So even though an HwndHost appears to be an element that can be interleaved with other elements in the element tree, the hosted child window is actually placed on top of anything WPF renders. This results in predictable problems: the child window is not clipped by containers like scroll-viewers, and nothing can render over them, not even adorners.

In fact, there is no good way for WPF to render over of a child window. Some people have experimented with relaxing the clipping behavior of Win32, but this causes much worse problems. It is better to play nicely with Win32 clipping rules, which is why WPF enforces WS_CLIPSIBLINGS and WS_CLIPCHILDREN.

Since a child window is painting itself, and is not rendered by WPF, it should be clear that WPF doesn’t have access to the contents of the window. For example, if you use a VisualBrush to render an element tree, the contents of any hosted child window will simply be missing. Another example is that you can’t apply a ShaderEffect to a hosted child window because the contents of the window are not actually available to WPF for processing.

Another common symptom people notice is that even simple things like transparency don’t work either. This is because the Win32 painting model doesn’t generally support transparency, so the child window that is placed on top of the WPF content, has no way to paint itself with partially transparent pixels. Using styles like WS_EX_TRANSPARENT don’t work for reasons that will be discussed later.

A final nuance is that a hosted child window is created when the HwndHost element detects that it is plugged into an HwndSource. If an HwndHost element is removed from one element tree and then added to another, the hosted child window will be reparented to be within the destination HwndSource parent window. As a consequence, the z-order is reset to the bottom. In addition, while setting the ZIndex property will cause WPF panels to render their children in a different order, the HwndHost will not adjust the corresponding z-order of its hosted child window. Not that it would matter relative to the WPF content it is floating over, but the z-order gets out of sync even relative to other hosted windows.

Render Thread Considerations

WPF uses a somewhat novel technique of splitting the user-interface responsibilities between a traditional user-interface thread and a dedicated render thread. The user interface thread (or threads, as there can be more than one) is responsible for processing user input and updating the element tree. The render thread is responsible for rasterizing the vector graphics described by the element tree and updating the display. This design even allows the render thread to be in a separate process, even a process on on another machine. This was the infrastructure that allowed WPF to support high-fidelity zooming on Vista that preserved the sharp vector graphics, and the efficient performance over remote-desktop connections that really improved the experience compared to bitmap-based remoting. Unfortunately, this design required the various OS remoting components to be updated in sync with the WPF releases, and this eventually proved untenable and the features were disabled in WPF 3.5 SP1.

Another goal of this design was to allow a degree of independence for rendering tasks so that they wouldn’t be blocked by stalls on the user-interface thread. The obvious scenario is “independent animations” that the user interface could start and then forget about, and they would animate on the render thread, free from any glitches or stuttering due to processing on the user interface thread. This would be ideal for transitions, which typically involve a lot of work on the user interface thread to load in new content. Unfortunately, WPF never got to complete the feature work required for independent animations. In fact, the only real benefit from the separate render thread is a certain amount of parallelism and that the MediaElement can render frames independent of the user interface thread once playback is started. More recent incarnations of XAML platforms have better support for independent animations, such as Silverlight 5.

Unfortunately, this design has its own problems too. The render data needs to be duplicated for both the user-interface and render threads, and updates have to be sent and received, and object lifetimes have to be carefully coordinated. This increases both the CPU and memory costs associated with heavy graphics content. WPF has implemented a number of optimizations for efficient partial updates, but there is still significant room for improvement. Another problem is that there are some situations where the user-interface thread is expected to synchronously complete a rendering operation. Two prominent examples are PrintWindow and bottom-to-top painting. PrintWindow redirects the device context for the window and sends a WM_PAINT to the window, the window is expected to paint its contents to the redirected device context, and when the window proc returns from handling WM_PAINT, the window is restored to its original state. PrintWindow is unreliable when used on a window with WPF content because the user-interface thread does not actually paint, so the content is missing. Bottom-to-top painting is configured by specifying the WS_EX_TRANSPARENT style on a child window. When this style is present, Win32 will paint the siblings beneath the window first, and then paint the window. This allows the child window to selectively let that content show through, though it can be tricky to implement well. But because WPF doesn’t paint of the user-interface thread, the contents of our window are not reliably available for any sibling over us to use. There is also the unusual WS_EX_COMPOSITED style that doesn’t work with WPF content either.

Finally, there are times when the user-interface thread needs to copy bits from the screen and this is problematic with WPF. An example is when moving a child window without the SWP_NOCOPYBITS flag (note the double negative; this means moving a window with copying bits). To avoid unnecessarily repainting a window when it moves, Win32 will copy the existing content of the window from its old location to its new location, and only repaint what is left behind. But since the old contents that GDI is trying to copy might be updated at anytime by the separate render thread via DirectX, the necessary coordination is missing, and sometimes garbage is copied. Another example is when a top-level window over a WPF window has the CS_SAVEBITS style. With this style, Win32 will copy the bits before it displays the window and then put the bits back when the window is hidden. Menus and message boxes are the typical windows with this style. As Raymond Chen discusses, Win32 tries to detect when something makes the saved bits invalid and will fall back to just repainting the background. But WPF’s separate rendering thread using DirectX can go undetected, so Win32 may put back the old contents.

The splitting of user-interface processing and rendering across separate threads is unusual in Win32 applications, and doubtless the source of many subtle bugs. If integrating child windows into your WPF application is a primary concern, I strongly recommend disabling WPF hardware acceleration. By disabling hardware acceleration, WPF will render using GDI. This eliminates many issues because GDI can coordinate correctly even from multiple threads. If disabling hardware acceleration is not an option, we will review some mitigations that work around most of the rendering artifacts I know of. Unfortunately these mitigations are not always trivial.

DWM Considerations

Originally, painting operations for a window were clipped to the visible region of that window and then allowed to update the video memory of the screen directly. This was very efficient, but rules out such effects as transparency and can lead to lagging visual artifacts as updates to the screen were gated by the responsiveness of applications.

Windows 2000/XP introduced layered windows via the WS_EX_LAYERED style. For layered windows, the operating system retains a bitmap of the contents of the window and can compose it on the screen as needed, without waiting for the application to respond, and can apply effects like transparency. The contents of this bitmap can either be provided by the application directly, or can be filled by “redirecting” the traditional painting operations for the window. SetLayeredWindowAttributes controls the appearance of windows with redirected content; but it is restricted to simple effects like constant transparency or a color key. Applications that can provide their own bitmap, rather than using redirection, can use UpdateLayeredWindow instead. This API can accept a bitmap with a per-pixel alpha channel, so this is the API that WPF uses when you set the Window.AllowsTransparency property to true. Note that this kind of layered window only displays the contents of the bitmap that the application explicitly provides; nothing drawn via GDI or DirectX to a device context for this window or any child windows is shown. This is the root cause of the problem where the WPF WebBrowser control cannot be displayed in a layered window.

Redirected layered windows provide “output redirection”; where the operating system redirects the output of rendering operations normally destined for the screen into a bitmap; which allows it the flexibility to compose the bitmap in various ways. However, if the composition is non-trivial, the operating system must also provide some kind of “input redirection” where messages and APIs related to the input coming into an application accounts for the composition effects. It is also important that APIs related to location, size, and coordinate transforms also account for the composition. So far, the operating system doesn’t compose layered windows in a way that would require input redirection.

Windows Vista introduced the Desktop Window Manager which redirects all top-level windows (that aren’t already layered) and then composes the contents of all of the windows onto the desktop. DWM is capable of composing the window contents in more interesting ways than XP was able to; the Flip3D feature is probably the most dramatic - but note that it disables input to the windows and so it does not need to worry about input redirection. The DWM can also scale a window to accommodate higher DPI settings; but this is a simple scale transform and required only modest updates to input processing coordinate transform APIs. The redirection services of the DWM remained limited to top-level windows.

Difficulties are encountered on Vista if both GDI and DirectX are rendering to the same window. Vista handles this scenario in the DWM by maintaining separate surfaces for the GDI and DirectX content. This mode is called “Vista Blt”, and the DWM chooses what to which to display on the screen based on a heuristic as to which rendered last. The DWM will present from one or the other, or it may even display some content from both. Windows 7 introduced a better mode called “Device Bitmaps”, which allow both GDI and DirectX to be captured into the same backing surface. Unfortunately, Device Bitmaps are not compatible with all APIs and will be disabled if those APIs are called (the primary example is DwmGetCompositionTimingInfo), and it has its own problems.

Windows 8 introduces Direct Composition, which extends the redirection and composition services of the DWM to child windows. This is a major development, and finally enables child windows to be layered. Unfortunately, Direct Composition still uses the DWM as an external compositor, rather than being intimately integrated with the application. In other words, you configure what you want and the DWM composes it for you. This precludes certain kinds of composition effects, such as texturing a child window around a 3D model within your application or processing it through a pixel shader. It is also worth mentioning that even though Direct Composition allows for composition effects that tear away from the input and coordinate APIs, there is still no support for input redirection.

In the past you could disable the DWM if it didn’t support a particular scenario very well; though that was rarely a good idea. However, starting with Windows 8, the DWM can no longer be disabled.

Redirection

As perhaps hinted at above, the ideal solution to these problems would be to let application frameworks like WPF compose the contents of child windows themselves. This kind of “internal” compositor would enable far richer composition effects. However, “input redirection” must also be provided for any such composition system to be used for more than passive “transitions”.

When developing WPF 4.5, we decided to take on the challenge of implementing a complete redirection solution so that WPF applications could compose the contents of child windows as naturally as any other kind of content. We explored many techniques, but finally settled on using some form of API interception to shim the problematic GDI, DirectX, and Win32 APIs. We settled on Detours, because it is a very robust and complete interception library developed here at Microsoft. It is important to note that this is an in-process library, nothing we did had impact outside of the WPF process that opted into this new feature.

Output redirection for DirectX was pretty straight forward. DirectX is a great API in this regard because all rendering operations effectively occur off-screen until a final swap-chain (or device) “present” is invoked. Our basic technique was to intercept this present call and implement it by copying the contents to another video surface; which we would then display in WPF via a D3DImage. We did, however, have to support Direct3D 9, 10, 11, and potentially future versions too. This was only tenable because the DirectX team has had the foresight to refactor modern Direct3D APIs on top of DXGI and this is where the important present API exists. But even with this refactoring, our code has to create intermediate surfaces and such which require a specific Direct3D version. Again, the DirectX team came through with the ability of DirectX 11 to emulate the device state of previous versions. The remaining complexity was in handling all possible surface formats and usage restrictions. We also had a hard time handling D3DSWAPEFFECT_FLIPEX, and eventually just stripped that flag when creating swap chains.

Output redirection for GDI was more involved. GDI has a large API surface area, and GDI does not have the equivalent of “present”. Instead, GDI rendering APIs take effect whenever the driver decides to flush the command stream. We ended up intercepting 200+ GDI functions, and replaying them to an offscreen DC. The vast majority of these interception shims were mechanically generated by a tool based on whether they read state, set state, or actually rendered to the device context. The hardest challenge was dealing with clipping. We needed to mirror some (but not all) of the clipping state from the real device context onto our redirected device context. Some of the clipping state is set deep inside the operating system, so detecting when the clips were changed was beyond the reach of our interception library. Other clipping state needed to be excluded, such as when a child window is partially outside the bounds of its parent, but composed such that those regions are visible. In such cases we have to somehow get WM_PAINT and related messages to still be generated. Clipping too much is clearly a problem, but so is clipping too little. We eventually developed our own extensive clipping model for redirected windows that tried to emulate what Win32 did internally, but this was a continuous source of bugs.

Input redirection posed another difficult challenge. Any input device that depends on the location of elements to direct the input stream needs to take the composition effects into account. WPF has already been plumbed for this, of course, since our elements have always been fully composed our input events are routed appropriately. But Win32 stubbornly considers windows to be rectangles that are rigidly aligned with the horizontal and vertical axes. This restriction is baked deep into the kernel where the window tree data structures are kept. The kernel takes raw input from devices like the mouse and performs an initial hit test to decide where the input should go. Since our solution was in-proc only, we have to retain the restriction that our top-level window obeys these rules. (Note that with layered windows, WPF can render any shape, and this allows us the apparent freedom to do things like rotate our top-level window. To see this is action, rotate a ComboBox and the drop down the list.)

The kernel will deliver “pointer” input to the thread queue that owns the window (including child windows) it thinks is under the mouse. The thread that owns the queue will eventually pick up that input and perform the hit-test processing again and then deliver the appropriate messages, update state, etc. We have to play by those rules, but still find some way to appropriately handle composited child windows. We explored two basic approaches: prevent the kernel from finding any composited child windows, or make sure the kernel finds the correct composited window.

Child windows can be hidden from the kernel’s input processing in a variety of ways, but specifying the WS_EX_TRANSPARENT style or disabling the window is the easiest. The kernel will find the top-level window and deliver the input to that queue. From there we could perform our own hit testing to find the appropriate child window, and generate our own window messages to send it. The hardest challenge with this technique is sending all of the correct messages, with the correct payload, in the correct order, at the correct times, and somehow updating all of the correct global state. By the time you are done, you would have effectively reverse engineered a significant portion of Win32! Further, since child windows can be running in other threads and processes, we would have to synchronize messages and state across these boundaries. We soon decided that this would be too difficult.

Our other option was to ensure that the kernel finds the child window that we know is the right one. This technique is to eagerly perform a hit-test ourselves, and then move the appropriate window under the mouse position so that the kernel will find it. All of the messages and state are then handled normally by Win32, including across thread and process boundaries. For example, say that we hit-test the mouse position through the element tree - taking into account all composition effects - and determine that the lower-right corner of a child window is what is found. We would then align the entire window hierarchy so that the appropriate child window’s lower-right corner is what the kernel will find when it hit-tests the mouse coordinate. We needed to do this alignment before the kernel does, so we chose a mouse hook. We started with a low-level mouse hook, because those are pretty easy and they are invoked early enough. But low-level mouse hooks will block the kernel until the application responds. So if your app is busy, mouse processing for everyone is stalled. A common example of this for developers, is when you are debugging your application, the mouse would stop working! That is unacceptable, so we changed to using a regular mouse hook. This kind of mouse hook is invoked before the thread further handles the mouse input that the kernel gave it. It turns out this this is sufficient, because Win32 will correctly adapt to changes in the window tree between when the kernel posted the input to the queue and when the queue was processed by the thread. So as long as the kernel finds some window belonging to our thread so it can deliver the input to our queue for further processing. Now an unresponsive application will only stall input processing for itself. There are lots of details to sort out, but this technique worked pretty well. The major restriction of this technique is that we cannot support any kind of multi-touch, because a window cannot be in two places at the same time. We decided to accept this limitation for composing child windows, and hope that new multi-touch controls would be written in pure WPF.

Layered windows (recall that WPF uses the type of layered window where we provide the bitmap of the contents ourselves) remained a problem. Even though we could redirect the output from child windows within, the OS refused to deliver paint messages to them, so they would simply never bother to paint. Content like videos would work fine because they typically initiate their own painting instead of waiting for a WM_PAINT message. We explored a couple of options, including sending our own fake paint messages, or hosting those child windows in their own invisible top-level system-redirected layered windows, but composing their contents in our window. We will revisit this technique later in this article.

With this system in place, we were able to successfully integrate child windows - without any airspace limitations - into scenarios that were simply impossible before:

A 3D cube with standard WebBrowser controls on each face, much like Chris Cavanagh’s YouCube.YouCube
Simple transition effects, like spinning and fading
Standard stuff like putting ActiveX controls within a scrollviewer and using an AdornerLayer to adorn them.
A carrousel control with a mix of WinForms, WPF, ActiveX, and straight-up Win32 controls on it; with a mirror effect and much smoother animation.
An MDI solution that could host legacy controls in each floating window, with transparency, scaling, and visual brushes.
A RemoteDesktop ActiveX client running in a layered window.
Using other video playback controls rather than our troubled MediaElement.
And many, many more examples.

I hesitate to overstate this, but it was quite possibly the most exciting work I have ever done during my 15 years at Microsoft. I was very proud of the accomplishments of my engineering team in solving one of the most vexing problems that afflicted the otherwise fantastic WPF platform.

You can imaging my heartbreak when, after an extensive review, we decided we could not actually ship this feature. Our concern was that we had to hack too deeply into the system, and in ways that were too difficult to explain - let alone maintain. Even though we required that developers explicitly turn on this feature for each HwndHost, we felt the kinds of problems they would encounter would be baffling to them and training our support engineers to handle the escalations would be very difficult. Even towards the end of our development, we were struggling with a long bug tail and performance concerns.

This is exactly the kind of deep system integration that needs to be done by the Win32 platform team, officially sanctioned and supported. With Win8, we are beginning to see some incremental improvements in this space, as noted before in the DirectComposition API. Unfortunately, it is still not possible to build the same kind of rich composited experience we had developed.

The rest of this article will focus on how external developers like you can build some of this functionality without hacking as deeply as we did. I will leverage the lessons we learned during our development to provide reasonable real-world workarounds to the most common problems. I will build an example MDI framework as a real-world application of the mitigations we will discuss. Please note that this is an example, it is not intended to be a feature-complete solution you can just drop into a project. But it should serve to showcase the problems and workarounds nicely.

Subclassing HWNDs

Before we dig into mitigating the airspace issues, we should first discuss intercepting window messages. WPF offers several hooks for intercepting the window messages received by the windows we integrate with. HwndSource.AddHook and HwndHost.MessageHook are examples. However, it is sometimes the case that you need to intercept messages sent to other windows. Win32 programmers have long used a technique called subclassing to replace the HWND’s window proc with their own, thereby allowing them access to the messages dispatched to the window. This is a very powerful technique, but surprisingly tricky to do from managed code. Since the new window procedure is usually implemented in managed code, but the function pointer is stored in unmanaged memory, you must carefully control the lifetime of the delegate to ensure that the GC doesn’t decide to collect the delegate while a reference is still being held in unmanaged memory. This requirement has led to a popular myth that claims you actually need to pin your delegate in order to also prevent the GC from relocating the delegate. It turns out that this is not actually required, as Chris Brumme - an authority on these matters - explained in his blog:

Along the same lines, managed Delegates can be marshaled to unmanaged code, where they are exposed as unmanaged function pointers. Calls on those pointers will perform an unmanaged to managed transition; a change in calling convention; entry into the correct AppDomain; and any necessary argument marshaling. Clearly the unmanaged function pointer must refer to a fixed address. It would be a disaster if the GC were relocating that! This leads many applications to create a pinning handle for the delegate. This is completely unnecessary. The unmanaged function pointer actually refers to a native code stub that we dynamically generate to perform the transition & marshaling. This stub exists in fixed memory outside of the GC heap.

However, the application is responsible for somehow extending the lifetime of the delegate until no more calls will occur from unmanaged code. The lifetime of the native code stub is directly related to the lifetime of the delegate. Once the delegate is collected, subsequent calls via the unmanaged function pointer will crash or otherwise corrupt the process. In our recent release, we added a Customer Debug Probe which allows you to cleanly detect this - all too common - bug in your code. If you haven’t started using Customer Debug Probes during development, please take a look!

Keeping your managed delegate alive as long as native code references the function pointer is actually hard to do in practice because things like AppDomain.Unload can tear down your managed objects quite abruptly. Even more complicated when dealing with subclassing is that someone else could subclass the window after you. Since subclassing just replaces the window procedure with another function pointer, and it is up to that implementation to call the previous one, the pattern does not support unsubclassing from the middle of a “chain” of window procs. A final wrinkle is that finalizers and events like AppDomain.Unloaded run on a separate thread, and this is almost always at odds with the thread-affinity model of Win32 (and WPF).

Microsoft improved the situation in ComCtl32 v6, with the addition of ”safe subclassing” APIs like SetWindowSubclass. Disappointingly, these new APIs remain relatively unknown. In the code accompanying this article, I have provided a managed class called HwndHook that uses these APIs to allow you to arbitrarily hook the window proc of any window in your process and owned by your thread. This will come in handy mitigating some of the issues we will discover in our MDI solution.

Mitigating Rendering Artifacts

As mentioned above, when changing the position of a child window, Win32 has an optimization to copy the contents from the old location to the new location, and just repaint the old location. When WPF is rendering with DirectX from our separate render thread, reading the screen contents from GDI on the UI thread is unreliable. The preferred solution is to disable WPF hardware acceleration, which causes WPF to use GDI, and GDI can coordinate appropriately across threads; but that may not be desirable for other reasons.

Another way to avoid the problem with this rendering artifacts is to disable the problematic optimization that copies the pixels. The key is the WM_WINDOWPOSCHANGING message. This message is sent to a window before it is actually moved or sized, giving it an opportunity to adjust the various parameters; and one of those parameters contains the flags which can be set to include the SWP_NOCOPYBITS flag.

HwndHost already responds to this message in order to coerce the HWND to have the size and position determined by layout; but it does not currently set this flag. It is easy enough to do this for HwndHost instances you create, by overriding the WndProc virtual. But what about existing HwndHost-derived classes like WebBrowser? Since HwndHost exposes the MessageHook event, it is relatively straightforward to add an extension method to add a handler to that event, look for the WM_WINDOWPOSCHANGING message, and set the SWP_NOCOPYBITS flag. You could also you use the HwndHook class introduced above to accomplish the same thing.

When you include the SWP_NOCOPYBITS flag, Win32 will move the window and then send it paint messages to render the contents in the new location. Painting in Win32 is split into two passes: painting the background, and then painting everything else. The background painting can be done automatically by specifying an appropriate brush when creating the window. Window procedures can also respond to the WM_ERASEBKGND message and do their own painting. Background painting is usually done by filling the window with a solid color. The WM_PAINT message is sent after the WM_ERASEBKND and this is where most of the “look” of an window is painted. This two-pass painting scheme can cause flicker on the screen, since GDI can flush its commands to the screen at any time, as you can see the window being cleared and then repainted.

This flicker is what is referred to in KB article 969728. This article recommends clearing the SWP_NOCOPYBITS flag to avoid this flickering for Windows Forms controls. This is, of course, the opposite advice than what I am giving! It is true that clearing this flag will reduce flicker, but it will increase the risk rendering artifacts if you then move around the hosted windows.

The WebBrowser control, however, is an interesting case in which we can do something about the flicker. When IE8 is installed, it seems to clear the background in response to the WM_ERASEBKND message. However, it also seems to completely paint the entire contents of the window in response to the WM_PAINT message. The flicker can be avoided by simply discarding the WM_ERASEBKND message, and letting WM_PAINT handle everything. It turns out that the window that actually does the painting is buried several levels inside the window that HwndHost is hosting, so some fiddly code is required to wait until the window tree is created and to dig down to the appropriate descendent window. Once we have found the window that does the actual painting, we can use the HwndHook class to intercept the WM_ERASEBKND message and discard it. This technique is shown in the sample code as an attached property to WebBrowser, and it is very specific to IE.

Note that IE9 has changed the painting behavior such that it no longer flickers, which suggests it is not clearing the background in response to WM_ERASEBKND anymore. Note also that IE9 now supports hardware accelerated rendering, which changes things substantially. This hardware accelerated rendering is controlled by the FEATURE_GPU_RENDERING key, and is disabled by default. Unfortunately, the only way to enable this setting is by a registry setting, which can be done in the sample application. I have also included an option for disabling script errors, which are really annoying when using the WebBrowser control.

Remember - none of this is needed if you disable hardware acceleration for WPF.

Another interesting discovery was that moving an HwndHost around can cause the old contents to remain on the screen for longer than usual. This can lead to “trails” of old content smearing around the window if you are interactively moving the child window around - say by dragging around an MDI element with a child window inside of it. The reason for this is the when the window moves, Win32 invalidates the region where the window used to be. Invalid regions are updated when the thread gets done processing most other things, but input processing is higher priority, so dragging around the window just keeps accumulating the dirty regions for later. When the WM_PAINT message is eventually sent to our window proc, we just queue a request over to the render thread to update, which is another source of latency. An effective mitigation for this is to paint some content in the HwndHost.OnRender method. Now when the element is moved, WPF knows it needs to immediately update both the old and new positions - without waiting for Win32 to send us a WM_PAINT message.

Mitigating Clipping Issues

The fact that HwndHost elements are not overlap or clip like other visuals is probably the most noticeable airspace issue people encounter. In Win32, overlap is almost always handled by clipping (there are exceptions, such as WS_EX_TRANSPARENT). While in WPF, overlap is almost always handled by overdraw. The Win32 way can be more efficient (though clipping itself can be expensive for complex geometries), while the WPF way allows for effects like transparency. Win32 generally enforces clipping (with a few hints from window styles) while WPF generally makes clipping something you have to explicitly request. To avoid this common class of airspace issues, we need to make WPF and Win32 play by the same overlap/clipping rules.

If we had a redirection model, we could make the Win32 content play by WPF’s rules by using WPF to compose its bitmap with the rest of the scene. Without a redirection model we need to make WPF play by Win32’s rules, which is what we will investigate here.

Win32 clipping occurs in the following situations:

Painting operations are clipped to the window being painted.
This is always done for Win32 windows, but WPF actually rarely clips rendering instructions. This is because, in part, the “shape” of an element is not so easy to determine - and is often defined by what the element actually renders. So how can you know what to clip the rendering to? However, all elements have a Clip property that can be used to enforce clipping.
Painting operations are clipped to avoid children.
This means that parent windows cannot (generally - see WS_CLIPCHILDREN) paint on top of their children. This is also the case in WPF - with the exception that rendering operations for a parent are actually rendered behind the children, rather than clipped. In WPF, AdornerLayers offer the functionality of rendering over children.
Painting operations are clipped to avoid overlapping siblings.
This means that windows cannot (generally - see WS_CLIPSIBLINGS) paint on top of their siblings that are higher in z-order. This is also the case in WPF - with the exception that rendering operations for an element are actually rendered behind their siblings according to their z-order.
Painting operations are clipped to the visible regions of ancestors.
This means that child windows cannot paint outside of their parent windows. Surprisingly, this is not a restriction generally enforced by WPF. Again, this is partly because the “shape” of an element is determined by what it renders - and what it renders is often the result of a subtree of simpler elements. However, a panel can set the ClipToBounds property to cause WPF to clip the children to its layout. Drawing outside of the bounds of ancestors is an important technique for various effects, such as transitions. There are several panels that will clip their children by default; ScrollViewer is an example where clipping is very important to the core functionality of the control, and airspace problems are particularly noticeable.

The “shape” of a window is either a simple rectangle, or defined by a region explicitly set on a window. This shape is used for clipping, as well as for input hit-testing. For example, regions are how top-level windows are given rounded corners (though this region is supplied by the system, not by application code). Note that layered windows - especially the type where the application provides a bitmap with an alpha channel - are slightly different in that the OS will ignore completely transparent pixels when hit-testing. This is not true for any other type of window.

One could imagine calculating enough information from WPF’s layout and rendering data to determine an appropriate shape for the visible part of a hosted child window, and setting a region to reflect that shape. However, much of what appears to be clipping in WPF due to overlapping content is actually just overdraw. WPF implements the painters algorithm (remember Win32/GDI normally implements the reverse painters algorithm), and is happy to overdraw content. This would have to be detected and accounted for to be a general solution. Another concern would be performance. WPF can animate the contents of the visual tree, and the effective shape of the visible region of the HwndHost could be constantly changing. I have not pursued this approach because it is complicated to extract the necessary information from WPF in a performant way. It might be an interesting exercise for the interested reader.

Child windows are always clipped to their parents. This could be used to add clipping behavior to a container, such as a ScrollViewer, that need to clip its contents to the viewport. ScrollViewer could be configured to use a child window as the viewport, and all of the content (including other child windows) would be placed within this intermediate child window and Win32 would naturally enforce the clipping.

Child windows are also clipped to their siblings, assuming the appropriate window styles are set. If we have two elements that are supposed to overlap each other, we could contain each one within their own intermediate child windows, and then the elements would clip properly against each other. Of course, the shape would be either a simple rectangle or an explicit region, but Win32 would use clipping handle the overlap. This would address the problems if there was a hosted child window within the subtrees, but it disables overdraw so effects like transparency are lost.

The key to this “intermediate child window” technique is to create a WPF control that will host a child window, and that will in turn host more WPF content within the child window. The contents of such a control would be clipped to the bounds of the child window, and it would clip properly when overlapped with other such controls. WPF actually provides the two major pieces of required functionality in the HwndHost and HwndSource classes; all we basically have to do is plug them together. HwndHost hosts a child window within an element tree, and HwndSource hosts an element tree within a window. Usually HwndSource is used to host a WPF element tree within a top-level window, but it works just as well for a child window.

The trick is keeping all of the normal WPF features working, so that the inner element tree appears to be fully connected to the outer element tree. We can’t hook together the visual tree, because the elements are not visually connected. However, WPF is designed to support most features between elements connected through the so-called “logical” tree. Once we hook up the logical tree, events route correctly and properties inherit as expected.

Input events are initiated from their corresponding window messages, such as WM_MOUSEMOVE or WM_KEYDOWN, which Win32 dispatches to particular windows based on who the mouse is over, or who has keyboard focus. Introducing a child window will cause some of these messages to be dispatched directly to the child window, rather than going to the top-level window. The HwndSource that we are using in the child window has all of the logic to respond to these messages and drive the WPF input system, and since the disjoint element trees are logically connected, the input events are routed through the tree as expected.

Keyboard input is handled differently from mouse input. In well-behaved applications, WPF reacts to keyboard events when the corresponding message is processed through the ComponentDispatcher. This is supposed to happen before a message is dispatched to any particular window; but that depends on the cooperation of the application’s message pump. Top-level WPF windows will respond to keyboard events from the ComponentDispatcher when focus is anywhere within them, and convert the messages into calls through IKeyboardInputSink. HwndHost has a stub implementation of this interface, but it is the responsibility of the class that derives from HwndHost to provide the full implementation. Conveniently, HwndSource also implements this interface, so once again all we have to do is plug the two together. There is one interesting wrinkle, though. The IKeyboardInputSync interface on HwndHost is called in response to keyboard events routing to it, and HwndSource will raise new keyboard events when its IKeyboardInputSync is called. The problem is that once we logically connect the inner element tree to the outer element tree, the keyboard event routes are unified. This means that the keyboard event raised by the inner HwndSource will actually route through the outer HwndHost, which will then call into IKeyboardInputSync again, creating an infinite loop. We work around this strange situation by temporarily severing the logical tree during the calls to IKeyboardInputSync.

In the sample code accompanying this article, this “intermediate child window” technique is implemented in a class called HwndSourceHost.

Mitigating Z-Order Issues

Above we showed that the problem of overlapping and clipping WPF content along with Win32 content could be mitigated by introducing intermediate child windows. However, the z-order of these windows still needs to be kept in sync with WPF’s sense of z-order.

When a WPF panel allows its children to overlap, the “z-order” of the the children becomes important, as this determines which children are in front of others. WPF determines the z-order of visuals it is rendering by the order they are returned from GetVisualChild. Calls to GetVisualChild are often just passed through to a controls underlying UIElementCollection, so adding a removing elements from the UIElementCollection can obviously impact the z-order of the elements.

When an HwndHost is first connected to an element tree that is rooted beneath an HwndSource, the HwndHost is asked to provide the hosted window. If the HwndHost is removed from the tree, the existing window is not destroyed, it is simply reparented under a non-visible window; and when the HwndHost is added to another element tree, that window is simply reparented under the new HwndSource. When reparenting a child window, it is placed on top of the existing children, which may not correctly reflect the order of the WPF elements in the collection.

WPF panels also support a more efficient mechanism of using an indirection index to map the index of an element in the collection to the appropriate visual index. The Panel.ZIndex property is the public exposure of this indirect indexing feature. Simply setting the ZIndex property will cause the panel to adjust the order it returns the children from GetVisualChild. It is probably possible to monitor changes to this property, but it seems incomplete since panels are free to adjust the order they return elements from GetVisualChild however they please.

Since it does not seem practical to automatically detect changes that might influence the z-order of the elements in a panel, we rely on the application knowing when the z-order of possibly overlapping HwndSourceHost elements have changed. The application can then call the static method HwndSourceHost.UpdateZOrder passing in the panel in which the HwndSourceHost elements are being overlapped. This static method simply enumerates the children by calling GetVisualChild and corrects the Win32 z-order to match. The order of elements returned by GetVisualChild is necessarily the proper z-order, so this is correct, if a slight burden on the application.

Redirection Revisited

The mitigations presented so far have been pretty straight-forward, but still limit the fidelity of composing hosted Win32 windows within WPF element trees. In fact, we have further constrained the regular WPF portions of the tree to align with the clipping rules of Win32. Effects that require overdraw, such as transparency, are no longer possible. Effects that require rasterizing through an intermediate surface, such as a visual brush or shader effects, are also broken. To provide for those features, we need to use some form of redirection instead.

In this section, we turn our attention to implementing a simple form of redirection. We will handle output redirection for GDI - and DirectX on Windows 7 and above. We will also handle input redirection for the mouse; recall that keyboard input does not require redirection at all since it is not dependent on the rendered appearance of the elements. We will not be using any form of API interception; instead we will be using the fully supported features of redirected layered windows. The penalty we will pay is primarily in performance; so make sure you investigate the impact on the performance of your scenario on your target hardware.

The basic idea of this technique is to place each child window we want to redirect within a separate top-level redirected layered window that has its transparency set to 0. This is a standard top-level window with the WS_EX_LAYERED style, and the transparency has been configured with a call to SetLayeredWindowAttributes. It is important to note that this is an officially supported type of window, we are not coercing the system into some unusual state. This top-level window is transparent so that it is not visible on the screen. Since the transparency of this window could change at any time, Win32 continues to redirect and retain the contents of this top-level window even when it is completely transparent. That is the most important feature we are using - that the window is painting, and the contents are captured into a bitmap, even though the window is not visible.

To display the hosted window in WPF, we copy the content from the invisible top-level window into a bitmap, and display that bitmap as a regular element in the WPF element tree. This makes complete composition possible - overdraw, clips, transforms, shader effects, etc. Even though the top-level window is invisible, Win32 allows access to the content by either calling PrintWindow or just blitting from the GDI device context for the top-level window. Ideally we would copy the contents into a bitmap only when the contents actually change. Unfortunately, I have not found a technique to reliably detect this for both GDI and DirectX content. Instead, we have to resort to updating the bitmap on a timer. As you can imagine, this results in unnecessary work and contributes to the performance penalty of this technique.

Copying the contents of the bitmap works great for GDI content, but DirectX content is more problematic. In Windows XP, redirected windows did not redirect DirectX content at all. Starting with windows Vista, the DWM (when it is enabled, of course) assumes responsibility for managing the surfaces backing top-level windows. Vista uses the “VistaBlt” mode, in which GDI and DirectX are redirected into separate surfaces, and then combined together to be presented. It is not until Windows 7, with the introduction of device bitmaps, that GDI and DirectX content are redirected into the same surface. GDI methods like PrintWindow, or reading from the GDI device context directly, access the surface that contains the GDI content. Only with device bitmaps (Windows 7 and above) will our technique work for DirectX content. This is important because if you are actually hosting some WPF content within your redirected child window, WPF will render with DirectX unless you disable hardware acceleration. Of course, DirectX content can come from other technologies than WPF; such as XNA, Direct2D, Windows Media Player, etc. Also recall that DirectX surfaces presented with D3DSWAPEFFECT_FLIPEX are handled specially by the DWM; they are not actually combined into the redirected surface but rather are composed on top when the DWM presents the window. These surfaces are not accessible to our redirection technique, so you will have to either configure the component you are hosting to disable this mode or fall back to the clipping techniques discussed previously. One common component that uses D3DSWAPEFFECT_FLIPEX is the Windows Media Player, and I am not aware of any way to disable it. It is very important that you consider all of the requirements you need to satisfy carefully when using this technique, and test thoroughly to avoid being caught by surprise by some limitation.

For mouse input redirection we use the basic technique of aligning the appropriate point of the child window under the mouse by repositioning the corresponding top-level layered window that contains it. We perform a hit-test in WPF to the image of the hosted window that we are rendering. The image has the same dimensions as the hosted window, and so the offset into the image gives us the appropriate position within the hosted child window that needs to be under the mouse. A little bit of simple math and coordinate transforms can calculate where the containing top-level layered window needs to be. We then move the window, z-order it to the top, and change the transparency to 1 out of 256. This tiny fraction of visibility is enough that Win32 will pass it mouse input, but transparent enough that it is not detectable to the human eye (as far as I can tell). Note that moving around a layered window, and even changing its transparency, is fairly inexpensive since the application does not need to repaint the contents.

Because we are positioning the windows so that mouse input will get to the right destination, it might be appropriate to use a mouse hook. However, that complicates things if you want to host child windows from other processes (such as a WebBrowser containing an XBAP). For simplicity, and because we are already using a timer for updating the bitmap, we also use a timer to synchronize the position and transparency of the top-level layered windows. This is not perfect, but it works reasonably well. I will leave the implementation of a mouse hook to the interested reader.

In the sample code accompanying this article, this technique is implemented in the RedirectedHwndHost class.

MDI Demo

Many people have noted that WPF does not support the Multiple Document Interface (MDI) style of user interfaces out-of-the-box. The reason for this is the age-old rationalization that we believed our time was better spent working on other features. At the time, MDI user interfaces were being deemphasized in favor of other options like tabbed interfaces. MSDN even has this discouraging language on its section on Multiple Document Interfaces:

Many new and intermediate users find it difficult to learn to use MDI applications. Therefore, you should consider other models for your user interface. However, you can use MDI for applications which do not easily fit into an existing model.

We posted our position on the matter way back in 2005. However, MDI continues to be a very popular user interface model in certain types of applications. Since WPF is a very powerful platform, it is fairly straightforward to roll your own MDI solution. Indeed, there are already many MDI solutions for WPF provided by third parties. Some are very simple technology walkthroughs, while others are professional-quality offerings. A quick Bing search shows some of what is available (this is not an exhaustive list!):

Many of these solutions suffer from the airspace issues discussed in this article, so I thought it would be instructive to show how to incorporate the various mitigation techniques in an MDI application. In the demo application on http://microsoftdwayneneed.codeplex.com, I have demonstrated building a proprietary MDI-ish solution primarily because I do not want to endorse any particular third-party offering. While the guidance we are providing is designed to assist all developers in mitigating the airspace issues in WPF applications, it should be of particular use to developers of MDI solutions, and to enterprise developers who are willing to roll their own solutions.

The demo application lets you interactively adjust many settings to turn on/off the various of the techniques discussed in this article. However, it is fair to say that there are still some significant problems, especially in the redirection technique. One particularly vexing problem is that some sequence causes menus to no longer respond to the mouse. I would normally hold off publishing this whitepaper until I had resolved the issue. However, I think the bulk of the content of this post will be of interest (maybe even use) to some of you, and don’t want to hold up publishing indefinitely. It would be great if sufficiently interested and motivated readers would take the information I have presented and run with it.

Fun with floating point

Dwayne Need [MSFT] — Fri, 07 May 2010 06:15:16 GMT

Introduction

WPF uses double-precision floating point numbers (double in C#) in much of its public API and it uses single-precision floating point for much of its internal rendering. So floating point math is something we deal with constantly. Oddly enough, I actually knew very little about the gory details until I recently tried to write a container that had to use doubles as a key, which required working around the problems with precision in floating point math. I found the whole exercise to be very interesting, and so I’ll present what I learned here.

The Problem

Consider this code:

if (0.8 - 0.7 == 0.1)
{
    Console.WriteLine("Math makes sense.");
}

Believe it or not, the comparison will fail. This bothers me, because I don’t know how to write code that will work as expected. (In this simple case, the compiler will actually generate a warning the the code in unreachable.)

Binging around the internet, I found a great article, which I recommend as a good read:
http://www.cygnus-software.com/papers/comparingfloats/comparingfloats.htm

A common approach is to never compare floating point numbers directly, but instead just check that the absolute value of the difference is small enough. The rub, of course, is that it is very hard to pick the right value for “small enough”. Especially since floating point numbers cover a huge range – from very, very small to very, very large. For example, +/- 0.0001 is probably fine for some values, but is probably not great for very, very large numbers and is certainly a bad choice for very, very small numbers where this tolerance is actually larger than the numbers being worked with.

Another idea is to scale the tolerance with the size of the floating point number. Basically, this technique uses a tolerance that is a percentage of the value. This seems like a reasonable idea to me, though might be difficult to implement efficiently.

Another idea is to consider equal two doubles who only differ in the least-significant bits of their significands, which is the premise of the article above.

No matter what you do, something to keep in mind is the requirements of the .Net Framework around equality and hashcodes. You need to ensure that anything that compares as equal will generate the same hashcode. Failing to do so will lead to subtle errors, such as loosing data in a hashtable.

Before designing a solution, let’s dig into the details of the floating point format.

Background

First, read the wikipedia article on the IEEE 754-2008 standard. The two formats we care most about are binary64 (double in C#) and binary32 (float in C#). Of course, the principles of the IEEE standard can be applied to other sizes, such as the fun Minifloat formats.

Today, we normally represent numbers in decimal positional notation. For example, the number 12.34. Each digit can be understood to be multiplied by some power (indicated by the position of the digit) of the base of the number system. Again, using decimal (base 10), we can expand this shorthand representation to be:

Exponential notation can be used as a more compact representation of either very large or very small numbers. For instance, the number 1,234,000,000,000 could be written a number of ways using exponential notation:

This last form is known as normalized exponential notation, and there is a unique representation of every number in this notation. Which helps a lot when we try to represent numbers in computers.

The magnitude of the number is determined by the exponent, and the precision of the number is determined by the coefficient, otherwise know as the significand.

Base 10 is easy, how about Base 2

Of interest to us, of course, is base 2, since that is what computers use in their normal binary representation of data. The same principles apply: namely that each digit (0 or 1) is multiplied by some power (indicated by the position of the digit) of the base of the number system (2). So our original example of 12.25 (base 10) can be written as:

A fascinating observation, and one that is exploited in the binary IEEE 754 formats, is that in base 2 normalized exponential notation, the leading digit is always 1. You will see how this comes in handy in the next section.

Binary Representation

As stated in the introduction, the IEEE 754-2008 standard defines a number of binary formats. For now we will consider binary64 (double in C#). As the name suggests, this binary format is 64-bits wide, and allocates the bits as follows:

Sign: 1 bit
Exponent: 11 bits
Significand: 52 bits

A picture always helps:

The sign bit is simple enough: 0 for positive, 1 for negative. It is important to note that this is different way of encoding negative numbers than the more typical “two’s complement” technique. There are, for example, both a positive and a negative zero.

The exponent is a little more complicated. Being 11 bits, it can encode values between 0 and 2047, inclusive. (Note that 0 and 2047 both have special meanings, and are not used as exponents.) The exponent field needs to store both positive and negative values (for very small and very large numbers). Instead of storing a sign bit, the IEEE 754 standard specifies an exponent bias of 1023. Before storing a desired exponent value, you add 1023 and store the result. When reading the exponent field, you subtract 1023 to get the originally intended value. The result is that the exponent can effectively be between –1022 and 1023, inclusive.

As mentioned before, both 0 and 2047 (0x7FF) are special values for the exponent field. A value of 0 indicates that the number is subnormal. Previously we noted that a normalized exponential notation required that there be one non-zero digit before the decimal point, and the rest of the digits after the decimal point – and the whole thing multiplied by the base raised to some exponent. We also noted that in binary, the digit before the decimal point would always be 1. There are two cases that break this rule:

Zero
A pure zero obviously cannot have a non-zero digit!
Really, really small numbers
Because of the limited range of exponents, numbers even smaller than 1 * 2^-1022 cannot be represented with a leading digit of 1.

For these cases, the exponent field contains 0. The actual exponent value considered to be –1022 (as if the field contained 1), and the number is assumed to have a leading 0 before the decimal, which means that technically the number is no longer in normalized exponential notation. However, the representation is still unique because there is no other way to represent these really, really small numbers given the limits of the exponent field.

The other special value in the exponent field is 2047 (0x7FF). When the exponent field contains this value, the number is either NaN (Not A Number) or Infinity. If the significand is 0, then the number is considered to be Infinity. If the significant is non-zero, then the number is considered to be NaN. A couple of immediate observations: Infinity can be either positive or negative, and is basically one value larger than the maximum non-infinite value (or one value less than the minimum negative non-infinite value). There can be many different NaN values – all of the different possible significand values – and this is sometimes used to carry extra information. The sign bit can be set of cleared for a NaN, it is simply ignored. Supposedly NaNs can be either quiet or signalling, but the actual format is unclear (at least to me).

Representable Values

Since a double is 64-bits, we know that there are a finite number of values that can be represented. Counting all representable values – NaNs, Infinities, positive and negative, subnormals, normals, etc – is a simple calculation: 2⁶⁴ = 18,446,744,073,709,551,616.

Now that is a lot of numbers! Visualizing the representation of floating point numbers (not the floating point values) as a signed 64-bit number helps reinforce the intuition behind the format:

Note that the diagram is not to scale. By that I mean that the vast majority of representable numbers are in the range of Normals. The Subnormals and the NaNs account for a very small fraction of the representable values. I try to indicate this with the dashed line.

This diagram only shows the positive half. The negative half is exactly the same, with all of the numerical values being negative. Note that there are both a postitive and a negative 0, and there are both a positive and a negative infinity. While there are technically both positive and negative NaNs, the sign is ignored.

Distribution of floating point values

Now that we’ve seen how the representation are distributed between 0, subnormals, normals, infinity, and NaNs, let’s see how the actual floating point values are distributed. A 64-bit double contains way too many values to reason about, so I will take the liberty of switching to a simplified minifloat format. I’ve chosen a 6-bit format: 1 sign bit, 3 exponent bits, and 2 significand bits (but remember, we also have an implicit significand bit), and an exponent bias of 2. In excel I created the following table:

S	E2	E1	E0	S0	S1	S2	Exponent	Significand	As Integer	Value	Classification
1	1	1	1	1	1	1	5	1.75	-31	-56.000	NaN
1	1	1	1	1	1	0	5	1.50	-30	-48.000	NaN
1	1	1	1	1	0	1	5	1.25	-29	-40.000	NaN
1	1	1	1	1	0	0	5	1.00	-28	-32.000	-Infinity
1	1	1	0	1	1	1	4	1.75	-27	-28.000	-Normal
1	1	1	0	1	1	0	4	1.50	-26	-24.000	-Normal
1	1	1	0	1	0	1	4	1.25	-25	-20.000	-Normal
1	1	1	0	1	0	0	4	1.00	-24	-16.000	-Normal
1	1	0	1	1	1	1	3	1.75	-23	-14.000	-Normal
1	1	0	1	1	1	0	3	1.50	-22	-12.000	-Normal
1	1	0	1	1	0	1	3	1.25	-21	-10.000	-Normal
1	1	0	1	1	0	0	3	1.00	-20	-8.000	-Normal
1	1	0	0	1	1	1	2	1.75	-19	-7.000	-Normal
1	1	0	0	1	1	0	2	1.50	-18	-6.000	-Normal
1	1	0	0	1	0	1	2	1.25	-17	-5.000	-Normal
1	1	0	0	1	0	0	2	1.00	-16	-4.000	-Normal
1	0	1	1	1	1	1	1	1.75	-15	-3.500	-Normal
1	0	1	1	1	1	0	1	1.50	-14	-3.000	-Normal
1	0	1	1	1	0	1	1	1.25	-13	-2.500	-Normal
1	0	1	1	1	0	0	1	1.00	-12	-2.000	-Normal
1	0	1	0	1	1	1	0	1.75	-11	-1.750	-Normal
1	0	1	0	1	1	0	0	1.50	-10	-1.500	-Normal
1	0	1	0	1	0	1	0	1.25	-9	-1.250	-Normal
1	0	1	0	1	0	0	0	1.00	-8	-1.000	-Normal
1	0	0	1	1	1	1	-1	1.75	-7	-.875	-Normal
1	0	0	1	1	1	0	-1	1.50	-6	-.750	-Normal
1	0	0	1	1	0	1	-1	1.25	-5	-.625	-Normal
1	0	0	1	1	0	0	-1	1.00	-4	-.500	-Normal
1	0	0	0	0	1	1	-1	0.75	-3	-.375	-Subnormal
1	0	0	0	0	1	0	-1	0.50	-2	-.250	-Subnormal
1	0	0	0	0	0	1	-1	0.25	-1	-.125	-Subnormal
1	0	0	0	0	0	0	-1	0.00	0	.000	-0
0	0	0	0	0	0	0	-1	0.00	0	.000	+0
0	0	0	0	0	0	1	-1	0.25	1	.125	+Subnormal
0	0	0	0	0	1	0	-1	0.50	2	.250	+Subnormal
0	0	0	0	0	1	1	-1	0.75	3	.375	+Subnormal
0	0	0	1	1	0	0	-1	1.00	4	.500	+Normal
0	0	0	1	1	0	1	-1	1.25	5	.625	+Normal
0	0	0	1	1	1	0	-1	1.50	6	.750	+Normal
0	0	0	1	1	1	1	-1	1.75	7	.875	+Normal
0	0	1	0	1	0	0	0	1.00	8	1.000	+Normal
0	0	1	0	1	0	1	0	1.25	9	1.250	+Normal
0	0	1	0	1	1	0	0	1.50	10	1.500	+Normal
0	0	1	0	1	1	1	0	1.75	11	1.750	+Normal
0	0	1	1	1	0	0	1	1.00	12	2.000	+Normal
0	0	1	1	1	0	1	1	1.25	13	2.500	+Normal
0	0	1	1	1	1	0	1	1.50	14	3.000	+Normal
0	0	1	1	1	1	1	1	1.75	15	3.500	+Normal
0	1	0	0	1	0	0	2	1.00	16	4.000	+Normal
0	1	0	0	1	0	1	2	1.25	17	5.000	+Normal
0	1	0	0	1	1	0	2	1.50	18	6.000	+Normal
0	1	0	0	1	1	1	2	1.75	19	7.000	+Normal
0	1	0	1	1	0	0	3	1.00	20	8.000	+Normal
0	1	0	1	1	0	1	3	1.25	21	10.000	+Normal
0	1	0	1	1	1	0	3	1.50	22	12.000	+Normal
0	1	0	1	1	1	1	3	1.75	23	14.000	+Normal
0	1	1	0	1	0	0	4	1.00	24	16.000	+Normal
0	1	1	0	1	0	1	4	1.25	25	20.000	+Normal
0	1	1	0	1	1	0	4	1.50	26	24.000	+Normal
0	1	1	0	1	1	1	4	1.75	27	28.000	+Normal
0	1	1	1	1	0	0	5	1.00	28	32.000	+Infinity
0	1	1	1	1	0	1	5	1.25	29	40.000	NaN
0	1	1	1	1	1	0	5	1.50	30	48.000	NaN
0	1	1	1	1	1	1	5	1.75	31	56.000	NaN

Which is a pretty nice, compact summary of how the IEEE floating point format works. However, if we graph the results, we see that the floating point values follow an interesting curve:

Again, the horizontal axis is the naive interpretation of the 6 bits as an integer (though the sign is kept as a separate bit, rather than using two’s complement). The vertical axis is the actual floating point value represented by those bits, according the the IEEE-ish minifloat format we chose.

As you can see, the distribution is not linear. In fact, and I suppose unsurprisingly, the distribution is exponential.

Actually, within each exponent band, the distribution is linear, which I have color-coded in the graph above. The exponent of each band is basically the slope of the line along which the values are evenly distributed. You can also see that the floating point values close to 0 are densely packed along the vertical axis, while the larger floating point values are spread out along the vertical axis. The spacing along the horizontal axis is fixed.

Which leads us back to our problem: presumably errors in floating point math are accumulated in the least significant bits. But the least significant bits can actually represent quite a range of values, from very small to very large. For instance, in our minifloat format used above, you can see that the least significant bit caused the smallest representation larger than 0 to be 0.125, while the least significant bit caused the next-to-largest value to change by 4.0 to become the largest value.

And that is just in our silly minifloat format. We can do the math for the full 64-bit format. The least-significant bit in the 64-bit double-precision floating point format can range in value from:

In absolute terms, that is a mind-boggling range of possible errors.

Introducing the Binary64 type

For experimentation and investigation into the structure of the binary64 format (double in C#) I’ve built the Binary64 class which you can find on my codeplex site.

The Binary64 struct uses explicit field offsets to overlap a 64-bit double with a 64-bit ulong – the managed equivalent of a union. The size of this struct is still 64-bits, so is as efficient to pass around as either a double or a uint.

A Binary64 can be constructed from either a double or a ulong, and it can be implicitly casted to and from either one. There are also methods to get the next and previous representable values. Knowing what we know now, it is clear that these just increment and decrement the ulong – except for handling a few cases, such as positive and negative 0.

The Binary64 struct also exposes the interesting pieces of a floating point number: the sign, the exponent, and the significand. I exposed these carefully through types, such as a Binary64Exponent for the exponent, and a Binary64Significand for the significand.

As a side-note, I also provide extension methods via the DoubleExtensions class to add most of this functionality to the double class.

And finally, I provide a Round method , which we will discuss next.

Investigating the error

What exactly was wrong with the 0.8 – 0.7 == 0.1 condition at the start of this article? Using the Binary64 class, it is easy to investigate.

Literal Value	0.8
Binary64 Bits	0x3FE999999999999A
Sign	Positive
Unbiased Exponent	0x3FE
Biased Exponent	-1
Significand	0x000999999999999A
Implicit leading digit?	Yes
Significand Numerator	7,205,759,403,792,794
Significand Denominator	4,503,599,627,370,496
Significand Fraction	1.6000000000000000888178419700125

So, the actual value is a little more than 0.8 due to the difficulty of encoding this exact value in base 2. Note that the previous representable value is a little less than 0.8.

Literal Value	0.7
Binary64 Bits	0x3FE6666666666666
Sign	Positive
Unbiased Exponent	0x3FE
Biased Exponent	-1
Significand	0x0006666666666666
Implicit leading digit?	Yes
Significand Numerator	6,305,039,478,318,694
Significand Denominator	4,503,599,627,370,496
Significand Fraction	1.3999999999999999111821580299875

So the actual value is a little less than 0.7 due to the same difficulty of representing this exact number in base 2. Note that the next representable value is a little more than 0.7.

Obviously, since we are dealing with two inexact representations, there is an error accumulating in each. The result of any math operation will likely still contain the error. Indeed, we can see:

Literal Value	0.8 - 0.7
Binary64 Bits	0x3fb99999999999A0
Sign	Positive
Unbiased Exponent	0x3FB
Biased Exponent	-4
Significand	0x00099999999999A0
Implicit leading digit?	Yes
Significand Numerator	7,205,759,403,792,800
Significand Denominator	4,503,599,627,370,496
Significand Fraction	1.6000000000000014210854715202004

Obviously this is not exactly right either. But more interestingly, there are 6 representable binary64 values between the result of calculating 0.8-0.7 and the literal 0.1. Looking at the binary64 representation for each we see that the error has accumulated in the least significant 6 bits:

0.8 - 0.7

0x3FB99999999999A0

11111110111001100110011001100110011001100110011001100110100000

0.1

0x3FB999999999999A

11111110111001100110011001100110011001100110011001100110011010

Proposal: Rounding by reducing precision

As we have seen, floating point math is imprecise, and rounding strategies are challenging due to the huge range in the magnitude of error introduced by a single change in the least significant bit.

I propose performing calculations in higher precision than comparisons. This is not really a novel idea. Graphics cards, for instance, have been able to perform calculations in the pipeline at higher precision than available programmatically in pixel shaders. When reducing precision for comparison, we would obviously discard the least-significant bits first.

To implement rounding, not just truncation, I use the trick of adding half before truncating. The Binary64.Round method takes as a parameter the number of insignificant digits. Based on this it can easily calculate the number of representable floating-point numbers in those insignificant digits, and it adds half of them before clearing those insignificant digits.

Remember – this kind of rounding is about rounding to the nearest nth-representable value, it isn’t about rounding in any absolute sense. Just think of it as reducing precision.

For floating point numbers, the precision of comparison is so important I think it should be part of the type signature. C++ could use int values in a template signature, but sadly C# cannot do the same with generics. If it could we could have a sweet template like Binary64Compare<42> to represent a 42-bit significand instead of the standard 52 (ignoring the implicit leading digit). Such a type would clearly indicate that the 10 least-significant bits are considered to accumulate error and will be rounded away.

Introducing the Double42 type

Since I want to encode the precision information in the type, and given the limitations of .Net generics, the next best thing we can do is declare concrete types. For my purposes, I made the assumption that 10 bits of error was a reasonable amount to tolerate. You can make your own domain-specific choice and implement your own class.

The Double42 struct just containts a double, so it should be as efficient to pass around as a double. It does not change the value of the double, but performs rounding as needed internally.

A Double42 can be constructed from a double , and it can be implicitly casted to and from a double.

This type is only intended for comparison, in support of the principal of doing calculations in higher precision than comparisons, so it does not offer any mathematical operations. It implements both IEquatable and IComparable.

As mentioned in the beginning of this article, Double42 also implements GetHashcode and Equals correctly, so that any two numbers that are “equal” also produce the same hashcode.

Finally, we get to write the code we wanted in the first place:

double a = 0.8 - 0.7;
if ((Double42)a == 0.1)
{
    Console.WriteLine("Math makes sense.");
}

The supporting code for this blog entry can be found on my codeplex site: http://microsoftdwayneneed.codeplex.com

Transparent Windows in WPF

Dwayne Need [MSFT] — Tue, 09 Sep 2008 01:46:00 GMT

Introduction

WPF can obviously render transparent elements within its own window, but it also supports rendering the entire window with per-pixel transparency. This feature comes with a few issues, which I'll discuss in this post.

Layered Windows

Windows supports transparency at the HWND level through a feature called "layered windows". Layered windows are represented by a bitmap, and the OS renders the bitmap whenever it needs to. There are two layered windows modes currently supported by the OS: System-Redirected Content and Application-Provided Content. In either case, the HWND is given the WS_EX_LAYERED extended style. The mode is determined by which API you use to update the window. Windows only supports the layered feature for top-level windows. This is not a technique that can be used for child window transparency.

System-Redirected Content

This mode is determined by using the SetLayeredWindowAttributes API. The bitmap for the window is allocated by the OS, and every attempt to render into a DC for the window is redirected to render into the bitmap. The window receives WM_PAINT messages like normal, though the frequency of these messages is reduced since the OS does not need to have the application repaint the window as it moves around. This mode provides an excellent level of compatibility with existing code. However, because GDI does not preserve the alpha channel in all of its APIs, this mode does not support per-pixel transparency. Instead, the SetLayeredWindowAttributes API lets you specify a color key and an opacity for the entire window.

Note: on XP there was a bug where the redirection was not respected by DirectX. This caused DX to render directly to the screen, while the OS assumed the window content was in a bitmap. This caused lots of visual artifacts on the screen. This issue was fixed in Vista.

Application-Provided Content

This mode is determined by using the UpdateLayeredWindow API. The bitmap for the window is allocated by the application, and is passed to the OS which makes a copy. The window does not receive WM_PAINT messages, since there is no need to ask the application to ever repaint itself. If the application has fresh content, it is responsible for calling UpdateLayeredWindow again. The application is completely responsible for generating its content. This API allows the application to provide a bitmap that has a per-pixel alpha channel. But because the OS does not redirect any GDI painting calls, this mode does not support compatibility with existing code.

WPF Chooses Application-Provided Content

WPF chose to only support the Application-Provided Content mode of layered windows. This was chosen primarily because it enabled the broadest range of features, and we didn't want to confuse our API with a mix of modes and features. WPF deeply supports per-pixel transparency in its rendering, so it was a natural fit. Per-pixel transparency naturally allows WPF's anti-aliased rendering to work on the layered window too, which makes the edges look much nicer.

This choice has a few important ramifications:

Non-Client Area

The "non-client" area of a window is a general reference to the parts of the window that the windowing system normally renders for the application. This includes the title bar, the resize edges, the menu bar, the scroll bars, etc. These parts are drawn by the windowing system to the DC of the window. Because layered windows that use the application-provided content mode do not redirect GDI calls, the windowing system is not able to render into the bitmap that the application is using to represent its content. Further, on Vista, the windowing system uses a complex shader to distort the background behind the non-client area. This distortion cannot simply be encoded into a bitmap anyway. And because the bitmap WPF is going to provide will represent the entire window (including the non-client area), WPF would have to carefully handle the non-client messages. WPF chose to avoid all of this complexity by expanding the client area to fill the entire window by handling the WM_NCCALCSIZE message. No matter what your window styles may suggest, transparent WPF windows do not have any visible non-client area. This is fine for many scenarios where the intent is to create a custom window shape, but it can be annoying for people who just want to "fade in" a normal window.

Child Windows

Again, because layered windows that use the application-provided content mode do not redirect GDI calls, child windows do not automatically get rendered into the bitmap that the application is going to use to represent its content. You can try to get the child window to paint into a system-memory bitmap, and then blit the bitmap into WPF. The problem with this is knowing when the child window is "dirty". Most implementations use a timer to poll the content from the child window. This is not efficient, but may be suitable for your needs. A popular implementation of this technique is:

http://www.codeplex.com/WPFWin32Renderer

Construction Only

Windows will let you switch a top-level window in and out of the various layered modes. It is a bit tedious, as you have to clear the WS_EX_LAYERED extended style, then set it again, and call the appropriate API to lock in the new mode. For simplicity, WPF chose not to support this ability. You have to decide at construction time whether the window will be layered or not, and the window cannot be changed afterwards. If needed, you can accomplish something similar by creating a new window, and transferring the element tree over to the new window. WPF enforces this constraint rigidly, rejecting any attempt to alter the underlying WS_EX_LAYERED extended window style.

HwndSource

You can specify that you want to use a layered window when you construct the HwndSource by setting the UsesPerPixelOpacity property of the HwndSourceParameters that you pass to the constructor. When this setting is specified, the HwndSource class will set the WS_EX_LAYERED extended window style, prevent the window from being themed, and configures the WPF render target to have a transparent back buffer.

HwndSourceParameters p = new HwndSourceParameters("TestWindow", 100, 100);
p.UsesPerPixelOpacity = true;
p.WindowStyle |= 0x10000000; // WS_VISIBLE

Ellipse ellipse = new Ellipse();
ellipse.Width = 100;
ellipse.Height = 100;
ellipse.Fill = Brushes.Red;
ellipse.Opacity = 0.5;

HwndSource hwndSource = new HwndSource(p);
hwndSource.RootVisual = ellipse;

Window

Of course, most people don't use HwndSource directly, but use the Window class instead. The Window class offers its own property called AllowsTransparency. The Window class delays creating the underlying HwndSource until it is shown, so this property is simply passed to the HwndSource constructor at that time. Because the underlying HwndSource expands the client area to fill the entire window, the Window class also requires that the WindowStyle property be set to None to avoid confusion.

The Window class is a control with a template that renders a background, so if you often want to set the Background property to a brush with transparency. Of course, you can also specify an opacity for the entire window by setting the Opacity property.

Here is some example XAML markup for create a layered window:

<Window xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
Width="100"
Height="100"
AllowsTransparency="True"
WindowStyle='None'
Background="Transparent">
<Ellipse Width="100" Height="100" Fill="Red" Opacity="0.5"/>
</Window>

Hit Testing

There are two important phases to determining where to send mouse events:

Which HWND is the mouse over?

The operating system must respond very quickly to mouse movement. Windows uses a dedicated Raw Input Thread (RIT), running in the kernel, to handle the signals from the mouse hardware. The RIT quickly scans over the HWND hierarchy to see which window the mouse is over. Because the system must be responsive, no application code is invoked. For normal windows, the RIT checks the mouse position against the window's rectangle (or region, if one is set). But for layered windows, the RIT looks in the bitmap that specifies the content for the window and checks the effective transparency at that location (which can be affected by the constant opacity setting, the color key setting, or the per-pixel alpha channel). If the pixel is 100% transparent, the RIT skips the window and keeps looking. Once a window has been located, the mouse move flag is set on the thread that owns the window. This will cause the thread to receive a WM_MOUSEMOVE message the next time it calls GetMessage() and there are no other higher-priority messages.

Which UIElement is the mouse over?

WPF only uses an HWND as its outer container. Within the HWND is a rich hierarchy of elements, and WPF determines which element to route the mouse events to via its own hit-testing. In particular, WPF uses the UIElement.InputHitTest method. WPF performs the hit-test against the geometry used for the rendering primitives. If the geometry is filled, with any kind of brush, it is considered solid. Even if the brush is "Transparent", WPF still considers the geometry to be solid, like a piece of glass. On the other hand, if the geometry is not filled with a brush, it is considered hollow. The important point here is that WPF does not check the transparency at the mouse position. WPF hit-testing actually involves many other things - IsHitTestVisible, HitTestCore, Visibility, IsEnabled, etc - but those are best left to another article.

Also note that WPF routes the mouse events through the visual tree. So mouse events will pass through all of the parent elements on the way to, and from, the destination element. The event route is strictly based on the hierarchy of elements - it does not reflect what ordering of visuals you might see on the screen.

When using a layered window, you need to understand both hit-testing passes. For WPF to receive the mouse messages at all, you must have at least a partially opaque pixel under the mouse. If that condition is met, then the normal WPF rules apply.

Sometimes people want the top-level window to look transparent, but still receive mouse messages. One technique is to use a nearly-transparent brush for your background. This works pretty well on normal desktops, but will be noticeable on a lower color-depth; such as over terminal services. Another technique is to grab mouse capture, but this is not a passive technique, and it will impact the user interactions with other windows. Finally, you can install a mouse hook, but WPF does not expose any methods for this. You would have to PInvoke to Win32 yourself.

Performance

The performance of WPF's transparent windows is a topic of much concern. WPF is designed to render via DirectX. We do offer a software rendering fallback, but that is intended to provide a full-featured fallback, not for high performance. The layered window APIs, on the other hand, take a GDI HDC parameter. This causes different issues for XP and Vista.

XP

DirectX does provide the IDirect3DSurface9::GetDC method, which can return a DC that references the DirectX surface. Unfortunately there was a restriction in DX9c that would fail this method if it were called on a surface that contained an alpha channel. Of course, the entire point of our layered window API is to enable per-pixel transparency. This restriction was lifted for Vista, but our initial release forced WPF to use its software rendering fallback with rendering to a layered window on XP. We were able to lift this restriction for XP too, which we released as a hot fix (KB 937106). This hot fix was also included in XP SP3, so go get it! Now, on XP, we can render via DirectX and pass the results of IDirect3DSurface9::GetDC directly to UpdateLayeredWindow. On good video drivers, the resulting copy will remain entirely on the video card, leading to excellent performance. Some video drivers, however, may choose to perform this copy through system memory. The performance on such systems will not be nearly as good, but should still be reasonable for many scenarios.

Experience suggests that a full-screen constantly updating layered window on good XP machine can consume as little as 3% CPU in overhead.

Vista

As already noted, Vista supports IDirect3DSurface9::GetDC on surfaces with an alpha channel. The problem on Vista is that GDI is now completey emulated in software via the Canonical Display Device (CDD). The CDD operates on system-memory buffers, so even though we could get a DC to our DX surface, blitting from it would cause the contents to be copied into system memory first. The technique used for this copy was very inefficient. We were able to improve this scenario by using the GetRenderTargetData API to fetch the entire contents into system memory, and then create a DC around the system memory before calling UpdateLayeredWindow. The nature of this technique favors cards with higher GPU->CPU bandwidth, such as PCIe. We released this as a hot fix (KB 938660). This hot fix is also included in Vista SP1, so go get it! While this technique is still slower on Vista than on XP, it should still be reasonable for many scenarios.

Experience suggests that a full-screen constantly updating layered window on good Vista machine can consume as much as 30% CPU in overhead. For performance improvements, reduce the amount of updates to the window, in either frequency or area. WPF is optimized around dirty rect updates, so if you change just a portion of the window, WPF will be able to update just that portion to the screen.

Bugs

We discovered a number of issues with WPF's use of layered windows when suspending/resuming, high CPU usage while fast user switching, layered windows not updating across a terminal services connection, minimized layered windows not updating when the desktop was locked, and crashes when changing the system color depth to 8bpp or less. All known issues in these areas have been fixed in our 3.5 SP1 release.

We also discovered an issue where our call to UpdateLayeredWindow was failing with E_INVALIDARG. The error resulted from the window no longer being in the Application-Provided Content mode. But since WPF only allows the mode to be specified at construction time, and we only accept the Application-Provided Content mode, and we carefully reject attempts to change the relevant style bits at runtime, we were at a loss to explain how the window was changing modes. It turns out that when you call PrintWindow, the OS will switch the window into System-Redirected Content mode so that it can grab a snap shot. Since WPF renders on a separate thread, any use of PrintWindow on a WPF layered window was basically a race condition. And it turns out that the IntelliPoint software was using PrintWindow to grab snapshots of all windows for its display. IntelliPoint is widely installed, so any WPF apps that used layered windows were vulnerable to crashing. While this is a real bug in the OS, we worked around it by simply trying to render again if the call to UpdateLayeredWindow fails in this manner. This work-around is included in our 3.5 SP1 release.

On XP we also discovered that sometimes transparent windows would show up behind other windows, instead of in front. This was a huge problem since in WPF menus, tooltips, and combo-box drop-downs are all implemented using transparent windows. We eventually tracked this down to a layered window bug in XP, and a hotfix was made available. This fix should be included in XP SP3. However, we continue to here some (though much fewer) reports of the problem persisting on patched XP systems. At this point we are unable to reproduce this behavior, so we do not have a fix.

Implementing A Custom BitmapSource

Dwayne Need [MSFT] — Sat, 21 Jun 2008 00:11:33 GMT

Windows Imaging Component

WPF uses the Windows Imaging Component (WIC) library to process bitmap data. WIC presents a data flow for bitmap processing that is both simple and powerful. The basic interface for bitmap processing in WIC is IWICBitmapSource. This interface provides a common way of accessing and linking together bitmaps, decoders, format converters, and other bitmap processing components. Components that implement this interface can be connected together in a graph to pull imaging data through. For example:

Once the bitmap processing graph is configured, the data flow is initiated by calling CopyPixels on the component at the front of the graph. CopyPixels instructs the component to produce pixels according to its algorithm - this may involve decoding a portion of a JPEG stored on disk, copying a block of memory, or even analytically computing a complex gradient. The algorithm is completely dependent on the object implementing the interface. In the most common scenario, bitmap processing components are chained together, as shown in the example above. In such scenarios, the components typically call CopyPixels on the component it is connected to to get the actual bitmap data, and then performs some computation on the results.

WIC provides a rich set of components that provide important functionality. These include encoders and decoders for popular image formats (BMP, GIF, ICO, JPEG, PNG, TIFF, HDPhoto), pixel format converters, color space transforms, clippers, scalers, etc.

WIC in WPF

WPF uses WIC internally to implement various features, and it also exposes much of the WIC API through its imaging classes. The base class for bitmap processing components in WPF is BitmapSource. The standard WIC components are exposed as derived classes.

Many of the WPF imaging classes implement the ISupportInitialize interface. On instances of these classes, properties can only be set between calls the BeginInit and EndInit. Setting properties after EndInit is called have no effect. The non-default constructors typically call BeginInit and EndInit appropriately. The default constructors do not call these methods, and you must call them yourself. This allows you to set the properties yourself. Neglecting to call EndInit will prevent any bitmap processing.

Some of the WPF imaging classes accept URIs that identify the location of bitmap data. These URIs can specify data embedded in the application, on the local disk, and on the network. WPF will download the bitmap data for a network location on a background thread. This helps prevent the UI thread from blocking while downloading. The imaging classes that accept URIs also implement the IUriContext interface. This interface allows a BaseUri to be specified in order to resolve any relative URIs. In many cases WPF will set the BaseUri automatically.

WPF implements an aggressive caching policy to avoid duplicate work. In addition to the standard WinINet cache, both bitmaps and decoders are cached, allowing multiple references to the same Uri to share the same bitmap instance and avoid redundant decoding. There are flags that can be used to bypass the caches.

The WPF rasterizer only natively renders bitmaps in Bgr32 and PBgra32 formats. Even though the WPF imaging classes can process bitmap data with different formats, they are converted to either Bgra32 or PBgra32 before being rendered. There are certainly scenarios that could benefit from other pixel formats, either for less memory usage (such as black/white formats) or for higher color fidelity (such as 16-bit grayscale); however WPF does not yet provide rendering support for them.

Finally, most of the WPF imaging classes are sealed. This design choice was made to ensure a higher quality product, but it interferes with many legitimate scenarios. We will discuss this more later.

Let's take a moment to review the imaging classes in WPF:

BitmapSource
This is the abstract base class of the WPF bitmap classes. BitmapSource class does not directly implement the IWICBitmapSource interface. Instead, it is a class designed to wrap an IWICBitmapSource implementation. The BitmapSource class has virtual members that provide the same basic functionality as IWICBitmapSource: CopyPixels, Format, PixelWidth, PixelHeight, DpiX, DpiY, and Pallete.

BitmapFrame
Represents image data returned by a decoder and accepted by encoders. You can call one of the static Create methods to create an instance from a URI, a Stream, or an existing BitmapSource. Creating a BitmapFrame will automatically assemble the imaging components needed to decode the data.

FormatConvertedBitmap
This class wraps the standard WIC pixel format converter (IWICImagingFactory::CreateFormatConverter). This component provides the means of converting from one pixel format to another, handling dithering and halftoning to indexed formats, palette translation and alpha thresholding. The Source, DestinationFormat, DestinationPalette, and AlphaThreshold properties are used to initialize the underlying component via IWICFormatConverter::Initialize. The dither type is hard-coded to be WICBitmapDitherTypeErrorDiffusion. The palette translation type is hard-coded to be WICBitmapPaletteTypeMedianCut. The ISupportInitialize interface is used to snap the values of the properties when initialization is complete. Further changes to the properties are ignored.

ColorConvertedBitmap
This class wraps the standard WIC color transformer (IWICImagingFactory::CreateColorTransformer). This component transforms the color space of the input bitmap source. The Source, SourceColorContext, DestinationColorContext, and DestinationFormat properties are used to initialize the underlying component via IWICColorTransform::Initialize. The ISupportInitialize interface is used to snap the values of the properties when initialization is complete. Further changes to the properties are ignored.

TransformedBitmap
This class wraps both the standard WIC scaler (IWICImagingFactory::CreateBitmapScaler) and flip-rotator (IWICImagingFactory::CreateBitmapFlipRotator). The scaler component produces a resized version of the input bitmap source using a resampling or filtering algorithm. The interpolation mode is hard-coded to be WICBitmapInterpolationModeFant. The flip-rotator component produces a flipped (horizontal or vertical) and/or rotated (by 90 degree increments) version of the input bitmap source. Rotations are done before the flip. The Transform property is used to specify both the scale and the rotation to apply to the bitmap source. Along with the Source property, this data is used to initialize to the underlying components via IWICBitmapScaler::Initialize and/or IWICBitmapFlipRotator::Initialize. The ISupportInitialize interface is used to snap the values of the properties when initialization is complete. Further changes to the properties are ignored.

CroppedBitmap
This class wraps a standard WIC clipper (IWICImagingFactory::CreateBitmapClipper). This component produces a clipped version of the input bitmap source for a specified rectangular region of interest. The Source and SourceRect properties are used to initialize the underlying component via IWICBitmapClipper::Initialize. The ISupportInitialize interface is used to snap the values of the properties when initialization is complete. Further changes to the properties are ignored.

CachedBitmap
This class represents bitmap data that is cached in system memory. You can call one of the static BitmapSource.Create methods to create an instance from a memory buffer (IWICImagingFactory::CreateBitmapFromMemory). The contents of the memory are copied into the CachedBitmap instance, and changes to the memory do not change the contents of the cached bitmap. You can also call the CachedBitmap constructor and pass an bitmap source to cache (IWICImagingFactory::CreateBitmapFromSource). The contents of the bitmap source are copied into the CachedBitmap instance, and changes to the bitmap source do not change the contents of the cached bitmap.

BitmapImage
This class is the main WPF imaging class for downloading and decoding a bitmap. The ISupportInitialize interface is used to snap the values of the properties when initialization is complete. Further changes to the properties are ignored. This class is very complex, and offers the following functionality:

Loading from a stream
To load bitmap data from a stream, set the StreamSource property.
Loading from a Uri
To load bitmap data from a Uri, set the UriSource property. BitmapImage implements IUriContext to facilitate the handling of relative Uris. To control how the Uri is fetched from the WinINet cache, set the UriCachePolicy property. For convenience in XAML, there is a custom type converter to convert from the HttpRequestCacheLevel enumeration.
Downloading
BitmapImage will download bitmap data from a network location on a background thread to avoid blocking the UI thread. The IsDownloading property will indicate if the data is being downloaded. The DownloadCompleted, DownloadFailed, and DownloadProgress events are raised as appropriate. Note that the DownloadCompleted event is not raised if no downloading occurred.
Decoding
The bitmap data can be of any format, and WIC will search for an appropriate decoder. For efficiency, you can specify the DecodePixelWidth and DecodePixelHeight properties. These properties will control the size of the decoded image in memory. While downloading is done on a background thread, decoding is done on the UI thread.
Caching
BitmapImage will store the decoded bitmap in system memory. You can control when this happens by setting the CacheOption property. BitmapImage also maintains a cache of previous BitmapImage instances (via weak references) so that loading the same Uri multiple times will share the same instance. To avoid this cache, you can include the BitmapCreateOptions.IgnoreImageCache flag in the CreateOptions property.
Cropping & Rotating
The SourceRect property can be set to indicate the area of the bitmap to crop, and the Rotation property can be set to specify a rotation to apply. Certain decoders can implement these operations when decoding.

InteropBitmap
This class wraps standard WIC components for HICONs (IWICImagingFactory::CreateBitmapFromHICON), HBITMAPs (IWICImagingFactory::CreateBitmapFromHBITMAP), and shared memory sections(WICCreateBitmapFromSection). These bitmap sources are primarily used in interop scenarios. To create a bitmap source for an HICON, call Imaging.CreateBitmapSourceFromHIcon. To create a bitmap source for an HBITMAP, call Imaging.CreateBitmapSourceFromHBitmap. To create a bitmap source around a shared memory section, call Imaging.CreateBitmapSourceFromMemorySection. In all cases, the Imaging methods return a BitmapSource instance, even though the instance is really an InteropBitmap. This is generally not a problem since InteropBitmap does not offer any additional API. One exception is the Invalidate method. This method only works for InteropBitmap instances created around shared memory, and indicates that WPF should reload the bitmap from the data in the shared memory section. To call this method, you must first up-cast to InteropBitmap.

WriteableBitmap
This class represents a bitmap that can be written to by the application via the WritePixels method. In previous versions of WPF, this method would end up allocating a new bitmap on every update. We have improved this class significantly WPF 3.5 SP1, due out later this summer. Improvements include efficient double buffering, direct access to the back buffer, dirty rects, sub-byte pixel formats, greater flexibility with array types, and more! This class is supported in partial trust. Having addressed the major performance issues, we expect that this class will enable some really cool scenarios.

RenderTargetBitmap
This class represents a bitmap produced by rasterizing WPF visuals. You specify the bitmap dimensions, pixel format, etc when you construct an instance. Currently RenderTargetBitmap only supports Pbgra32. Simply call Render and pass the visual you want to rasterize into the bitmap. You can also call Clear to clear the bitmap if needed.

Static Vs Dynamic Bitmaps

The WPF imaging classes are primarily designed for static image processing - such as loading bitmaps. Once a bitmap has been processed, the results are cached in memory. Any further changes to the imaging components are ignored. This poses a problem for bitmap data that is asynchronous. As noted above, WPF downloads bitmap data from network locations on a background thread. It is possible that when WPF renders and calls through IWICBitmapSource.CopyPixels that the bitmap data will not be available. The call to CopyPixels is just a no-op, and the empty results are cached. Even when the download finishes, the bitmap is not updated since there is already a cache of it in memory.

The problem can be seen with this simple XAML:

<Page xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml">
<Image>
    <Image.Source>
      <TransformedBitmap>
        <TransformedBitmap.Source>
          <BitmapImage UriSource="http://www.microsoft.com/presspass/presskits/windowsvista/images/icons/IE.jpg"/>
        </TransformedBitmap.Source>
      </TransformedBitmap>
    </Image.Source>
</Image>
</Page>

The first time you load this XAML, there is a good chance that you won't see the image. Change the UriSource to some other "fresh" URI to reproduce the problem, because once WPF has downloaded the image data, it will always work (until you restart the app) since there is no background thread download needed anymore.

The WPF way of doing this would be to listen for the DownloadCompleted event (only if IsDownloading is true) and waiting to hook up the TransformedBitmap until the download completes. While this is certainly doable by an application, it is rather non-obvious and annoying, and it interferes with expressing the bitmap processing chain declaratively.

Instead of providing a fully dynamic bitmap processing pipeline, WPF bakes support for common scenarios into the BitmapImage class itself. There are various properties on BitmapImage you can set, and then BitmapImage will construct the chain of appropriate WIC bitmap sources for you when the download completes. Indeed, BitmapImage is your one-stop shop for assembling the most common standard WIC components. Depending on the properties specified, BitmapImage will assemble the following chain:

WPF is also very careful about running arbitrary code on the render thread. By caching the result of bitmap processing, the heavy work (such as decoding the bitmap) is performed on the UI thread, and not on the render thread.

Even though the WPF imaging classes are primarily designed for static image processing, there are a few imaging classes designed for dynamic bitmap scenarios: WriteableBitmap, RenderTargetBitmap, and an InteropBitmap created around a shared memory section. These classes are specially handled by WPF for dynamic content. In our forth-comming 3.5 SP1 release, the WriteableBitmap implementation has been substantially improved to provide a reasonably efficient, double-buffered, constant-memory solution for dynamically updating the contents of a bitmap.

Custom BitmapSources

While most of the imaging classes are sealed, WPF does actually support external classes that derive from BitmapSource. In this case, WPF will create an internal implementation of IWICBitmapSource that simply calls the virtual methods on BitmapSource. WPF adds some additional virtuals, including overloads of CopyPixels, and some events for downloading and decoding. All derived classes must override all of these virtuals, regardless of whether or not you need them. In particular, the derived class must supply an implementation for:

virtual PixelFormat Format {get;}
virtual int PixelWidth {get;}
virtual int PixelHeight {get;}
virtual double DpiX {get;}
virtual double DpiY {get;}
virtual BitmapPalette Palette {get;}
virtual void CopyPixels(Int32Rect sourceRect, Array pixels, int stride, int offset);
virtual void CopyPixels(Array pixels, int stride, int offset);
virtual void CopyPixels(Int32Rect sourceRect, IntPtr buffer, int bufferSize, int stride);
Important note: because your custom bitmap class has to override the CopyPixels method that handles a raw memory pointer it can only run in full trust.
virtual bool IsDownloading {get;}
virtual event EventHandler DownloadCompleted {add; remove;}
virtual event EventHandler<DownloadProgressEventArgs> DownloadProgress {add; remove;}
virtual event EventHandler<ExceptionEventArgs> DownloadFailed {add; remove}
virtual event EventHandler<ExceptionEventArgs> DecodeFailed {add; remove;}

Introducing Microsoft.DwayneNeed.Media.Imaging

Most code from my blog gets posted to my codeplex site: http://www.codeplex.com/MicrosoftDwayneNeed Feel free to download it and play with it, or even just link it into an app if you find it useful. For this blog I have added some custom bitmap classes:

CustomBitmap
This class implements the required virtuals from BitmapSource and provides reasonable default values. For example, the DPI is returned as 96.0, the resolution is 0 x 0, the format is Bgr32, etc. The three flavors of CopyPixels are sealed and redirected to a single CopyPixelsCore method. This is the main method you will need to override to implement your logic. There are some virtual members that make no sense at this level, such as IsDownloading and the various downloading and decoding events. CustomBitmap simply provides storage for the event handlers and provides a protected method for raising the events. You will need to call these protected raise methods if you actually implement downloading or decoding functionality. A BitmapSource inherits from Freezable (eventually), and the freezable contract can be tedious to implement correctly. CustomBitmap implements CreateInstanceCore by using reflection to create a new instance of itself. This should be sufficient for most scenarios, and you don't need to provide an implementation in your derived classes unless you want to provide a more performant implementation or if you need to call a non-default constructor. Further, the four "copy" virtual functions CloneCore, CloneCurrentValueCore, GetAsFrozenCore, and GetCurrentValueAsFrozenCore are all sealed and redirected to a single CopyCore method. You will need to override this method if you have state that is not stored in DependencyProperties and that needs to be copied to freshly cloned instances. You will also need to override FreezeCore if your additional state needs to be made readonly when the instance is frozen. If your state is stored in DependencyProperties, the base Freezable implementation will correctly copy and freeze it as needed.

ChainedBitmap
This class represents a custom bitmap that will be chained such that it processes the output of another bitmap source. The default implementation of the BitmapSource virtual members is to delegate to the chained bitmap source. This makes sense for most properties like DpiX, DpiY, PixelWidth, PixelHeight, etc, as in many scenarios these properties are the same for the entire chain of bitmap sources. However, derived classes should pay special attention to the Format property. Many bitmap processors only support a limited number of pixel formats, and they should return this for the Format property. ChainedBitmap will take care of converting the pixel format as needed in the base implementation of CopyPixels. ChainedBitmap also adds handlers to the download and decode events of the chained bitmap source and raises its own events in response. This allows a download or decode event to propagate through the chain of bitmap sources.

ColorKeyBitmap
This class processes a source and produces Bgra32 bits, where all source pixels are opaque except for one color that is transparent. You can specify this color explicitly be setting the TransparentColor property. If this property is not set, the color of the pixel at (0,0) will be used as the transparent color. For simplicity, we only process Bgra32 formatted bitmaps, anything else is converted to Bgra32. Converting a pixel format that does not have an alpha channel to Bgra32 simply creates an opaque alpha value. ColorKeyBitmap then changes the alpha channel for pixels that match the transparent color.

SepiaBitmap
This class processes a source and converts the image to a sepia color scheme. The algorithm is taken from http://msdn.microsoft.com/en-us/magazine/cc163866.aspx, and operates in the linear scRGB color space. For simplicity, we only process Bgr32 or Bgra32 formatted bitmaps, anything else is converted to one of those formats. Bgr32 and Bgra32 share the same memory layout for the RGB channels.

GrayscaleBitmap
This class processes a source and converts the image to a grayscale color scheme. The algorithm is taken from http://en.wikipedia.org/wiki/Grayscale, and operates in the linear scRGB color space. For simplicity, we only process Bgr32 or Bgra32 formatted bitmaps, anything else is converted to one of those formats. Bgr32 and Bgra32 share the same memory layout for the RGB channels.

Demo

I created a simple demo and included it on the codeplex site. It simply allows you to select a URI (I provide a list of images from www.microsoft.com, or you can enter your own), and then chain together up to two of the custom bitmap sources. The app handles the case where the BitmapImage may still be downloading, and defers creating the chain until the content is ready. Make your own CustomBitmap classes and try them out!

Stay Tuned...

Next we'll turn our attention to addressing the static processing limitations of the built-in WPF imaging classes.

Blurry Bitmaps

Dwayne Need [MSFT] — Sat, 06 Oct 2007 01:59:00 GMT

Background: resolution independence

WPF was designed from the start to be resolution independent. So instead of designing your UI in terms of pixels, you are encouraged to use a physical measuring unit (like inches). When you specify coordinates in WPF, you are actually using "measure units", which are defined to be 1/96th of an inch. This was chosen because 96 DPI monitor settings are very common.

When WPF renders to the screen, it takes into account the system DPI, which you can set from the display control panel. Of course, one problem with this is that no one ever sets this correctly. When was the last time you took out a ruler, measured your screen, then divided your resolution by the result? Another reason people don't set the DPI correctly is that Windows doesn't always look good at non-standard DPI settings. This is something we will be improving in the future. Of course, since you have to recalculate the DPI settings everytime you adjust your resolution, it would be best if the hardware could just report it. For these reasons, WPF may not actually be able to draw a real inch on the screen, but it should come reasonably close.

Another aspect of resolution independence is being able to draw with more precision than pixels. If you recall the GDI APIs, drawing calls specified their coordinates as integers, which match the pixel grid. WPF specifies its coordinates using floating point numbers, so you can easily specify coordinates like (1.2,5.234). This is sometimes referred to as sub-pixel positioning. Of course, what does it mean to render smaller than a pixel, given that each pixel has a single final color? The answer is that we blend all of the contributors to a given pixel. You can think of this as a weighted average, so that if one primitive covers most of a pixel, it will contribute more than other primitives that cover very little of the pixel. But blending loses detail, so this can be very noticeable.

Bitmaps: what size should they be?

Bitmaps are often produced by designers using high-end editing tools. Presume that the goal is to preserve the designer's intent, what size should we display the image at? If the designer made a 96x96 bitmap, on a 96 DPI machine it will be 1 inch by 1 inch. But if you show that bitmap on a super-high-end 300 DPI monitor (I've seen one, its awesome), then the bitmap would be less than a centimeter in each dimension. So, did the designer mean 96 pixels, or one inch? Interestingly, modern image formats allow you to embed the designer's intended DPI into the file. WPF will utilize this information, and scale the bitmap so that it ends up being the desired size. However, here we have a problem - WPF always presumes the designer meant the image to be a certain physical size. Often designers mean for the image to be a certain pixel size - after all, they are carefully assigning colors to each pixel in the image. Its not the designer's intent to be a certain physical size, but the tools they use often stamp some DPI setting into the file anyways (commonly, some standard DPI setting like 72 or 96). WPF sees this DPI information and scales the image to match.

One problem: pixel scaling

Bitmaps are difficult because they encode "high frequency" information in the form of per-pixel color information. If the bitmap is displayed at a slightly different size, the result can be very blurry. I won't bore you with lots of examples, but consider this one. Say you have a bitmap that is 3x3 pixels. It looks like this:

Now, if we scale this just a bit, such that the image will fit into 4x4 pixels, the result suffers from the fact that we can't represent each source pixel equally in the destination grid. In fact, each original pixel "expands" by 1/3. But of course, the final pixel can only have one color, so we have to combine overlapping samples. We can try various blending techniques, but they all have problems. The result will look something like this:

The human eye can easily detect this artifact, and we tend to associate the result with being out of focus, or blurry. It can be very irritating.

Another problem: pixel alignment

As I mentioned in the background section, WPF can render with sub-pixel precision. What happens if you render the original 3x3 bitmap at the correct size, but at pixel coordinates (0.33,0.33)? The solid red line ends up straddling two pixel columns, and the best we can do is to blend the contributions, which will result in nearly the same result as the previously described scale.

SnapsToDevicePixels

WPF anticipated that there would be cases where people wanted to align with the pixel grid instead of using sub-pixel precision. You can set the SnapsToDevicePixels property on any UIElement. This will cause us to try and render to the pixel grid, but there are quite a few cases that don't work - including images. We will be looking to improve this in the future.

Now what?

All is not lost. Even though WPF wants really badly to be resolution independent, we can force it to be pixel aligned ourselves. In fact, this isn't even very hard. We just need to do two things:

Size ourselves to real pixel sizes.
Position ourselves on pixel boundaries.

For sizing, we can easily participate in the measure pass, and return a measure size that is equivalent to the desired pixel size. The transform used to factor in the system DPI is provided for us in CompositionTarget.TransformFromDevice. The pixel size is in "device" coordinates, and we transform from the device into measure units. Easy enough.

For positioning, we could participate in the arrange pass, and maybe apply a render transform to ourselves to offset appropriately. However, for this example I'm focusing on bitmaps, so I choose to participate in rendering and pass the appropriate offset to by DrawImage call. Either way would work, but the real problem is that we need to know when anything might shift our position. The WPF layout system is very conservative, meaning that it tries to do as little work as possible. Once you have been measured and arranged, you won't get called again unless your individual layout state is invalidated. If some parent decides to shift you around a little bit, you will most likely not be laid out again. I work around this by subscribing to the LayoutUpdated event. Even though this event is an instance event on UIElement, it actually gets raised anytime any layout activity happens. Lucky for us, this is pretty much what we want.

Bitmap class

The code included below introduces a new class called Bitmap. Bitmap is a replacement for Image, but instead of displaying any image source, it only displays bitmap sources. This lets me access the PixelWidth and PixelHeight properties for determining the appropriate size. The important aspects of this class are:

Derived from UIElement instead of FrameworkElement because I don’t want things like MinWidth, MaxWidth, or even Width.
Bitmap.Source can be set to any BitmapSource.
When measured, it will return the appropriate measure units to display the bitmap’s PixelWidth and PixelHeight
When rendered, it will offset the image it draws to align with the pixel grid
Whenever layout updates, it checks to see if it needs to re-render to align to the pixel grid again.

Its a pretty straight-forward class, check out the source code for the details.

The sample application

Included in the project is a very simple sample application. I have embedded a number of small bitmaps with high-frequency data in them. Further, I encoded the bitmaps are various designer DPIs. In the app, the top stack panel contains instances of Bitmap. The bottom stack panel contains instances of Image. You can clearly see how Image responds to the bitmap DPI, while Bitmap does not. For fun, you can use the arrow keys which will adjust a render transform on the root. Bitmap will snap to the nearest pixel, while Image will use sub-pixel positioning.

Is there a down side?

Of course. By using the pixel size of the bitmap, the bitmap will look very different on different machines. The 300 DPI super monitor will make your bitmap look very small. Further, since we align to the pixel boundary, animating will cause the bitmap to jump by full pixels. Sub-pixel positioning is very good for animations. Finally, you can get gaps between elements due to rounding to the pixel grid. If you can live with these limitations, then maybe this Bitmap class will be useful to you.

Multithreaded UI: HostVisual

Dwayne Need [MSFT] — Thu, 26 Apr 2007 21:47:00 GMT

Background: The WPF Threading Model

In general, objects in WPF can only be accessed from the thread that created them. Sometimes this restriction is confused with the UI thread, but this is not true, and it is perfectly fine for objects to live on other threads. But it is not generally possible to create an object on one thread, and access it from another. In almost all cases this will result in an InvalidOperationException, stating that “The calling thread cannot access this object because a different thread owns it.”

Freezables

Of course, there are exceptions to this restriction. A familiar one is the Freezable class. Freezable objects can be frozen, at which point they become read-only and we lift the single-thread restriction. Examples of frozen Freezables include the standard brushes, available from the Brushes class. These brushes can be used on any thread at any time.

This is incredibly useful, and used for all of the graphics primitive resources (pens, brushes, transforms, etc). You can even derive your own types from Freezable and play by the same rules.

Separate Windows

Unfortunately, that read-only restriction can be a real problem. There are many scenarios where we want to run separate pieces of the UI on separate threads. If these pieces of UI are independent from each other, you can host them in separate windows, and run the windows on separate threads. This can be a reasonable solution for some scenarios, especially scenarios where the separate threads can be run in independent top-level windows. The big restriction for this approach is that the graphics from one cannot be composited with the graphics of the other. So while you could use a child window for the other thread’s UI, the system will just render one on top of the other. You cannot have transparency, you cannot use one as a brush for 3D content, you can’t draw over it, etc.

HostVisual

If your scenario doesn’t require interactivity (meaning input), then there is another option that WPF provides: HostVisual. This option leverages the powerful composition engine in WPF that is already capable of aggregating the rendering primitives from multiple threads into one scene. The element tree owned by the worker thread is rendered into its own composition target (called VisualTarget), and the results are composed into the HostVisual owned by the UI thread.

Issue #1: Hosting a Visual in XAML

The first problem to solve is that the HostVisual class derives from Visual. I can't use an existing panel, such as Border, to host this visual. Border derives from Decorator, which is the standard base class for panels that have a single child. Unfortunately, the child is strongly typed to be a UIElement. I have to use a HostVisual, which does not derive from UIElement. There is no built-in way that I know of to place a Visual as a child of one of the standard elements (such as Border, Grid, Canvas, etc). So we make our own:

[ContentProperty("Child")]

public class VisualWrapper : FrameworkElement

{

public Visual Child

{

get

{

return _child;

}

set

{

if (_child != null)

{

RemoveVisualChild(_child);

}

_child = value;

if (_child != null)

{

AddVisualChild(_child);

}

protected override Visual GetVisualChild(int index)

{

if (_child != null && index == 0)

{

return _child;

}

else

{

throw new ArgumentOutOfRangeException("index");

}

protected override int VisualChildrenCount

{

get

{

return _child != null ? 1 : 0;

}

private Visual _child;

}

Issue #2: Layout and the Loaded event

WPF provides a very convenient event called “Loaded”. This event basically signals when an element has been fully initialized, measured, arranged, rendered, and plugged into a presentation source (such as a window). Many elements use this event, including the MediaElement, but sadly this event is not raised for element trees that are not plugged into a presentation source, and displaying an element tree through the HostVisual/VisualTarget doesn’t count. So work around this, we make our own presentation source and use it to root the element tree that the worker thread will own. This immediately leads into another problem: layout is suspended on all elements until resumed by a presentation source. Unfortunately the official mechanism to do this is internal, so the best we can do is to explicitly measure and arrange the root element. Thus we have our VisualTargetPresentationSource class:

public class VisualTargetPresentationSource : PresentationSource

{

public VisualTargetPresentationSource(HostVisual hostVisual)

{

_visualTarget = new VisualTarget(hostVisual);

}

public override Visual RootVisual

{

get

{

return _visualTarget.RootVisual;

}

set

{

Visual oldRoot = _visualTarget.RootVisual;

// Set the root visual of the VisualTarget. This visual will

// now be used to visually compose the scene.

_visualTarget.RootVisual = value;

// Tell the PresentationSource that the root visual has

// changed. This kicks off a bunch of stuff like the

// Loaded event.

RootChanged(oldRoot, value);

// Kickoff layout...

UIElement rootElement = value as UIElement;

if (rootElement != null)

{

rootElement.Measure(new Size(Double.PositiveInfinity,

Double.PositiveInfinity));

rootElement.Arrange(new Rect(rootElement.DesiredSize));

}

protected override CompositionTarget GetCompositionTargetCore()

{

return _visualTarget;

}

public override bool IsDisposed

{

get

{

// We don't support disposing this object.

return false;

}

private VisualTarget _visualTarget;

}

Background Threads

It is easy to make threads in C#. One trick to be aware of is that you must mark the thread as being a “background” thread, otherwise the application will keep running as long as those threads are alive. Also remember that parts of WPF require that its threads to be initialized for COM’s “Single Threaded Apartment”. All of this is easy enough to do, and you’ll see this code later on:

Thread thread = new Thread(/*…*/);

thread.ApartmentState = ApartmentState.STA;

thread.IsBackground = true;
thread.Start(/*…*/);

The Demo

The demo will be a grid showing 3 movies, each rendered on a different background thread. The demo is very simple, but hopefully gets the salient points across. It is certainly possible to communicate from the UI thread to the worker threads, and given that all threads have their own dispatchers this is actually really easy, but this demo does not do that. Interested readers should consult the documentation for Dispatcher.BeginInvoke.

The XAML

The XAML simply defines a grid with 3 columns and puts a VisualWrapper in each column. This is where we will be putting the HostVisual from code. It would be cool to build a class that automatically displays its content on another thread, but we would have to change the parser to support switching threads since the thread an object is created on is the thread that owns that object. Anyways, the simple XAML is:

<Window x:Class="VisualTargetDemo.Window1"

xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"

xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"

xmlns:local="clr-namespace:VisualTargetDemo"

Title="VisualTargetDemo"

SizeToContent="WidthAndHeight"

Loaded="OnLoaded"

<Grid>

<Grid.ColumnDefinitions>

</Grid.ColumnDefinitions>

<local:VisualWrapper Grid.Column="0"

Width="200" Height="100" x:Name="Player1"/>

<local:VisualWrapper Grid.Column="1"

Width="200" Height="100" x:Name="Player2"/>

<local:VisualWrapper Grid.Column="2"

Width="200" Height="100" x:Name="Player3"/>

</Grid>

</Window>

The Code

The code is also pretty simple. For each of the 3 players, it creates a HostVisual on the UI thread, then spins up a background thread, creates a MediaElement, puts it inside a VisualTarget (which points back to the HostVisual), and puts it all inside our hacky VisualTargetPresentationSource.

public partial class Window1 : System.Windows.Window

{

public Window1()

{

InitializeComponent();

}

private void OnLoaded(object sender, RoutedEventArgs e)

{

Player1.Child = CreateMediaElementOnWorkerThread();

Player2.Child = CreateMediaElementOnWorkerThread();

Player3.Child = CreateMediaElementOnWorkerThread();

}

private HostVisual CreateMediaElementOnWorkerThread()

{

// Create the HostVisual that will "contain" the VisualTarget

// on the worker thread.

HostVisual hostVisual = new HostVisual();

// Spin up a worker thread, and pass it the HostVisual that it

// should be part of.

Thread thread = new Thread(new ParameterizedThreadStart(MediaWorkerThread));

thread.ApartmentState = ApartmentState.STA;

thread.IsBackground = true;

thread.Start(hostVisual);

// Wait for the worker thread to spin up and create the VisualTarget.

s_event.WaitOne();

return hostVisual;

}

private FrameworkElement CreateMediaElement()

{

// Create a MediaElement, and give it some video content.

MediaElement mediaElement = new MediaElement();

mediaElement.BeginInit();

mediaElement.Source = new Uri("http://download.microsoft.com/download/2/C/4/2C433161-F56C-4BAB-BBC5-B8C6F240AFCC/SL_0410_448x256_300kb_2passCBR.wmv?amp;clcid=0x409");

mediaElement.Width = 200;

mediaElement.Height = 100;

mediaElement.EndInit();

return mediaElement;

}

private void MediaWorkerThread(object arg)

{

// Create the VisualTargetPresentationSource and then signal the

// calling thread, so that it can continue without waiting for us.

HostVisual hostVisual = (HostVisual)arg;

VisualTargetPresentationSource visualTargetPS = new VisualTargetPresentationSource(hostVisual);

s_event.Set();

// Create a MediaElement and use it as the root visual for the

// VisualTarget.

visualTargetPS.RootVisual = CreateMediaElement();

// Run a dispatcher for this worker thread. This is the central

// processing loop for WPF.

System.Windows.Threading.Dispatcher.Run();

}

private static AutoResetEvent s_event = new AutoResetEvent(false);

}

Summary

In this demo I’ve shown how to use the HostVisual and VisualTarget classes to compose pieces of UI from different threads. There are some limitations: namely that the UI owned by the worker threads do not receive input events. There were also some annoyances we had to work around along the way, but those proved to be fairly minimal.

The source code for this project is now included in my codeplex site: http://www.codeplex.com/MicrosoftDwayneNeed