Presentation Source

Transparent Windows in WPF

2008-09-09T01:46:00Z

Introduction

WPF can obviously render transparent elements within its own window, but it also supports rendering the entire window with per-pixel transparency. This feature comes with a few issues, which I'll discuss in this post.

Layered Windows

Windows supports transparency at the HWND level through a feature called "layered windows". Layered windows are represented by a bitmap, and the OS renders the bitmap whenever it needs to. There are two layered windows modes currently supported by the OS: System-Redirected Content and Application-Provided Content. In either case, the HWND is given the WS_EX_LAYERED extended style. The mode is determined by which API you use to update the window. Windows only supports the layered feature for top-level windows. This is not a technique that can be used for child window transparency.

System-Redirected Content

This mode is determined by using the SetLayeredWindowAttributes API. The bitmap for the window is allocated by the OS, and every attempt to render into a DC for the window is redirected to render into the bitmap. The window receives WM_PAINT messages like normal, though the frequency of these messages is reduced since the OS does not need to have the application repaint the window as it moves around. This mode provides an excellent level of compatibility with existing code. However, because GDI does not preserve the alpha channel in all of its APIs, this mode does not support per-pixel transparency. Instead, the SetLayeredWindowAttributes API lets you specify a color key and an opacity for the entire window.

Note: on XP there was a bug where the redirection was not respected by DirectX. This caused DX to render directly to the screen, while the OS assumed the window content was in a bitmap. This caused lots of visual artifacts on the screen. This issue was fixed in Vista.

Application-Provided Content

This mode is determined by using the UpdateLayeredWindow API. The bitmap for the window is allocated by the application, and is passed to the OS which makes a copy. The window does not receive WM_PAINT messages, since there is no need to ask the application to ever repaint itself. If the application has fresh content, it is responsible for calling UpdateLayeredWindow again. The application is completely responsible for generating its content. This API allows the application to provide a bitmap that has a per-pixel alpha channel. But because the OS does not redirect any GDI painting calls, this mode does not support compatibility with existing code.

WPF Chooses Application-Provided Content

WPF chose to only support the Application-Provided Content mode of layered windows. This was chosen primarily because it enabled the broadest range of features, and we didn't want to confuse our API with a mix of modes and features. WPF deeply supports per-pixel transparency in its rendering, so it was a natural fit. Per-pixel transparency naturally allows WPF's anti-aliased rendering to work on the layered window too, which makes the edges look much nicer.

This choice has a few important ramifications:

Non-Client Area

The "non-client" area of a window is a general reference to the parts of the window that the windowing system normally renders for the application. This includes the title bar, the resize edges, the menu bar, the scroll bars, etc. These parts are drawn by the windowing system to the DC of the window. Because layered windows that use the application-provided content mode do not redirect GDI calls, the windowing system is not able to render into the bitmap that the application is using to represent its content. Further, on Vista, the windowing system uses a complex shader to distort the background behind the non-client area. This distortion cannot simply be encoded into a bitmap anyway. And because the bitmap WPF is going to provide will represent the entire window (including the non-client area), WPF would have to carefully handle the non-client messages. WPF chose to avoid all of this complexity by expanding the client area to fill the entire window by handling the WM_NCCALCSIZE message. No matter what your window styles may suggest, transparent WPF windows do not have any visible non-client area. This is fine for many scenarios where the intent is to create a custom window shape, but it can be annoying for people who just want to "fade in" a normal window.

Child Windows

Again, because layered windows that use the application-provided content mode do not redirect GDI calls, child windows do not automatically get rendered into the bitmap that the application is going to use to represent its content. You can try to get the child window to paint into a system-memory bitmap, and then blit the bitmap into WPF. The problem with this is knowing when the child window is "dirty". Most implementations use a timer to poll the content from the child window. This is not efficient, but may be suitable for your needs. A popular implementation of this technique is:

http://www.codeplex.com/WPFWin32Renderer

Construction Only

Windows will let you switch a top-level window in and out of the various layered modes. It is a bit tedious, as you have to clear the WS_EX_LAYERED extended style, then set it again, and call the appropriate API to lock in the new mode. For simplicity, WPF chose not to support this ability. You have to decide at construction time whether the window will be layered or not, and the window cannot be changed afterwards. If needed, you can accomplish something similar by creating a new window, and transferring the element tree over to the new window. WPF enforces this constraint rigidly, rejecting any attempt to alter the underlying WS_EX_LAYERED extended window style.

HwndSource

You can specify that you want to use a layered window when you construct the HwndSource by setting the UsesPerPixelOpacity property of the HwndSourceParameters that you pass to the constructor. When this setting is specified, the HwndSource class will set the WS_EX_LAYERED extended window style, prevent the window from being themed, and configures the WPF render target to have a transparent back buffer.

HwndSourceParameters p = new HwndSourceParameters("TestWindow", 100, 100);
p.UsesPerPixelOpacity = true;
p.WindowStyle |= 0x10000000; // WS_VISIBLE

Ellipse ellipse = new Ellipse();
ellipse.Width = 100;
ellipse.Height = 100;
ellipse.Fill = Brushes.Red;
ellipse.Opacity = 0.5;

HwndSource hwndSource = new HwndSource(p);
hwndSource.RootVisual = ellipse;

Window

Of course, most people don't use HwndSource directly, but use the Window class instead. The Window class offers its own property called AllowsTransparency. The Window class delays creating the underlying HwndSource until it is shown, so this property is simply passed to the HwndSource constructor at that time. Because the underlying HwndSource expands the client area to fill the entire window, the Window class also requires that the WindowStyle property be set to None to avoid confusion.

The Window class is a control with a template that renders a background, so if you often want to set the Background property to a brush with transparency. Of course, you can also specify an opacity for the entire window by setting the Opacity property.

Here is some example XAML markup for create a layered window:

<Window xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
Width="100"
Height="100"
AllowsTransparency="True"
WindowStyle='None'
Background="Transparent">
<Ellipse Width="100" Height="100" Fill="Red" Opacity="0.5"/>
</Window>

Hit Testing

There are two important phases to determining where to send mouse events:

Which HWND is the mouse over?

The operating system must respond very quickly to mouse movement. Windows uses a dedicated Raw Input Thread (RIT), running in the kernel, to handle the signals from the mouse hardware. The RIT quickly scans over the HWND hierarchy to see which window the mouse is over. Because the system must be responsive, no application code is invoked. For normal windows, the RIT checks the mouse position against the window's rectangle (or region, if one is set). But for layered windows, the RIT looks in the bitmap that specifies the content for the window and checks the effective transparency at that location (which can be affected by the constant opacity setting, the color key setting, or the per-pixel alpha channel). If the pixel is 100% transparent, the RIT skips the window and keeps looking. Once a window has been located, the mouse move flag is set on the thread that owns the window. This will cause the thread to receive a WM_MOUSEMOVE message the next time it calls GetMessage() and there are no other higher-priority messages.

Which UIElement is the mouse over?

WPF only uses an HWND as its outer container. Within the HWND is a rich hierarchy of elements, and WPF determines which element to route the mouse events to via its own hit-testing. In particular, WPF uses the UIElement.InputHitTest method. WPF performs the hit-test against the geometry used for the rendering primitives. If the geometry is filled, with any kind of brush, it is considered solid. Even if the brush is "Transparent", WPF still considers the geometry to be solid, like a piece of glass. On the other hand, if the geometry is not filled with a brush, it is considered hollow. The important point here is that WPF does not check the transparency at the mouse position. WPF hit-testing actually involves many other things - IsHitTestVisible, HitTestCore, Visibility, IsEnabled, etc - but those are best left to another article.

Also note that WPF routes the mouse events through the visual tree. So mouse events will pass through all of the parent elements on the way to, and from, the destination element. The event route is strictly based on the hierarchy of elements - it does not reflect what ordering of visuals you might see on the screen.

When using a layered window, you need to understand both hit-testing passes. For WPF to receive the mouse messages at all, you must have at least a partially opaque pixel under the mouse. If that condition is met, then the normal WPF rules apply.

Sometimes people want the top-level window to look transparent, but still receive mouse messages. One technique is to use a nearly-transparent brush for your background. This works pretty well on normal desktops, but will be noticeable on a lower color-depth; such as over terminal services. Another technique is to grab mouse capture, but this is not a passive technique, and it will impact the user interactions with other windows. Finally, you can install a mouse hook, but WPF does not expose any methods for this. You would have to PInvoke to Win32 yourself.

Performance

The performance of WPF's transparent windows is a topic of much concern. WPF is designed to render via DirectX. We do offer a software rendering fallback, but that is intended to provide a full-featured fallback, not for high performance. The layered window APIs, on the other hand, take a GDI HDC parameter. This causes different issues for XP and Vista.

XP

DirectX does provide the IDirect3DSurface9::GetDC method, which can return a DC that references the DirectX surface. Unfortunately there was a restriction in DX9c that would fail this method if it were called on a surface that contained an alpha channel. Of course, the entire point of our layered window API is to enable per-pixel transparency. This restriction was lifted for Vista, but our initial release forced WPF to use its software rendering fallback with rendering to a layered window on XP. We were able to lift this restriction for XP too, which we released as a hot fix (KB 937106). This hot fix was also included in XP SP3, so go get it! Now, on XP, we can render via DirectX and pass the results of IDirect3DSurface9::GetDC directly to UpdateLayeredWindow. On good video drivers, the resulting copy will remain entirely on the video card, leading to excellent performance. Some video drivers, however, may choose to perform this copy through system memory. The performance on such systems will not be nearly as good, but should still be reasonable for many scenarios.

Experience suggests that a full-screen constantly updating layered window on good XP machine can consume as little as 3% CPU in overhead.

Vista

As already noted, Vista supports IDirect3DSurface9::GetDC on surfaces with an alpha channel. The problem on Vista is that GDI is now completey emulated in software via the Canonical Display Device (CDD). The CDD operates on system-memory buffers, so even though we could get a DC to our DX surface, blitting from it would cause the contents to be copied into system memory first. The technique used for this copy was very inefficient. We were able to improve this scenario by using the GetRenderTargetData API to fetch the entire contents into system memory, and then create a DC around the system memory before calling UpdateLayeredWindow. The nature of this technique favors cards with higher GPU->CPU bandwidth, such as PCIe. We released this as a hot fix (KB 938660). This hot fix is also included in Vista SP1, so go get it! While this technique is still slower on Vista than on XP, it should still be reasonable for many scenarios.

Experience suggests that a full-screen constantly updating layered window on good Vista machine can consume as much as 30% CPU in overhead. For performance improvements, reduce the amount of updates to the window, in either frequency or area. WPF is optimized around dirty rect updates, so if you change just a portion of the window, WPF will be able to update just that portion to the screen.

Bugs

We discovered a number of issues with WPF's use of layered windows when suspending/resuming, high CPU usage while fast user switching, layered windows not updating across a terminal services connection, minimized layered windows not updating when the desktop was locked, and crashes when changing the system color depth to 8bpp or less. All known issues in these areas have been fixed in our 3.5 SP1 release.

We also discovered an issue where our call to UpdateLayeredWindow was failing with E_INVALIDARG. The error resulted from the window no longer being in the Application-Provided Content mode. But since WPF only allows the mode to be specified at construction time, and we only accept the Application-Provided Content mode, and we carefully reject attempts to change the relevant style bits at runtime, we were at a loss to explain how the window was changing modes. It turns out that when you call PrintWindow, the OS will switch the window into System-Redirected Content mode so that it can grab a snap shot. Since WPF renders on a separate thread, any use of PrintWindow on a WPF layered window was basically a race condition. And it turns out that the IntelliPoint software was using PrintWindow to grab snapshots of all windows for its display. IntelliPoint is widely installed, so any WPF apps that used layered windows were vulnerable to crashing. While this is a real bug in the OS, we worked around it by simply trying to render again if the call to UpdateLayeredWindow fails in this manner. This work-around is included in our 3.5 SP1 release.

Implementing A Custom BitmapSource

2008-06-21T00:11:33Z

Windows Imaging Component

WPF uses the Windows Imaging Component (WIC) library to process bitmap data. WIC presents a data flow for bitmap processing that is both simple and powerful. The basic interface for bitmap processing in WIC is IWICBitmapSource. This interface provides a common way of accessing and linking together bitmaps, decoders, format converters, and other bitmap processing components. Components that implement this interface can be connected together in a graph to pull imaging data through. For example:

Once the bitmap processing graph is configured, the data flow is initiated by calling CopyPixels on the component at the front of the graph. CopyPixels instructs the component to produce pixels according to its algorithm - this may involve decoding a portion of a JPEG stored on disk, copying a block of memory, or even analytically computing a complex gradient. The algorithm is completely dependent on the object implementing the interface. In the most common scenario, bitmap processing components are chained together, as shown in the example above. In such scenarios, the components typically call CopyPixels on the component it is connected to to get the actual bitmap data, and then performs some computation on the results.

WIC provides a rich set of components that provide important functionality. These include encoders and decoders for popular image formats (BMP, GIF, ICO, JPEG, PNG, TIFF, HDPhoto), pixel format converters, color space transforms, clippers, scalers, etc.

WIC in WPF

WPF uses WIC internally to implement various features, and it also exposes much of the WIC API through its imaging classes. The base class for bitmap processing components in WPF is BitmapSource. The standard WIC components are exposed as derived classes.

Many of the WPF imaging classes implement the ISupportInitialize interface. On instances of these classes, properties can only be set between calls the BeginInit and EndInit. Setting properties after EndInit is called have no effect. The non-default constructors typically call BeginInit and EndInit appropriately. The default constructors do not call these methods, and you must call them yourself. This allows you to set the properties yourself. Neglecting to call EndInit will prevent any bitmap processing.

Some of the WPF imaging classes accept URIs that identify the location of bitmap data. These URIs can specify data embedded in the application, on the local disk, and on the network. WPF will download the bitmap data for a network location on a background thread. This helps prevent the UI thread from blocking while downloading. The imaging classes that accept URIs also implement the IUriContext interface. This interface allows a BaseUri to be specified in order to resolve any relative URIs. In many cases WPF will set the BaseUri automatically.

WPF implements an aggressive caching policy to avoid duplicate work. In addition to the standard WinINet cache, both bitmaps and decoders are cached, allowing multiple references to the same Uri to share the same bitmap instance and avoid redundant decoding. There are flags that can be used to bypass the caches.

The WPF rasterizer only natively renders bitmaps in Bgr32 and PBgra32 formats. Even though the WPF imaging classes can process bitmap data with different formats, they are converted to either Bgra32 or PBgra32 before being rendered. There are certainly scenarios that could benefit from other pixel formats, either for less memory usage (such as black/white formats) or for higher color fidelity (such as 16-bit grayscale); however WPF does not yet provide rendering support for them.

Finally, most of the WPF imaging classes are sealed. This design choice was made to ensure a higher quality product, but it interferes with many legitimate scenarios. We will discuss this more later.

Let's take a moment to review the imaging classes in WPF:

BitmapSource
This is the abstract base class of the WPF bitmap classes. BitmapSource class does not directly implement the IWICBitmapSource interface. Instead, it is a class designed to wrap an IWICBitmapSource implementation. The BitmapSource class has virtual members that provide the same basic functionality as IWICBitmapSource: CopyPixels, Format, PixelWidth, PixelHeight, DpiX, DpiY, and Pallete.

BitmapFrame
Represents image data returned by a decoder and accepted by encoders. You can call one of the static Create methods to create an instance from a URI, a Stream, or an existing BitmapSource. Creating a BitmapFrame will automatically assemble the imaging components needed to decode the data.

FormatConvertedBitmap
This class wraps the standard WIC pixel format converter (IWICImagingFactory::CreateFormatConverter). This component provides the means of converting from one pixel format to another, handling dithering and halftoning to indexed formats, palette translation and alpha thresholding. The Source, DestinationFormat, DestinationPalette, and AlphaThreshold properties are used to initialize the underlying component via IWICFormatConverter::Initialize. The dither type is hard-coded to be WICBitmapDitherTypeErrorDiffusion. The palette translation type is hard-coded to be WICBitmapPaletteTypeMedianCut. The ISupportInitialize interface is used to snap the values of the properties when initialization is complete. Further changes to the properties are ignored.

ColorConvertedBitmap
This class wraps the standard WIC color transformer (IWICImagingFactory::CreateColorTransformer). This component transforms the color space of the input bitmap source. The Source, SourceColorContext, DestinationColorContext, and DestinationFormat properties are used to initialize the underlying component via IWICColorTransform::Initialize. The ISupportInitialize interface is used to snap the values of the properties when initialization is complete. Further changes to the properties are ignored.

TransformedBitmap
This class wraps both the standard WIC scaler (IWICImagingFactory::CreateBitmapScaler) and flip-rotator (IWICImagingFactory::CreateBitmapFlipRotator). The scaler component produces a resized version of the input bitmap source using a resampling or filtering algorithm. The interpolation mode is hard-coded to be WICBitmapInterpolationModeFant. The flip-rotator component produces a flipped (horizontal or vertical) and/or rotated (by 90 degree increments) version of the input bitmap source. Rotations are done before the flip. The Transform property is used to specify both the scale and the rotation to apply to the bitmap source. Along with the Source property, this data is used to initialize to the underlying components via IWICBitmapScaler::Initialize and/or IWICBitmapFlipRotator::Initialize. The ISupportInitialize interface is used to snap the values of the properties when initialization is complete. Further changes to the properties are ignored.

CroppedBitmap
This class wraps a standard WIC clipper (IWICImagingFactory::CreateBitmapClipper). This component produces a clipped version of the input bitmap source for a specified rectangular region of interest. The Source and SourceRect properties are used to initialize the underlying component via IWICBitmapClipper::Initialize. The ISupportInitialize interface is used to snap the values of the properties when initialization is complete. Further changes to the properties are ignored.

CachedBitmap
This class represents bitmap data that is cached in system memory. You can call one of the static BitmapSource.Create methods to create an instance from a memory buffer (IWICImagingFactory::CreateBitmapFromMemory). The contents of the memory are copied into the CachedBitmap instance, and changes to the memory do not change the contents of the cached bitmap. You can also call the CachedBitmap constructor and pass an bitmap source to cache (IWICImagingFactory::CreateBitmapFromSource). The contents of the bitmap source are copied into the CachedBitmap instance, and changes to the bitmap source do not change the contents of the cached bitmap.

BitmapImage
This class is the main WPF imaging class for downloading and decoding a bitmap. The ISupportInitialize interface is used to snap the values of the properties when initialization is complete. Further changes to the properties are ignored. This class is very complex, and offers the following functionality:

Loading from a stream
To load bitmap data from a stream, set the StreamSource property.
Loading from a Uri
To load bitmap data from a Uri, set the UriSource property. BitmapImage implements IUriContext to facilitate the handling of relative Uris. To control how the Uri is fetched from the WinINet cache, set the UriCachePolicy property. For convenience in XAML, there is a custom type converter to convert from the HttpRequestCacheLevel enumeration.
Downloading
BitmapImage will download bitmap data from a network location on a background thread to avoid blocking the UI thread. The IsDownloading property will indicate if the data is being downloaded. The DownloadCompleted, DownloadFailed, and DownloadProgress events are raised as appropriate. Note that the DownloadCompleted event is not raised if no downloading occurred.
Decoding
The bitmap data can be of any format, and WIC will search for an appropriate decoder. For efficiency, you can specify the DecodePixelWidth and DecodePixelHeight properties. These properties will control the size of the decoded image in memory. While downloading is done on a background thread, decoding is done on the UI thread.
Caching
BitmapImage will store the decoded bitmap in system memory. You can control when this happens by setting the CacheOption property. BitmapImage also maintains a cache of previous BitmapImage instances (via weak references) so that loading the same Uri multiple times will share the same instance. To avoid this cache, you can include the BitmapCreateOptions.IgnoreImageCache flag in the CreateOptions property.
Cropping & Rotating
The SourceRect property can be set to indicate the area of the bitmap to crop, and the Rotation property can be set to specify a rotation to apply. Certain decoders can implement these operations when decoding.

InteropBitmap
This class wraps standard WIC components for HICONs (IWICImagingFactory::CreateBitmapFromHICON), HBITMAPs (IWICImagingFactory::CreateBitmapFromHBITMAP), and shared memory sections(WICCreateBitmapFromSection). These bitmap sources are primarily used in interop scenarios. To create a bitmap source for an HICON, call Imaging.CreateBitmapSourceFromHIcon. To create a bitmap source for an HBITMAP, call Imaging.CreateBitmapSourceFromHBitmap. To create a bitmap source around a shared memory section, call Imaging.CreateBitmapSourceFromMemorySection. In all cases, the Imaging methods return a BitmapSource instance, even though the instance is really an InteropBitmap. This is generally not a problem since InteropBitmap does not offer any additional API. One exception is the Invalidate method. This method only works for InteropBitmap instances created around shared memory, and indicates that WPF should reload the bitmap from the data in the shared memory section. To call this method, you must first up-cast to InteropBitmap.

WriteableBitmap
This class represents a bitmap that can be written to by the application via the WritePixels method. In previous versions of WPF, this method would end up allocating a new bitmap on every update. We have improved this class significantly WPF 3.5 SP1, due out later this summer. Improvements include efficient double buffering, direct access to the back buffer, dirty rects, sub-byte pixel formats, greater flexibility with array types, and more! This class is supported in partial trust. Having addressed the major performance issues, we expect that this class will enable some really cool scenarios.

RenderTargetBitmap
This class represents a bitmap produced by rasterizing WPF visuals. You specify the bitmap dimensions, pixel format, etc when you construct an instance. Currently RenderTargetBitmap only supports Pbgra32. Simply call Render and pass the visual you want to rasterize into the bitmap. You can also call Clear to clear the bitmap if needed.

Static Vs Dynamic Bitmaps

The WPF imaging classes are primarily designed for static image processing - such as loading bitmaps. Once a bitmap has been processed, the results are cached in memory. Any further changes to the imaging components are ignored. This poses a problem for bitmap data that is asynchronous. As noted above, WPF downloads bitmap data from network locations on a background thread. It is possible that when WPF renders and calls through IWICBitmapSource.CopyPixels that the bitmap data will not be available. The call to CopyPixels is just a no-op, and the empty results are cached. Even when the download finishes, the bitmap is not updated since there is already a cache of it in memory.

The problem can be seen with this simple XAML:

<Page xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml">
<Image>
    <Image.Source>
      <TransformedBitmap>
        <TransformedBitmap.Source>
          <BitmapImage UriSource="http://www.microsoft.com/presspass/presskits/windowsvista/images/icons/IE.jpg"/>
        </TransformedBitmap.Source>
      </TransformedBitmap>
    </Image.Source>
</Image>
</Page>

The first time you load this XAML, there is a good chance that you won't see the image. Change the UriSource to some other "fresh" URI to reproduce the problem, because once WPF has downloaded the image data, it will always work (until you restart the app) since there is no background thread download needed anymore.

The WPF way of doing this would be to listen for the DownloadCompleted event (only if IsDownloading is true) and waiting to hook up the TransformedBitmap until the download completes. While this is certainly doable by an application, it is rather non-obvious and annoying, and it interferes with expressing the bitmap processing chain declaratively.

Instead of providing a fully dynamic bitmap processing pipeline, WPF bakes support for common scenarios into the BitmapImage class itself. There are various properties on BitmapImage you can set, and then BitmapImage will construct the chain of appropriate WIC bitmap sources for you when the download completes. Indeed, BitmapImage is your one-stop shop for assembling the most common standard WIC components. Depending on the properties specified, BitmapImage will assemble the following chain:

WPF is also very careful about running arbitrary code on the render thread. By caching the result of bitmap processing, the heavy work (such as decoding the bitmap) is performed on the UI thread, and not on the render thread.

Even though the WPF imaging classes are primarily designed for static image processing, there are a few imaging classes designed for dynamic bitmap scenarios: WriteableBitmap, RenderTargetBitmap, and an InteropBitmap created around a shared memory section. These classes are specially handled by WPF for dynamic content. In our forth-comming 3.5 SP1 release, the WriteableBitmap implementation has been substantially improved to provide a reasonably efficient, double-buffered, constant-memory solution for dynamically updating the contents of a bitmap.

Custom BitmapSources

While most of the imaging classes are sealed, WPF does actually support external classes that derive from BitmapSource. In this case, WPF will create an internal implementation of IWICBitmapSource that simply calls the virtual methods on BitmapSource. WPF adds some additional virtuals, including overloads of CopyPixels, and some events for downloading and decoding. All derived classes must override all of these virtuals, regardless of whether or not you need them. In particular, the derived class must supply an implementation for:

virtual PixelFormat Format {get;}
virtual int PixelWidth {get;}
virtual int PixelHeight {get;}
virtual double DpiX {get;}
virtual double DpiY {get;}
virtual BitmapPalette Palette {get;}
virtual void CopyPixels(Int32Rect sourceRect, Array pixels, int stride, int offset);
virtual void CopyPixels(Array pixels, int stride, int offset);
virtual void CopyPixels(Int32Rect sourceRect, IntPtr buffer, int bufferSize, int stride);
Important note: because your custom bitmap class has to override the CopyPixels method that handles a raw memory pointer it can only run in full trust.
virtual bool IsDownloading {get;}
virtual event EventHandler DownloadCompleted {add; remove;}
virtual event EventHandler<DownloadProgressEventArgs> DownloadProgress {add; remove;}
virtual event EventHandler<ExceptionEventArgs> DownloadFailed {add; remove}
virtual event EventHandler<ExceptionEventArgs> DecodeFailed {add; remove;}

Introducing Microsoft.DwayneNeed.Media.Imaging

Most code from my blog gets posted to my codeplex site: http://www.codeplex.com/MicrosoftDwayneNeed Feel free to download it and play with it, or even just link it into an app if you find it useful. For this blog I have added some custom bitmap classes:

CustomBitmap
This class implements the required virtuals from BitmapSource and provides reasonable default values. For example, the DPI is returned as 96.0, the resolution is 0 x 0, the format is Bgr32, etc. The three flavors of CopyPixels are sealed and redirected to a single CopyPixelsCore method. This is the main method you will need to override to implement your logic. There are some virtual members that make no sense at this level, such as IsDownloading and the various downloading and decoding events. CustomBitmap simply provides storage for the event handlers and provides a protected method for raising the events. You will need to call these protected raise methods if you actually implement downloading or decoding functionality. A BitmapSource inherits from Freezable (eventually), and the freezable contract can be tedious to implement correctly. CustomBitmap implements CreateInstanceCore by using reflection to create a new instance of itself. This should be sufficient for most scenarios, and you don't need to provide an implementation in your derived classes unless you want to provide a more performant implementation or if you need to call a non-default constructor. Further, the four "copy" virtual functions CloneCore, CloneCurrentValueCore, GetAsFrozenCore, and GetCurrentValueAsFrozenCore are all sealed and redirected to a single CopyCore method. You will need to override this method if you have state that is not stored in DependencyProperties and that needs to be copied to freshly cloned instances. You will also need to override FreezeCore if your additional state needs to be made readonly when the instance is frozen. If your state is stored in DependencyProperties, the base Freezable implementation will correctly copy and freeze it as needed.

ChainedBitmap
This class represents a custom bitmap that will be chained such that it processes the output of another bitmap source. The default implementation of the BitmapSource virtual members is to delegate to the chained bitmap source. This makes sense for most properties like DpiX, DpiY, PixelWidth, PixelHeight, etc, as in many scenarios these properties are the same for the entire chain of bitmap sources. However, derived classes should pay special attention to the Format property. Many bitmap processors only support a limited number of pixel formats, and they should return this for the Format property. ChainedBitmap will take care of converting the pixel format as needed in the base implementation of CopyPixels. ChainedBitmap also adds handlers to the download and decode events of the chained bitmap source and raises its own events in response. This allows a download or decode event to propagate through the chain of bitmap sources.

ColorKeyBitmap
This class processes a source and produces Bgra32 bits, where all source pixels are opaque except for one color that is transparent. You can specify this color explicitly be setting the TransparentColor property. If this property is not set, the color of the pixel at (0,0) will be used as the transparent color. For simplicity, we only process Bgra32 formatted bitmaps, anything else is converted to Bgra32. Converting a pixel format that does not have an alpha channel to Bgra32 simply creates an opaque alpha value. ColorKeyBitmap then changes the alpha channel for pixels that match the transparent color.

SepiaBitmap
This class processes a source and converts the image to a sepia color scheme. The algorithm is taken from http://msdn.microsoft.com/en-us/magazine/cc163866.aspx, and operates in the linear scRGB color space. For simplicity, we only process Bgr32 or Bgra32 formatted bitmaps, anything else is converted to one of those formats. Bgr32 and Bgra32 share the same memory layout for the RGB channels.

GrayscaleBitmap
This class processes a source and converts the image to a grayscale color scheme. The algorithm is taken from http://en.wikipedia.org/wiki/Grayscale, and operates in the linear scRGB color space. For simplicity, we only process Bgr32 or Bgra32 formatted bitmaps, anything else is converted to one of those formats. Bgr32 and Bgra32 share the same memory layout for the RGB channels.

Demo

I created a simple demo and included it on the codeplex site. It simply allows you to select a URI (I provide a list of images from www.microsoft.com, or you can enter your own), and then chain together up to two of the custom bitmap sources. The app handles the case where the BitmapImage may still be downloading, and defers creating the chain until the content is ready. Make your own CustomBitmap classes and try them out!

Stay Tuned...

Next we'll turn our attention to addressing the static processing limitations of the built-in WPF imaging classes.

Blurry Bitmaps

2007-10-06T01:59:00Z

Background: resolution independence

WPF was designed from the start to be resolution independent. So instead of designing your UI in terms of pixels, you are encouraged to use a physical measuring unit (like inches). When you specify coordinates in WPF, you are actually using "measure units", which are defined to be 1/96th of an inch. This was chosen because 96 DPI monitor settings are very common.

When WPF renders to the screen, it takes into account the system DPI, which you can set from the display control panel. Of course, one problem with this is that no one ever sets this correctly. When was the last time you took out a ruler, measured your screen, then divided your resolution by the result? Another reason people don't set the DPI correctly is that Windows doesn't always look good at non-standard DPI settings. This is something we will be improving in the future. Of course, since you have to recalculate the DPI settings everytime you adjust your resolution, it would be best if the hardware could just report it. For these reasons, WPF may not actually be able to draw a real inch on the screen, but it should come reasonably close.

Another aspect of resolution independence is being able to draw with more precision than pixels. If you recall the GDI APIs, drawing calls specified their coordinates as integers, which match the pixel grid. WPF specifies its coordinates using floating point numbers, so you can easily specify coordinates like (1.2,5.234). This is sometimes referred to as sub-pixel positioning. Of course, what does it mean to render smaller than a pixel, given that each pixel has a single final color? The answer is that we blend all of the contributors to a given pixel. You can think of this as a weighted average, so that if one primitive covers most of a pixel, it will contribute more than other primitives that cover very little of the pixel. But blending loses detail, so this can be very noticeable.

Bitmaps: what size should they be?

Bitmaps are often produced by designers using high-end editing tools. Presume that the goal is to preserve the designer's intent, what size should we display the image at? If the designer made a 96x96 bitmap, on a 96 DPI machine it will be 1 inch by 1 inch. But if you show that bitmap on a super-high-end 300 DPI monitor (I've seen one, its awesome), then the bitmap would be less than a centimeter in each dimension. So, did the designer mean 96 pixels, or one inch? Interestingly, modern image formats allow you to embed the designer's intended DPI into the file. WPF will utilize this information, and scale the bitmap so that it ends up being the desired size. However, here we have a problem - WPF always presumes the designer meant the image to be a certain physical size. Often designers mean for the image to be a certain pixel size - after all, they are carefully assigning colors to each pixel in the image. Its not the designer's intent to be a certain physical size, but the tools they use often stamp some DPI setting into the file anyways (commonly, some standard DPI setting like 72 or 96). WPF sees this DPI information and scales the image to match.

One problem: pixel scaling

Bitmaps are difficult because they encode "high frequency" information in the form of per-pixel color information. If the bitmap is displayed at a slightly different size, the result can be very blurry. I won't bore you with lots of examples, but consider this one. Say you have a bitmap that is 3x3 pixels. It looks like this:

Now, if we scale this just a bit, such that the image will fit into 4x4 pixels, the result suffers from the fact that we can't represent each source pixel equally in the destination grid. In fact, each original pixel "expands" by 1/3. But of course, the final pixel can only have one color, so we have to combine overlapping samples. We can try various blending techniques, but they all have problems. The result will look something like this:

The human eye can easily detect this artifact, and we tend to associate the result with being out of focus, or blurry. It can be very irritating.

Another problem: pixel alignment

As I mentioned in the background section, WPF can render with sub-pixel precision. What happens if you render the original 3x3 bitmap at the correct size, but at pixel coordinates (0.33,0.33)? The solid red line ends up straddling two pixel columns, and the best we can do is to blend the contributions, which will result in nearly the same result as the previously described scale.

SnapsToDevicePixels

WPF anticipated that there would be cases where people wanted to align with the pixel grid instead of using sub-pixel precision. You can set the SnapsToDevicePixels property on any UIElement. This will cause us to try and render to the pixel grid, but there are quite a few cases that don't work - including images. We will be looking to improve this in the future.

Now what?

All is not lost. Even though WPF wants really badly to be resolution independent, we can force it to be pixel aligned ourselves. In fact, this isn't even very hard. We just need to do two things:

Size ourselves to real pixel sizes.
Position ourselves on pixel boundaries.

For sizing, we can easily participate in the measure pass, and return a measure size that is equivalent to the desired pixel size. The transform used to factor in the system DPI is provided for us in CompositionTarget.TransformFromDevice. The pixel size is in "device" coordinates, and we transform from the device into measure units. Easy enough.

For positioning, we could participate in the arrange pass, and maybe apply a render transform to ourselves to offset appropriately. However, for this example I'm focusing on bitmaps, so I choose to participate in rendering and pass the appropriate offset to by DrawImage call. Either way would work, but the real problem is that we need to know when anything might shift our position. The WPF layout system is very conservative, meaning that it tries to do as little work as possible. Once you have been measured and arranged, you won't get called again unless your individual layout state is invalidated. If some parent decides to shift you around a little bit, you will most likely not be laid out again. I work around this by subscribing to the LayoutUpdated event. Even though this event is an instance event on UIElement, it actually gets raised anytime any layout activity happens. Lucky for us, this is pretty much what we want.

Bitmap class

The code included below introduces a new class called Bitmap. Bitmap is a replacement for Image, but instead of displaying any image source, it only displays bitmap sources. This lets me access the PixelWidth and PixelHeight properties for determining the appropriate size. The important aspects of this class are:

Derived from UIElement instead of FrameworkElement because I don’t want things like MinWidth, MaxWidth, or even Width.
Bitmap.Source can be set to any BitmapSource.
When measured, it will return the appropriate measure units to display the bitmap’s PixelWidth and PixelHeight
When rendered, it will offset the image it draws to align with the pixel grid
Whenever layout updates, it checks to see if it needs to re-render to align to the pixel grid again.

Its a pretty straight-forward class, check out the source code for the details.

The sample application

Included in the project is a very simple sample application. I have embedded a number of small bitmaps with high-frequency data in them. Further, I encoded the bitmaps are various designer DPIs. In the app, the top stack panel contains instances of Bitmap. The bottom stack panel contains instances of Image. You can clearly see how Image responds to the bitmap DPI, while Bitmap does not. For fun, you can use the arrow keys which will adjust a render transform on the root. Bitmap will snap to the nearest pixel, while Image will use sub-pixel positioning.

Is there a down side?

Of course. By using the pixel size of the bitmap, the bitmap will look very different on different machines. The 300 DPI super monitor will make your bitmap look very small. Further, since we align to the pixel boundary, animating will cause the bitmap to jump by full pixels. Sub-pixel positioning is very good for animations. Finally, you can get gaps between elements due to rounding to the pixel grid. If you can live with these limitations, then maybe this Bitmap class will be useful to you.

Multithreaded UI: HostVisual

2007-04-26T21:47:00Z

Background: The WPF Threading Model

In general, objects in WPF can only be accessed from the thread that created them. Sometimes this restriction is confused with the UI thread, but this is not true, and it is perfectly fine for objects to live on other threads. But it is not generally possible to create an object on one thread, and access it from another. In almost all cases this will result in an InvalidOperationException, stating that “The calling thread cannot access this object because a different thread owns it.”

Freezables

Of course, there are exceptions to this restriction. A familiar one is the Freezable class. Freezable objects can be frozen, at which point they become read-only and we lift the single-thread restriction. Examples of frozen Freezables include the standard brushes, available from the Brushes class. These brushes can be used on any thread at any time.

This is incredibly useful, and used for all of the graphics primitive resources (pens, brushes, transforms, etc). You can even derive your own types from Freezable and play by the same rules.

Separate Windows

Unfortunately, that read-only restriction can be a real problem. There are many scenarios where we want to run separate pieces of the UI on separate threads. If these pieces of UI are independent from each other, you can host them in separate windows, and run the windows on separate threads. This can be a reasonable solution for some scenarios, especially scenarios where the separate threads can be run in independent top-level windows. The big restriction for this approach is that the graphics from one cannot be composited with the graphics of the other. So while you could use a child window for the other thread’s UI, the system will just render one on top of the other. You cannot have transparency, you cannot use one as a brush for 3D content, you can’t draw over it, etc.

HostVisual

If your scenario doesn’t require interactivity (meaning input), then there is another option that WPF provides: HostVisual. This option leverages the powerful composition engine in WPF that is already capable of aggregating the rendering primitives from multiple threads into one scene. The element tree owned by the worker thread is rendered into its own composition target (called VisualTarget), and the results are composed into the HostVisual owned by the UI thread.

Issue #1: Hosting a Visual in XAML

The first problem to solve is that the HostVisual class derives from Visual. I can't use an existing panel, such as Border, to host this visual. Border derives from Decorator, which is the standard base class for panels that have a single child. Unfortunately, the child is strongly typed to be a UIElement. I have to use a HostVisual, which does not derive from UIElement. There is no built-in way that I know of to place a Visual as a child of one of the standard elements (such as Border, Grid, Canvas, etc). So we make our own:

[ContentProperty("Child")]

public class VisualWrapper : FrameworkElement

{

public Visual Child

{

get

{

return _child;

}

set

{

if (_child != null)

{

RemoveVisualChild(_child);

}

_child = value;

if (_child != null)

{

AddVisualChild(_child);

}

protected override Visual GetVisualChild(int index)

{

if (_child != null && index == 0)

{

return _child;

}

else

{

throw new ArgumentOutOfRangeException("index");

}

protected override int VisualChildrenCount

{

get

{

return _child != null ? 1 : 0;

}

private Visual _child;

}

Issue #2: Layout and the Loaded event

WPF provides a very convenient event called “Loaded”. This event basically signals when an element has been fully initialized, measured, arranged, rendered, and plugged into a presentation source (such as a window). Many elements use this event, including the MediaElement, but sadly this event is not raised for element trees that are not plugged into a presentation source, and displaying an element tree through the HostVisual/VisualTarget doesn’t count. So work around this, we make our own presentation source and use it to root the element tree that the worker thread will own. This immediately leads into another problem: layout is suspended on all elements until resumed by a presentation source. Unfortunately the official mechanism to do this is internal, so the best we can do is to explicitly measure and arrange the root element. Thus we have our VisualTargetPresentationSource class:

public class VisualTargetPresentationSource : PresentationSource

{

public VisualTargetPresentationSource(HostVisual hostVisual)

{

_visualTarget = new VisualTarget(hostVisual);

}

public override Visual RootVisual

{

get

{

return _visualTarget.RootVisual;

}

set

{

Visual oldRoot = _visualTarget.RootVisual;

// Set the root visual of the VisualTarget. This visual will

// now be used to visually compose the scene.

_visualTarget.RootVisual = value;

// Tell the PresentationSource that the root visual has

// changed. This kicks off a bunch of stuff like the

// Loaded event.

RootChanged(oldRoot, value);

// Kickoff layout...

UIElement rootElement = value as UIElement;

if (rootElement != null)

{

rootElement.Measure(new Size(Double.PositiveInfinity,

Double.PositiveInfinity));

rootElement.Arrange(new Rect(rootElement.DesiredSize));

}

protected override CompositionTarget GetCompositionTargetCore()

{

return _visualTarget;

}

public override bool IsDisposed

{

get

{

// We don't support disposing this object.

return false;

}

private VisualTarget _visualTarget;

}

Background Threads

It is easy to make threads in C#. One trick to be aware of is that you must mark the thread as being a “background” thread, otherwise the application will keep running as long as those threads are alive. Also remember that parts of WPF require that its threads to be initialized for COM’s “Single Threaded Apartment”. All of this is easy enough to do, and you’ll see this code later on:

Thread thread = new Thread(/*…*/);

thread.ApartmentState = ApartmentState.STA;

thread.IsBackground = true;
thread.Start(/*…*/);

The Demo

The demo will be a grid showing 3 movies, each rendered on a different background thread. The demo is very simple, but hopefully gets the salient points across. It is certainly possible to communicate from the UI thread to the worker threads, and given that all threads have their own dispatchers this is actually really easy, but this demo does not do that. Interested readers should consult the documentation for Dispatcher.BeginInvoke.

The XAML

The XAML simply defines a grid with 3 columns and puts a VisualWrapper in each column. This is where we will be putting the HostVisual from code. It would be cool to build a class that automatically displays its content on another thread, but we would have to change the parser to support switching threads since the thread an object is created on is the thread that owns that object. Anyways, the simple XAML is:

<Window x:Class="VisualTargetDemo.Window1"

xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"

xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"

xmlns:local="clr-namespace:VisualTargetDemo"

Title="VisualTargetDemo"

SizeToContent="WidthAndHeight"

Loaded="OnLoaded"

<Grid>

<Grid.ColumnDefinitions>

</Grid.ColumnDefinitions>

<local:VisualWrapper Grid.Column="0"

Width="200" Height="100" x:Name="Player1"/>

<local:VisualWrapper Grid.Column="1"

Width="200" Height="100" x:Name="Player2"/>

<local:VisualWrapper Grid.Column="2"

Width="200" Height="100" x:Name="Player3"/>

</Grid>

</Window>

The Code

The code is also pretty simple. For each of the 3 players, it creates a HostVisual on the UI thread, then spins up a background thread, creates a MediaElement, puts it inside a VisualTarget (which points back to the HostVisual), and puts it all inside our hacky VisualTargetPresentationSource.

public partial class Window1 : System.Windows.Window

{

public Window1()

{

InitializeComponent();

}

private void OnLoaded(object sender, RoutedEventArgs e)

{

Player1.Child = CreateMediaElementOnWorkerThread();

Player2.Child = CreateMediaElementOnWorkerThread();

Player3.Child = CreateMediaElementOnWorkerThread();

}

private HostVisual CreateMediaElementOnWorkerThread()

{

// Create the HostVisual that will "contain" the VisualTarget

// on the worker thread.

HostVisual hostVisual = new HostVisual();

// Spin up a worker thread, and pass it the HostVisual that it

// should be part of.

Thread thread = new Thread(new ParameterizedThreadStart(MediaWorkerThread));

thread.ApartmentState = ApartmentState.STA;

thread.IsBackground = true;

thread.Start(hostVisual);

// Wait for the worker thread to spin up and create the VisualTarget.

s_event.WaitOne();

return hostVisual;

}

private FrameworkElement CreateMediaElement()

{

// Create a MediaElement, and give it some video content.

MediaElement mediaElement = new MediaElement();

mediaElement.BeginInit();

mediaElement.Source = new Uri("http://download.microsoft.com/download/2/C/4/2C433161-F56C-4BAB-BBC5-B8C6F240AFCC/SL_0410_448x256_300kb_2passCBR.wmv?amp;clcid=0x409");

mediaElement.Width = 200;

mediaElement.Height = 100;

mediaElement.EndInit();

return mediaElement;

}

private void MediaWorkerThread(object arg)

{

// Create the VisualTargetPresentationSource and then signal the

// calling thread, so that it can continue without waiting for us.

HostVisual hostVisual = (HostVisual)arg;

VisualTargetPresentationSource visualTargetPS = new VisualTargetPresentationSource(hostVisual);

s_event.Set();

// Create a MediaElement and use it as the root visual for the

// VisualTarget.

visualTargetPS.RootVisual = CreateMediaElement();

// Run a dispatcher for this worker thread. This is the central

// processing loop for WPF.

System.Windows.Threading.Dispatcher.Run();

}

private static AutoResetEvent s_event = new AutoResetEvent(false);

}

Summary

In this demo I’ve shown how to use the HostVisual and VisualTarget classes to compose pieces of UI from different threads. There are some limitations: namely that the UI owned by the worker threads do not receive input events. There were also some annoyances we had to work around along the way, but those proved to be fairly minimal.

The source code for this project is now included in my codeplex site: http://www.codeplex.com/MicrosoftDwayneNeed