Greg Schechter's Blog : DWM

The word is - "Java on Vista: Yes, it Works"

Greg Schechter — Mon, 09 Oct 2006 23:30:00 GMT

Some folks have noticed that when they run Java applets in the browser, that Aero and desktop composition would get disabled. This was due to Java locking the primary buffer, and when that happens, the DWM no longer has access to the resources needed to draw the desktop, therefore we disable desktop composition. Sun has since addressed this issue, as you can see in this post in Chet Haase's blog. Chet also provides lots more detail about what was up, what version it's fixed in, and when it will be available - or, as BubbleMan insists on saying, availabubble.

And while we're on the topic, earlier versions of Apple's QuickTime/iTunes also exhibited this primary-locking behavior. The lastest updates to QuickTime/iTunes no longer does this.

A Couple of Cool Uses of the DWM Thumbnail APIs

Greg Schechter — Thu, 28 Sep 2006 09:34:00 GMT

I've seen a couple of cool uses in the past week of the DWM Thumbnail API (described in my previous post) and wanted to share them here.

"Simon on the .NET" wrote a great-looking Exposé-like mini-app that, when you press F12, shows a representation of all the desktop windows in a grid. The original desktop is shown behind the grid. The thumbnails are, as expected, live and updating (though they don't accept input, as is true with DWM thumbnails in general).

Douglas Stockwell wrote a cool WPF application that PInvoke's to the thumbnail APIs and populates a WPF ItemsControl with thumbnails of windows as you click on them. As it stands, it has no particular purpose, but it shows some interesting integration between WPF and DWM thumbnails.

APIs in the Desktop Window Manager

Greg Schechter — Thu, 14 Sep 2006 11:33:00 GMT

For the most part, the Vista Desktop Window Manager is an end-user feature. However, because it so fundamentally changes the game of how the desktop gets composed, there was both the opportunity and the requirement to introduce a relatively small number of APIs that impact how the DWM operates, and allows the programmer to take advantage of DWM features. This post will discuss the functionality and APIs that we expose at a fairly high level, and reference MSDN documentation for drilling further in.

The DWM APIs are housed in dwmapi.dll, with declarations in dwmapi.h. dwmapi.dll is a very thin DLL that in almost all instances, simply LPC's over to the dwm.exe process in order to fulfill the API request. The DWM-related APIs fall into a number of categories:

Information querying and setting functionality
Scheduling and media presentation
Thumbnail registration manipulation
Client-area blur/glass manipulation

You can get more information about each of these in MSDN by searching at http://windowssdk.msdn.microsoft.com. The point of this post is to try to tell a coherent story about the set of them, that hopefully contains some of the motivation.

Information Querying and Setting

There are a number of functions that provide basic information for querying and setting certain window and system attributes:

DwmEnableComposition - enables and disables composition.
DwmIsCompositionEnabled - returns whether or not composition is occurring.
DwmGetColorizationColor - returns the current window colorization information.
Dwm{Set,Get}WindowAttribute - controls various attributes on windows, including the policy for rendering the non-client area; policies on the "thumbnail representation" for a window -- whether it should truly be a thumbnail, or be an iconic representation; how Flip3D treats a window; enable/disable DWM transitions for a window; policy controlling client's NC area painting; access to caption button bounds; etc.

Also, there are a number of Windows messages sent under different circumstances:

WM_DWMCOLORIZATIONCOLORCHANGED -- when the colorization color changes, apps can react to it as they see fit (for instance, if they have elements of their client area colored based on the colorization color.
WM_DWMCOMPOSITIONCHANGED -- when composition is turned on or off, this message is broadcast
WM_DWMNCRENDERINGCHANGED -- when the non-client area rendering status of a window has changed
WM_DWMWINDOWMAXIMIZEDCHANGE - windows can register to hear about when other windows have been maximized. This is useful for, for instance, the TaskBar or the SideBar which go opaque when other windows get maximized.

Scheduling and Media Presentation

These functions allow for finer grained control of exactly when the desktop is composited and presented. This is typically needed by media and video-playback applications for whom the DWM is running asynchronously from their own presentation schedule, which can lead to sampling artifacts if not tightly controlled.

DwmEnableMMCSS -- allows the DWM to participate in the Multimedia Class Schedule Service to have stronger control over the use of CPU resources in conjunction with another app using MMCSS (typically for video playback)
DwmGetCompositionTimingInfo -- provides information about the rate at which the DWM is composing, how many frames are completed, how many are pending, dropped frames, etc.
DwmModifyPreviousDxFrameDuration, DwmSetDxFrameDuration, DwmSetPresentParameters -- these allow applications to queue up multiple DirectX surfaces for presentation and precisely control when they're to be presented. They're typically used by video presentation applications to counteract the latency that's inherent in a compositing system.

Thumbnail Registration and Manipulation

Thumbnails in the DWM are not static snapshots of a window. They're dynamic, constant connections between a thumbnail source window, and a place on a target window that that source window will get a "thumbnail rendering". Thus, a thumbnail is really a registration between a source HWND and a destination HWND. Once a thumbnail is registered, one can update it's properties like the source and destination rectangles, opacity, visibility, and whether the source's NC area should be part of the thumbnail.

The functions available for thumbnail manipulation are:

DwmRegisterThumbnail -- establish a relationship between a source HWND and a destination HWND, and return a new thumbnail ID.
DwmUnregisterThumbnail -- end the relationship represented by the thumbnail ID.
DwmUpdateThumbnailProperties -- update source/destination rectangles, opacity, visibility, etc.
DwmQueryThumbnailSourceSize -- helper function ot figure out the source window size from a thumbnail, which is otherwise painful to figure out on your own.

The way thumbnails are implemented is effectively via the same VisualBrush mechanism that we have in WPF. The DwmRegisterThumbnail API effectively creates a link into a node representing a window in the visual tree that makes up the desktop (discussed in this post), and references that node from the thumbnail target HWND. Thus when the thumbnail source is a live video, the target HWND is rendered by rendering that live video there as well. So these thumbnails are very much "live".

I should also point out here that things like Flip3D cannot be built using the thumbnail API. That's because thumbnails are always rendered head on in their target window in 2D.

Because of how thumbnails work as described here, it's possible to make interesting topologies of thumbnails -- for instance, having a thumbnail of another thumbnail, and nested further than that, is perfectly viable (though cycles are prohibited and checked for). Having an application show many thumbnails in a window is just fine and in fact is exactly what Alt-Tab does in Vista Aero. Thumbnails can also be any size -- in fact, they can even be larger than their original source.

Even though the Client-Area Blur/Glass APIs described below have gotten more initial attention in the community, I'm hopeful that the thumbnail APIs will be put to really interesting uses. They really hold the promise of some very, very interesting uses. So if you're out there reading this -- get inspired and do something cool with these and let me know... I'd love to post about it.

Client-Area Blurred Glass Manipulation

Lastly, there's the set of APIs for adding the blurred glass affect to the client area of your windows. Many windows in Vista use this -- the Explorer and IE for their bread-crumb bar. Windows Media Player for its transport controls. Photo Gallery for its manipulation controls. The Sidebar Gadget Picker puts its Gadgets on a big pane of glass. The Alt-Tab UI is on a big pane of glass. There are two primary means of getting blurred glass in your client area. I'll explain with the first two function definitions:

DwmExtendFrameIntoClientArea - this insets from the margins and makes a border inset within the client area merge seamlessly with the blurred glass of the window frame.
DwmEnableBlurBehindWindow - this allows a GDI HRGN to be specified on a window, and the DWM will blur what's behind that region.

In both of these cases, it's the complete responsibility of the application to ensure that the areas being presented that will now have blurred glass behind them are rendered with the proper transparency values in the alpha channel. Unfortunately, very few GDI functions respect alpha (really the "alpha blit" function in GDI is the only one that reliably does). Thus you may be forced to render your GDI content offscreen, use memory operations to clean up the alpha channel, and then alpha blit back.

There's been a fair amount written in the community around how to use Client-Area Blurred Glass in your applications. Here are some choice examples:

Tim Sneath shows how to do client area glass in a Windows Forms application.
Adam Nathan shows how to do client area glass in a WPF application.
Martin Grayson provides a WPF example as well.
Daniel Moth also provides a few more uses.

It's very important though to realize and honor the performance implications of using blurred glass in more parts of your window. Since you can see through, updates in windows behind that otherwise wouldn't have caused repaints now do cause repaints. And the blurring itself is a somewhat expensive operation that happens in the GPU. So blurred glass should really be used sparingly in applications, and to good design purpose.

Responding to Comments on "High DPI Support in Windows Aero Vista"

Greg Schechter — Thu, 14 Sep 2006 10:17:00 GMT

I received a number of great comments and questions on my last post about High DPI Support in Windows Aero Vista.

Adrian makes a very solid point about applications that do in fact work correctly at high DPI, and scale appropriately, and in general are very good "DPI citizens", however, they haven't set the manifest for Vista to claim that they're DPI aware (and of course they haven't, since Vista wasn't around when they wrote their apps). These are applications that Vista will think are not DPI-aware, and subject them to bitmap scaling rather than letting them render at the true resolution, resulting in a worse experience. This is an issue that we've struggled with on the team, since we don't want to make the experience worse for apps that have gone to this trouble, yet we do want to provide the benefit for the large number of apps that haven't. Based on these conversations, we've made some changes post-RC1 that we hope will ease this issue substantially.

Chris Nahr makes a general comment about font scaling, and Joe asks about ClearType. In both of these cases, rendering windows scaled will result in blurrier, and possibly color fringed text, and without hinting at the larger sizes. But that's all because the application doesn't do the high DPI aware re-rendering themselves. Those apps that do (see above) we certainly want them to continue to do. The High DPI Scaling feature is really targeted at the very large class of applications that don't do this, for which at very large DPI settings, lack of font hinting or potential color fringing is a lesser evil than tiny rendering.

Raiker mentions scaling on vector interfaces good, scaling on bitmaps bad. We couldn't agree more. However, the reality is that most apps are not built on vector-based systems, and those that are can be DPI-aware and scale in a DPI-aware form providing all the benefits of a vector-based system. This feature is really for the very large class of apps that don't do DPI-aware rendering and are not vector based. If you look at Windows Presentation Foundation applications, for instance, those are all DPI aware apps, and will do the full-on vector-scaling thing without being bitmap-scaled by the High DPI feature.

Finally, Michael Giagnocavo talks about Vista Beta 2 DPI scaling being unusable. Many issues have been fixed then and you'll find a much better experience in RTM and in the RC1 that's out now.

High DPI Support in Windows Vista Aero

Greg Schechter — Mon, 07 Aug 2006 10:42:00 GMT

A good amount of ink has been spilled on this blog talking about all the cost, nuance, impact, and techniques we go through to get a composited desktop. Less ink is spilled on the benefits of the composited desktop. Those were most broadly covered in this initial post. I'd like to expand on one such benefit here -- High DPI Support (or High Resolution Support).

Monitor resolutions are going up while prices are going down. In general, pixel density is going up as well. Both on laptops that are regularly 120 DPI and 144 DPI these days (as opposed to the typical desktop experience of 96 DPI), and on desktops that are getting bigger and denser monitors. This can make for crisper, less jagged display of content. However, in order for this to work, applications typically need to do work to deal with different display resolutions. (This is not always true... for instance, Windows Presentation Foundation applications are natively resolution-aware, and the application developer needn't worry about it.) When applications aren't written to adapt to different resolutions, they just start to look smaller and smaller on higher and higher resolution monitors.

The seminal article for how to write your GDI/GDI+ applications to be DPI aware was written way back in 2001 by my colleague Nick Kramer, and can be found here. This is ultimately the best approach an application can take. However, not all applications do this, and yet users of course use these applications while they're running in a higher DPI environment. This is where the composited desktop re-enters the picture. As you recall, when the DWM is running, applications render to an offscreen bitmap. When the DWM recognizes that an app is not DPI aware and just rendering as it always does, but that the desktop is set to a non-standard DPI, then the DWM goes ahead and renders the window at a larger size.

This "rendering at a larger size" results in the app being somewhat fuzzier and not as crisp as if it were being rendered natively at the correct resolution. However, it's the "correct" size for everything else on the desktop, and typically represents the much better option between rendering at the right size somewhat fuzzily; or crisply but at a size much too small.

For the purposes of this blog, there are some interesting tidbits about the High DPI rendering that are worth discussing:

As this earlier post describes, the client area of the window is just a bitmap. Because of the way the DWM works and how it sits atop WPF technology and thus atop DirectX, scaling the client area is really just a matter of modifying a scale transform on the appropriate node of the "visual tree" that describes the desktop. This notion was described here.
When that transform scaling is performed on the visual tree, ultimately in DirectX, this results in performing a texturing operation with the same sized texture but a larger mesh (the target client area), resulting in stretching the client area bitmap.
The Non-Client Area (the window frame) is not scaled uniformly with the client area. As in non-DPI scaled situations, it's "painted" separately. It's not simply subjected to bitmap scaling because for the frame, we can natively scale it, and choose elements appropriate for the target resolution, render caption text appropriate for the target resolution, etc.
While one can imagine any scale factor coming about as a function of the target DPI resulting in windows that land off of pixel boundaries and represent fractional pixel contribution. However, we avoid this and clamp our scaling so that we always remain on pixel boundaries and remain as crisp as possible, within the confines of the graphical filtering algorithm in place.

More details for the app developer:

The preferred mechanism by which an application will be able to declare itself High DPI aware under Windows Vista will be via the manifest. Look for upcoming MSDN documentation discussing how to do this. In the event where a manifest isn't appropriate/available (for instance, when writing a library or framework), the the library should ensure that the "SetProcessDPIAware" API is called before creating any UI.
Nick writes a bit more about High DPI in the context of USER and GDI compatability.

How underlying WPF concepts and technology are being used in the DWM

Greg Schechter — Fri, 09 Jun 2006 11:24:00 GMT

In the earlier posts I've done on the DWM, there's been a hint of the relationship between it and the Windows Presentation Foundation (WPF, aka Avalon). This post delves deeper into that relationship. Because of core OS component requirements (the desktop itself of course being a core OS component) regarding the use of managed code, the DWM itself is not a managed application that directly uses WPF. However, in almost all other ways, the DWM really can and should be thought of as a WPF application. Most importantly, it does use the same native composition and rendering module, milcore.dll, used by WPF itself, and it uses it in ways very similar to how WPF uses it.

By the way, I recently did this Channel9 video about aspects of what's discussed here.

Before going further, a quick detour on how WPF applications work. The application author creates, through XAML or code, a tree of elements that becomes the applications "visual tree". This visual tree is communicated to the WPF rendering thread as a data structure, and the rendering thread now is responsible for traversing this visual tree and drawing its pixels. When the application wants to make changes to what is shown, the code they execute results in edits to the visual tree, and the rendering thread re-renders what needs to be re-rendered. This process of walking this tree and issuing rendering calls is known as "composition". In a WPF application, the composition and rendering work are done in the milcore.dll module.

So, that's a WPF application. What about the desktop itself? The DWM models the desktop as a visual tree where each node in the tree is a window. And each of these "window" nodes consists of nodes below it that represent the non-client area (frame) and the client area of the window. It so happens that the client area of the window happens to be a shared DirectX surface that comes from window redirection as described in this earlier post, but from the point of view of the composition engine, the whole desktop is pretty much just another visual tree. (There's no denying, though, that it's not just any old other visual tree... there are certainly some special optimizations we needed to make in order to get the desktop to the level of performance that it needed to be, but we also hope that those optimizations can make their way back for general WPF usage.)

OK, so now that we've gotten the desktop to look like a visual tree, and even with it being a natural fit, the obvious question comes up: Why? Why bother? What do we gain from this? Why not write the DWM straight to DirectX?

There are a number of substantial benefits that come from building on a general compositing and rendering system. Here are some of them:

Remoting support - the DWM itself can be remoted, and that uses the underlying WPF remoting support.
Multiple monitor abstraction - the DWM application needn't concern itself with most of the aspects of how to render across multiple monitors or adapters.
Integration of 3D with 2D - the Flip3D feature and the window transitions use 3D integrated into the 2D visual tree. This support is a key WPF differentiator.
2D tesselated anti-aliased primitives - the underlying WPF technology takes care of this.
Update management - as changes are made to the visual tree based on desktop activity by the user, the write invalidations and change propagation occurs in the visual tree.
Dirty region management - code within the composition engine ensures that only the regions of the display that are dirty need to be re-rendered, which is an absolutely vital performance requirement.
Occlusion culling support - similar to dirty region management, occlusion culling support allows the composition engine not to render content that's changing if it's covered by another part of the visual tree that's opaque.
Scheduling - scheduling of frames, maintenance of frame rate, compensation for variability in frame rate is all part of the scheduling subsystem.

All of these benefits come from the use of WPFs composition system.

I'm 95% sure the following question will come up, so let's get it out of the way here: Is milcore.dll available for public use? The answer is that in the Windows Vista timeframe, it is not exposed as a public API. We are certainly looking into what would be involved in exposing it, and clearly understand that there's interest out there for it.

I'm on TV... and just in time for Mother's Day...

Greg Schechter — Wed, 10 May 2006 09:02:00 GMT

I got to do this fun Channel9 video the other day with Charles Torre and Tim Sneath. In it we talk about some of the fundamentals and architecture of the underlying composition system for Windows Presentation Foundation and for the Desktop Window Manager.

Just in time for Mother's Day... she'll be so proud :-).

Redirecting GDI, DirectX, and WPF applications

Greg Schechter — Wed, 03 May 2006 08:28:00 GMT

As mentioned in earlier posts, by far the most important aspect of the DWM is the fact that application windows are redirected to render offscreen, and then the DWM is responsible for compositing those windows to the screen. So, how exactly does that happen? That's what this post is all about. Redirection is a fairly complex topic, but is completely central to the composited desktop. Thanks to Jevan Saks and Greg Swedberg for reviewing this post and answering some of my own questions here as well.

Before diving into this, I should clarify something that hasn't been brought up in earlier posts: the DWM only redirects top-level HWNDs. Thus, a Multiple Document Interface (MDI) application (Microsoft Management Console, mmc.exe, is a good example of this) will have its overall top-level HWND, with it's internal child HWNDs, composited as a single entity. The application process draws the child HWNDs, and their non-client areas, as it always has.

For the purposes of a discussion on redirection, there are really three types of windows that are of interest: GDI-rendered windows, DirectX-rendered windows, and windows rendered by a mix of DirectX and GDI. Let's discuss these in turn.

GDI-rendered windows

Today and for the near future, most applications use and will continue to use GDI to render their content. Traditionally, GDI applications were notified when a part of their window became unoccluded, and were asked to repaint that portion of the window. Under the DWM, that window is redirected, and the following happens:

A system memory surface the size of the window is allocated and associated with that window.
A video memory surface, in the target DirectX pixel format, is allocated, also the size of the window.
When an application retrieves the GDI DC of an HWND, it no longer is the DC of the primary video buffer, as it is in the non-composited, pre-DWM desktop. Instead, the DC is a DC onto the allocated system memory surface.
GDI operations on that DC then populate the system memory surface.
The system, based on a number of variables, decides to update the video memory surface from the system memory surface at the "right times".
The video memory surface is now up-to-date with the application, and the compositor comes around and uses the video memory surface to composite the desktop from.

There are a few implications of the above that are worth calling out.

Dual buffers per window - yes, it's true that GDI windows have both a system memory and a video memory representation. There is without doubt a memory cost to doing this. One obvious alternative is to simply have a video memory representation and have the GDI redirection mechanism render to that format. There are two primary problems with this. The first is that the formats are not the same, and GDI doesn't support rendering into the DirectX format. Even if that were resolved, the more fundamental issue remains. Many GDI operations (XORs, alpha blending, and text are examples) are read-modify-write operations. To do that to a native video memory surface would involve reading back from video memory into the CPU (and thus into system memory), performing the operation, and then writing back. This is typically a horribly slow and pipeline-stalling operation.
Minimized windows present a special issue. Typically when an application receives a minimization, the surface that it's asked to paint is a nominal size, like 130x30, just enough for shades of the non-client area. If the application updates the system memory surface at this point, and we continue our copying to the video memory surface, then any surface we may have had available to us for Flip3D or for thumbnail rendering is suddenly gone. Instead of doing this, we maintain the video memory surface in its last known state, and thus those "secondary window representations" are far more useful when windows are minimized.

DirectX-rendered windows

Unlike GDI applications, DirectX applications of course can natively render into the DirectX pixel format that the DWM expects. They also have a very clear indication of when they're done rendering due to the requirements that they call Present(). As such, DirectX applications only need a single window buffer to manage their redirection. DirectX window redirection is handled by having the DirectX system, when it's determining what surface to provide the app with to render to, make calls to the DWM in order to share a surface between the DirectX client application process, and the DWM process. This "shared surface" support is unique to DirectX atop the WDDM, and is another key reason why WDDM is an absolute requirement for running the DWM.

When a Present() happens to such a surface, the DWM is notified that there are dirtied surfaces that need to be composited to form the desktop, and that serves as an indication to perform a composition. (It's actually a fair bit more complicated than that, but this description certainly provides the gist of it.)

Certain DirectX-based applications have much more stringent scheduling requirements (for instance, video applications), and there are public APIs provided that allow the application to get a lot more information, and more control, over when they should render based upon the rendering schedule of the desktop compositor. That will be covered more in a future topic.

Finally, WPF (Avalon) applications are DirectX applications, so they render just as the DX applications described above render.

Mixed DirectX and GDI Windows

The other reasonably common rendering to a top level window involves mixing DirectX and GDI. There are two forms of "mixing" here, one is perfectly fine, and the other is problematic.

The form of mixing that is fine is when there is a window tree of the top level HWND and child HWNDs (and further children, etc), where each individual HWND is either rendered by DirectX or by GDI. In this situation, the redirection component of the DWM forms its own "composition tree" where each node in the tree represents a node or a set of "homogenously rendered nodes" in the "window tree" rooted at the top level HWND. Rendering occurs by having each of these render to their own surface, and then compositing this tree of surfaces to the desktop. Thus, mixed DirectX and GDI rendering works well, so long as the boundary between them is at least at the child HWND level.

The form of mixing that doesn't work well is when an application uses DirectX and GDI to target the same HWND. This has never been a supported scenario with DirectX, but there have been scenarios where it has happened to work. Under the DWM, this is much more problematic, because there can be no guarantee of ordering between the DirectX and the GDI rendering. This is most troublesome when GDI and DirectX are not only rendering to the same HWND, but to overlapping areas of the same HWND. As such, this usage pattern is not supported. Note that there is an alternative that can often work for an application -- DirectX is capable of handing back a DC to a DirectX surface, and applications can perform GDI rendering to that DC. From the DWM's perspective, that DirectX surface remains purely rendered by DirectX, and all is well.

Drawing To and Reading From the Screen -- Baaaad!

Lastly, since we're on the redirection topic, one particularly dangerous practice is writing to the screen, either through the use of GetDC(NULL) and writing to that, or attempting to do XOR rubber-band lines, etc. There are two big reasons that writing to the screen is bad:

It's expensive... writing to the screen itself isn't expensive, but it is almost always accompanied by reading from the screen because one typically does read-modify-write operations like XOR when writing to the screen. Reading from the video memory surface is very expensive, requires synchronization with the DWM, and stalls the entire GPU pipe, as well as the DWM application pipe.
It's unpredictable... if you somehow manage to get to the actual primary and write to it, there can be no predictability as to how long what you wrote to the primary will remain on screen. Since the UCE doesn't know about it, it may get cleared in the next frame refresh, or it may persist for a very long time, depending on what else needs to be updated on the screen. (We really don't allow direct writing to the primary anyhow, for that very reason... if you try to access the DirectDraw primary, for instance, the DWM will turn off until the accessing application exits)

The role of the Windows Display Driver Model in the DWM

Greg Schechter — Sun, 02 Apr 2006 10:02:00 GMT

The Problem

Ever since the advent of dedicated graphics processors, even old-school graphics processors that only accelerated GDI blits, the way you would program against them would be similar to how you programmed against the main CPU/memory system before there was virtual memory or interruptible/preemptible processes. That is, you'd have to be sure to directly manage all the video memory yourself, and just count on not being able to have your graphics instructions interrupted. Specifically, DirectX applications have always needed to deal with not getting the video memory they need, or deal with "surface lost" messages from video memory that got kicked out for one reason or another. This puts a major burden on the programmer, and, probably even more importantly, makes for a very poor ecosystem for running multiple video-memory resource intensive applications, because their likelihood of cooperating in a sensible way on resource management is virtually nil.

Well, the DWM is a DirectX application with a couple of unique challenges in this arena:

The memory requirements on the DWM vary widely. That's because they vary directly with the number of windows the user has open, and while there are known typical usage patterns, the user certainly isn't and cannot be limited to N open windows.
The DWM operates in an environment where other DirectX applications do operate. Video playback, WPF applications, windowed games (btw, Vista "inbox" games like Solitaire, etc., are now written in DirectX), etc. In fact, the DWM is responsible for the final presentation of those applications. So it's critical that such DirectX applications "play well together" and play well with the DWM.

The above challenges don't mesh well with the DirectX described in the first paragraph.

Enter WDDM

It's the Windows Display Driver Model (WDDM, formerly known as LDDM) that makes all of this viable. WDDM is the new DirectX driver model for Windows Vista and beyond. From the perspective of the DWM it does three main things:

Virtualizes video memory.
Allows interruptibility of the GPU.
Allows DirectX surfaces to be shared across processes.

The surface sharing feature is key for redirection of DirectX applications, but that's the topic of a later post. Here we're going to discuss the first two. There are other motivators for, and certainly a lot more details on the WDDM, but those aren't as immediately relevant to the DWM as what's discussed here.

Virtualizing Video Memory

With the WDDM, graphics memory is virtualized. This means that just like system memory, if there is a demand for memory and the memory is all allocated, then secondary storage is turned to, and the system manages all the paging algorithms and mechanics for faulting in the secondary storage into the primary storage when it needs to be operated on. In the case of video memory, the primary storage is video memory, and the secondary storage system memory.

In the event that video memory allocation is required, and both video memory and system memory are full, the WDDM and the overall virtual memory system will then turn to disk for video memory surfaces. This is an extremely unusual case, and the performance would suffer dearly in that case, but the point is that the system is sufficiently robust to allow this to occur and for the application to reliably continue.

The upshot of all of this is that applications don't need to be greedy to get all the memory they need, since they won't be guaranteed true video memory anyhow, and they can always be paged out. This brings the goal of a cooperative set of DirectX applications much, much closer to reality. It also means that there are effectively no more "surface lost" messages from DirectX, and no failed allocations.

From the DWM's perspective, this is all absolutely key because the DWM can and will allocate memory, and those memory allocations will be done in conjunction with allocations for other applications on the system, putting the "right" surfaces into the true video memory, and paging in and out as necessary. Now, naturally, this is a little bit of a naive viewpoint, since this is the first generation of this virtualizer, but we're observing it to be doing quite well, and it will keep improving.

Interruptibility of the GPU

So, memory's virtualized, that's good, but what about those little computrons that run around the GPU doing stuff? Can one application's GPU commands be preempted by another application? Prior to WDDM, they could not. With WDDM, they can be. This is referred to as WDDM scheduling, and WDDM arbitrates usage of the GPU, giving computation to the different applications requesting it. In order to do this, WDDM must be able to interrupt a computation going on on the GPU and context switch in a different processes operation. WDDM defines two levels of interruptibility to support this.

Basic Scheduling - this is the granularity of scheduling achievable in DirectX 9 class WDDM drivers and hardware, and means that an individual primitive and an individual shader program cannot be interrupted, and must run to completion before a context switch.
Advanced Scheduling - this is achievable in DirectX 10 class WDDM drivers and hardware, and here the GPU can be interrupted within an individual primitive and within an individual shader program, leading to much finer-grained preemptability. Note that while DX10 supports advanced scheduling, it's not a requirement for DX10 -- that is, only certain hardware will support it.

The Desktop Window Manager uses DirectX 9, and thus Basic Scheduling. So it's possible that an application that makes errant use of the GPU and uses complex shader programs across large primitives can potentially glitch the DWM. We have yet to see such applications, but there no doubt will be some that either do this unintentionally or are built specifically to do this. Nonetheless, we don't believe that this will be a common issue.

Desktop Window Manager Index of Post Topics

Greg Schechter — Sun, 26 Mar 2006 08:09:00 GMT

Here's a list of topics that I have posted on (with active links) or expect to post on (without links) related to the Desktop Window Manager in Windows Vista.

Under the Hood of the Desktop Window Manager -- a technical introduction (and response to comments)
DWM's use of DirectX, GPUs, and hardware acceleration (and response to comments)
The role of the Windows Display Driver Model in the DWM (and response to comments)
Redirecting GDI, DirectX, and WPF applications
How underlying WPF concepts and technology are being used in the DWM
High DPI support (and response to comments)
APIs in the Desktop Window Manager
Rendering and visibility optimizations
Memory usage in the DWM
Power management and battery life
How the DWM paints the window frame and other non-client area
Remoting, Magnification, and Accessibility under the DWM
Best practices for applications under the DWM

If there are specific other topics you're interested in, please comment away.

Responding to Comments from "DWM's use of DirectX, GPUs, and hardware acceleration"

Greg Schechter — Sun, 26 Mar 2006 04:53:00 GMT

My earlier post on "DWM's use of DirectX, GPUs, and hardware acceleration" generated some good comments and questions. I'll use this post to respond.

dzCepheus wonders about end users and developers creating their own effects, for instance, mapping window content onto spinning cylinders. This is definitely something that we've thought about, and are going to look into extensibility in these directions in a future release, but for Windows Vista, this level of extensibility isn't possible. What is possible is more limited -- the ability to extend glass/blur into the window, and the ability to manipulate live thumbnails of windows in your own applications. More on that in a future post.

steamy asks about DX8 cards rendering glass, in light of my comments about DX9 in the post. Not sure what DX8 cards worked, or when, but DX9/WDDM drivers are absolutely a requirement, as is PS 2.0 support.

steamy also wonders about software rendering for the DWM. Given the confluence of a) commoditization of sufficiently performing graphics hardware; b) not wanting to overly tax the CPU and take it away from other applications, and the long term power efficiency of doing these operations in the GPU and not the CPU; c) the requirement for pushing around lots of pixels and needing big texture bandwidth; and d) the importance of maintaining focus and executing well on one rendering path without bringing in others -- we opted away from pursuing a software path here.

Nidonocu asks about XGL for Linux desktops. I don't know a great deal about XGL (other than the fact that a different XGL used to be the way of programming 3D on Sun Microsystems machines back when I worked there in the early-90's), but I have seen some of the videos I've seen showing it in action. For Windows Vista, we've really focused on building the composited desktop, and working out all the kinks to allow two decades worth of Windows applications to run well on such a fundamentally different desktop rendering model. The fruits of the composition labor show off in big quality and usability wins for the desktop, and some choice places we leverage composition -- for instance Flip3D, high DPI support, and window thumbnails. Now that this groundwork is laid, we expect to really be able to leverage it quite a bit more moving forward -- but also to do so in a "UI-responsible" way really heeding the advice of our graphics and UI designers.

GRiNSER talks about other animations and transitions in the Vista GUI. Vista animations and transitions are definitely not limited to the DWM. There are definitely some choice ones in Explorer, and there are many many situations where GDI rendering for these does absolutely fine.

Finally, Princess talks about interaction between the DWM's usage of the GPU and other applications. This is going to be the main topic of my next entry on the WDDM.

DWM's use of DirectX, GPUs, and hardware acceleration

Greg Schechter — Sun, 19 Mar 2006 22:36:00 GMT

For the last few years, both desktop and laptop PCs have been outfitted with increasingly powerful graphics chipsets, including blazingly fast geometry and pixel processing, higher fill rates, and faster and faster bandwidth between system memory and video memory. Further, GPU computational increases has been exceeding Moore's Law for some time, outpacing CPU growth. The beneficiaries of this growth has almost entirely been PC games and the gaming community. However, this processing power has largely sat dormant during typical use of the Windows desktop and typical productity/business/everday applications.

One of the key goals of the Desktop Window Manager in Windows Vista is to harness all this spare graphics compute power, and bring it to bear on making the computing experience better for the majority of users who are not gamers, or at least not always gamers.

In order to do this, the DWM is built upon Microsoft DirectX, specifically Direct3D, which is the uniform way that software in the Windows environment talks to graphics hardware. More precisely, the DWM is actually built directly atop a layer we internally call "milcore", and milcore is built directly atop DirectX. (More on milcore in a future post.) But the end result is that Direct3D textures are created that represent the window content and the window frame, and DWM/milcore is responsible for issuing the proper Direct3D calls that result in composing all of the Direct3D textures to form the desktop.

One specific takeaway from this: the Windows Vista desktop (through the DWM) is a fullscreen Direct3D application.

Fundamentally 3D

You may know, though, that in typical usage of Direct3D, one doesn't directly bit-blit textures to a primary surface. This is also true in the DWM. The window contents aren't being asked to paint directly to the desktop surface. Instead, they're being treated as texture maps on a 3D mesh that is being rendered by Direct3D and the graphics hardware. For example, in the case of the standard desktop view, it so happens that the 3D mesh that the textures are being mapped to is a very simple rectangle (actually, two adjacent triangles), representing the client area of the window.

This is important, because once that's seen as being the case, it's not a far leap at all to understand how the DWM does it's 3D window transitions, or the 3D "Flip3D" view as shown in this initial DWM post. Our window transitions (open, close, minimize, restore), are simply 3D transformations of that rectangle that already exists in 3-space. It would not be much harder to modify any of these transitions to be more complex - for instance to map the window content onto a spinning cylinder for minimize. Thankfully we have a crack team of graphic and UI designers that prevent us from making such UI faux pas.

Pixel Shaders for Blur

One of the signature looks for Windows Vista is the window border's "glass" look. This look allows one to see through the border to what's behind, but also adds some amount of translucency and also blur to what's behind. From a design perspective, this makes window borders feel less heavy and in-your-face, and let's them more readily do their job without calling attention to themselves.

Translucency is readily achievable in most graphics systems (via the alpha channel) and really isn't such a big deal. However, even though it's not completely transparent, windows borders that only have translucency do not provide a good UI experience. That's because the content behind the borders is too clear and discernible, and distracts the user from the window they're manipulating.

Therefore, a key element in the Aero design is the blurring of content behind the "glass" borders, to avoid that content being too clear and discernible. This blur is surprisingly difficult to achieve, and the DWM team has spent a lot of brainpower and effort getting it to where it is:

First, there is the actual blurring of the underlying content. This is done by a custom PS 2.0 pixel shader. Pixel shaders are small programs that run directly on the GPU, which acts essentially as a SIMD machine able to process many pixels in parallel.
The source of the blurring must be a partially composited desktop - everything behind the window border that's being rendered, but not the window border or what's in front of it. And that needs to be pulled out into a different buffer.
Next, the way blur works is by taking neighboring pixels and averaging them together to create a resultant pixel. This means that when a window with "glass" moves, it's not sufficient to just re-render what's behind it. The area re-rendered needs to be extended out a little bit (by the radius of the blur). Without care, this extension can accumulate across multiple windows and lead to invalidating larger and larger portions of the screen.

All of these combine to create the smooth, seamless blur that Aero glass exhibits; and does so smoothly and quickly so the user doesn't perceive any hiccups on window motion or when "glass" is in front of updating scenes, like video playing.

Flipping and Tearing

Modern DirectX applications, on modern DirectX 9 class graphics chipsets (which are a requirement for running the Windows Vista DWM), often use "flipping" in order to present their content. Flipping is when the video card is able to present video memory surface A to the monitor while the application writes to video memory surface B. When the application is done with its update, it schedules a "flip" through DirectX, which is executed at the next monitor vertical blank. When executed, the video card now starts reading out from surface B, and the application is freed to write into surface A. On the next flip, they swap back.

The result of this is that the user perceives exactly what the application wants it to without any of the "tearing" that frequently accompanies applications writing directly to the buffer that's being displayed when a screen refresh happens in the middle of the application's update process.

The DWM uses exactly this technique to avoid window tearing during refresh, and this results in a much smoother desktop.

But Isn't There a Limit to Video Memory?

There are limits to how much video memory an application can use, and how much GPU time is available. So how can a general window system operate within these limits when there are no limits on, say, the number of windows that can be opened? Answer: the Windows Display Driver Model (WDDM, aka LDDM) makes all of this viable. How's that? That'll be the topic for my next blog entry on the DWM.

Responding to Comments from "Under the Hood of the DWM"

Greg Schechter — Sat, 11 Mar 2006 09:39:00 GMT

Wow... lots of great interest and comments in my previous post on "Under the Hood of the Desktop Window Manager". Rather than attempting to comment inline, I figured I'd try to address in a separate post. So here goes:

Anonymous: I'd like to know why DWM can't just use the alpha channel of the buffer that Avalon draws to, to be able to do completely hardware accelerated irregular shaped and alpha blended windows, but instead has to drop Avalon HW acceleration and go to software rendering...

The above is not the case, if I'm parsing correctly. An Avalon application is effectively a DirectX application, and renders to a DirectX surface. This can be a video memory, hardware accelerated surface, and its display through the DWM does not require that it be booted out to software.

Stephane Rodriguez: "Tear free experience"

How is this possible? Doesn't this depend on how much is getting drawn?

Two decades ago, on machines like Atari ST and Amiga, it was all about the vertical blank. People used to write applications (demo) that would draw based on the vertical blank frequency, avoid the drawing spot, resulting in smooth scrollers and so on. Whatever happened to this?

In other words, I don't think the tear experience scales well if you drawing too much on the screen at the same time, say you are running a couple videos concurrently.

And I still fail to see how tear-free experience will make mom and dad feel excited about Windows Vista. Sorry, it had to be said.

Today's "tear free" isn't the old Atari/Amiga approach which as you say tried to fit everything into vblank. Instead, DirectX is able to maintain a "flip chain" of buffers and swap between them for what's displayed on the screen. The "flip" happens at a very low level in the system and does occur during vblank. But it doesn't require drawing. The video card then simply scans out the new front buffer, and treats the old front buffer as the back buffer that the application (in this case the DWM) renders into.

Tear-free making Mom & Dad excited about Windows Vista? Yea, can't help you much there :-), but hopefully the accumulation of all the good stuff across Vista will be enough.

[3/18/06 - Update - tear-free exciting Mom&Dad. It occurs to me that I may have been too flip in this response. Tear-free on it's own won't excite them, however, this is just one of the many quality improvements being made where the sum of which will lead to just a better, more solid feeling product. Kam VedBrat also talks about this effect.]

JoeW: I would like to know more about the public APIs. Specifically I would like to know how Vista's native dialog boxes are rendered (message box, explorer windows etc). Do they still use GDI or do they use an unmanaged composition engine?

More on the public APIs in a later posting. However, regarding how dialog boxes are rendered, there's fundamentally no change here. GDI is still used, and the windows redirect through the DWM just like any other GDI application. The primary rendering that the DWM itself is responsible for is the non-client areas of windows.

David Larsson comments on issues he's hitting with High DPI mode in the latest Vista CTP. I can say that many of those rough edges are being addressed.

Another David asks about memory requirements for DWM given that windows contents are stored. It's definitely the case that there is a larger memory requirement when running the DWM -- but rather than getting further into it, I do expect that to be discussed in a future posting.

Shawn Oster: As I'm reading a few dozen questions come to mind. You hit on some of them in your "upcoming topics" hit list but here are the things that sparked:

1. How signifigant will hardware acceleration be to Glass? It used to be there was the 2D benchmarks for work apps then the 3D benchmark for the fun things. An amazing game card would often be horrible at having a lot of small Excel/Word/Visual Studio windows open.

2. One of the first things new developers learn is to do their own screen buffering, to a bitmap, metafile, etc. Does the DWM change that since it's already buffering?

3. What kind of impact will this have on apps like Photoshop? Are there things raster image editing programs can do to improve speed if they are running on Vista?

4. If you are running old hardward does everything still go through the DWM? Like inside a Virtual PC?

5. How much control will we have over DWM? For example could someone create Flip/Flip3D as a stand-alone app? The nerdling fun side of my brain imagines interfaces a la Minority Report (minus the gloves and Tom Cruise).

6. Hmm, speaking of the Minority Report UI, how well will video drag?

7. How heavy a load does DWM put on the system? I remember when shell replacements were big (LiteStep, geoShell, Cloud9, etc) and the goal was always reducing load while giving UI goodness. Sounds like DWM could be that as long as it isn't seen as a boat anchor.

Looking forward to more blog posts on this subject. Try to come up for air just a wee bit more often :)

and answers to those...

HW acceleration is a very significant determinant of DWM performance. Particularly video memory transfer rate. Windows Vista will enable the DWM on hardware that's capable of supporting it, and we do expect that to be a wide range of hardware.
Application doing screen buffering: I'll address that further down in response to another person's question.
The DWM itself doesn't really have substantial impact on apps like Photoshop (other than the buffering issues). There may be other Vista advice they can follow, but I don't have that.
Things will go through the DWM if the hardware qualifies, including being sure that a Vista Display Driver Model class graphics hardware is in use. More on that in a future post.
More on that in a discussion on DWM APIs
I assume you're referring to the issue on XP and before that when you drag a window with video, the video lags the window. That's because in those systems, hardware overlays are being used. In Vista, the window is composited, and everything is in sync, so no more lag.
We're working hard to be sure that the DWM doesn't put an overly hefty load on the systems that qualify for DWM.

DCMonkey asks about resize semantics under the DWM. This is something we're working on improving, and have made good progress there.

Kirill Osenkov asks: Does it mean I should get rid of my (now unnecessary) custom back buffer when I port my app to Vista? Does it also mean that all flickering applications will automatically become flicker-free?

Yes and No. Applications no longer need to double buffer when the DWM is running provided that their purpose in double buffering was to avoid video-system induced tearing. However, double buffering is also frequently used by applications to avoid "structural tearing" -- for instance, being sure that this triangle and this rectangle are presented atomically. This is not something that the DWM can help with. So it really depends on your usage of double buffering, and your original source of flickering. Each application author is going to need to evaluate that for themselves.

HTK asks whether the first screenshot with the bigger buttons is a real screenshot. No, it's not, sorry I wasn't more clear. That was really intended to show the design element/sensibility for Aero. All the other Windows Vista images are screenshots.

Chris Nahr asks about getting rid of traditional WM_PAINT processing altogether, and just drawing new elements as required. This is absolutely not the direction that this is going. WM_PAINT is still used in many places throughout the system for invalidation and rendering requests of client areas. Chris also asked about higher resolutions. That'll be covered in a future post on high DPI support.

ion and other asked about jaggies/aliasing in Flip3D. Yes, this is certainly a known issue that we're actively working on addressing.

Eli asks about glass blurring, and DWM accelerating GDI. Pixel shaders are indeed used for the blurring component of the "glass" look. And DWM does not accelerate GDI rendering. However, due to the nature of desktop composition, operations like window moves and resizes can be faster/more responsive because underlying content need not be re-rendered.

Anonymous asks about text rendering. The caption area text rendering is going through the same text rendering that WPF uses, and thus is subpixel ClearType. Furthermore, there is an exposed API in the theming DLL for rendering text atop "client area glass" -- more on that in a future post.

ping pong asks if DWM is written in a managed language. It is not. It uses the same unmanaged rendering component that the WPF/Avalon system does.

Mike Dimmick talks about running Java applets in Sun's JRE results in the DWM being disabled. Yes, Sun is well aware of this and is proactively working on a fix. And you're right, that locking the DirectDraw primary surface has this result, since the desktop now needs exclusive access to the DDraw primary for the DWM to work.

DGC asks about memory usage. Yes, there is a larger memory footprint with the DWM. More on that in a future post.

KJK::Hyperion asks about total abuse of this technology through 3rd party programmability. But of course ☺. We do have a certain amount of programmability of various key features like glass, thumbnails, and control of composition. More on that in a future post.

Keith Patrick and MueMeister both wonder about the relationship between DWM and WPF, and if there is any. There most definitely is a big relationship. Both DWM and WPF are written on the same unmanaged rendering layer that sits above DirectX and is responsible for all the actual rendering in both WPF and DWM.

Under the Hood of the Desktop Window Manager

Greg Schechter — Mon, 06 Mar 2006 09:59:00 GMT

I've made a grand total of one post in about the last 21 months. What have I been doing during this time? Why, working on the new Desktop Window Manager for Windows Vista, of course! The Desktop Window Manager (DWM) is one of the more visible features and changes to Windows coming out with the upcoming Windows Vista release.

Does this mean that I've abandoned Windows Presentation Foundation (Avalon) to work on the DWM? By no means. The DWM is built upon the core graphics layer of Avalon, and is being developed by the same team responsible for Avalon.

We've been heads down on design, development, and testing for quite some time, but now that there's an end in sight, I figured I'd come up for air and describe a bit about what we've done and the technical underpinnings of this visible feature. There's lots and lots that can potentially be discussed, so I'm going to keep this first post fairly broad and high level, and will be interested in feedback if there are other specific areas of interest that readers would like to delve further into. So please do comment with your thoughts!

The public face of the Desktop Window Manager

The DWM is of course just part of Windows Vista and not considered distinct from it. Its features are exclusively available in the Windows Vista Aero experience. I'm pulling out some of the more recognizable features here.

This is "Aero Glass", the semi-transparent look that Aero provides, with the blurry content behind the window frames, designed to allow the user to focus on the window itself, and not on what lies behind.
Here are the live thumbnail views provided on the Windows Vista taskbar.
And here are Windows Flip and Windows Flip3D -- the updated Windows Vista experiences invoked by Alt-Tab and Windows-Tab, respectively, for navigating between and selecting windows.

Desktop Composition

By far the largest change to Windows Vista in the way that windows are presented is the introduction of "desktop composition". This underlies everything that is done by the DWM. The primary takeaway for desktop composition: the way an application gets pixels on the screen has fundamentally changed.

In all versions of Windows, up until Windows XP, applications were asked by Windows to paint their visible region, and they would paint directly to the buffer that was to be displayed by the video card. With Windows Vista, applications are asked by Windows to paint their entire surface to an offscreen surface (alternatively known as a bitmap, buffer, or texture), and it's up to the DWM to take all of these offscreen surfaces, and "compose" them together to the onscreen buffer.

Read the previous paragraph again. From a windowing system display perspective, this has profound implications in terms of the features that can be implemented, and the quality that can be achieved. Some examples:

Access to windows - now that applications are rendered offscreen, those offscreen representations can be used in other places. This is precisely how the Flip, Flip3D, and thumbnail features work, and other features can be built that take advantage of this as well.
Don't involve background applications in window operations - when a window moves across the screen in XP and before, the portions of background windows that are newly visible only get painted when the background application wakes up and starts painting (in response to WM_PAINT messages it receives when the top window is moving). For non-responsive background applications, or even responsive ones that happen to be paged out, this can yield a very poor user experience.

Consider moving an 'Paint' program window over an IE window. On XP and before, the following symptoms are unfortunately all too common:

In both of these cases, the underlying Internet Explorer application was unable to repaint itself quick enough to avoid the "trails" that the moving Paint window left behind.

Under Windows Vista, this simply isn't the case -- underlying windows do not receive WM_PAINTs and are not asked to re-render, since their content is already available to the DWM and is used to composite the screen.
Tear free experience - given that the DWM orchestrates all rendering to the screen, the latest technologies provided by DirectX that are typically used for games can be used for the overall desktop experience. In particular, graphics cards' ability to "flip" the front buffer results in an absolutely tear free experience as windows are moved around the screen, increasing the smoothness and quality of the user experience.
High resolution support - the majority of applications out there are agnostic of the monitor resolution (DPI) that they're running at. On the increasingly popular higher resolution monitors (120 DPI, 144 DPI, and beyond), this can produce a bad experience where applications appear very small in physical space. Because the DWM has access to offscreen representation of the application window, the DWM is in a unique position to scale such DPI-unaware applications for their final presentation to the user, making for a much improved experience on the higher resolution monitor.

Possible Future Topics

Desktop composition is the most fundamental aspect of what the DWM provides, but we've really only scratched the surface of that topic, and there are still a lot of related topics to explore for a full understanding of what we're doing in Windows Vista. Here's a rough cut of likely topics, roughly in the order I'd expect to tackle them.

DWMs use of DirectX, GPUs, and hardware acceleration
The importance and impact of the Windows Vista Display Driver Model to the DWM
Redirecting GDI and DirectX applications
How underlying WPF concepts and technology are being used
How the DWM paints the window frame and other non-client area
Remoting, Magnification, and Accessibility under the DWM
High DPI support
Publicly exposed DWM APIs
Rendering and visibility optimizations
Memory usage in the DWM

Again, please do comment with your interests and where you'd like this conversation to go. No promises of course that we'll even get to any of them... but I do expect to.

Other DWM-related Sources of Information

While this is probably the most technically comprehensive discussion thus far of the DWM out on the web (though I'm certainly happy to be proven wrong), there are definitely other places with great sources of information and news:

My ever-esteemed colleague Kam VedBrat maintains a blog that often gives a great inside perspective about the DWM, specific DWM features, and Windows Vista Aero.
The microsoft.public.windows.developer.winfx.aero newsgroup carries discussion of all things Aero