I get a lot of questions about how to do transparent top-level windows in Avalon, or nonrectangular windows, or layered windows, or other variations on the same theme.  The short answer is you can do most of these things, but it’s really a Win32 feature rather than an Avalon feature, and you’ll do it using Win32 APIs.  And some of this stuff works better on Longhorn, or at least will once we finish Longhorn.  But the complete answer is quite a bit more involved…

First thing to know is that the only thing the window manager understands is a good old-fashioned Win32 HWND.  So every Avalon Window object is an hwnd.  And it has to play by the rules that hwnds play by – inside that hwnd, Avalon can do all sorts of crazy stuff, but when it comes to other hwnds on the desktop, Avalon has to play by Win32 rules.

In Win32, you can get a non-rectangular window two different ways.  One is by using an region (HRGN) to describe the shape of the window.  The other is to use layered windows (WS_EX_LAYERED), which allow you designate a certain color as being the “transparency key”, and Win32 will make any pixels of that color be completely transparent.  So if you want a circular window, paint everything outside your circle with the transparency key color.

Layered windows also allow you to make the entire window translucent (semi-transparent) by specifying an alpha value to apply to every pixel in the window.  (Win32 even supports per-pixel alpha, but this is very difficult to use in practical programs because in this mode you’ll need to draw any child hwnd yourself)

Avalon supports HRGN just fine in all its variations, we don’t provide any managed API for doing this but you can always use PInvoke and HwndSource and do it yourself.  (Why not?  Mostly as future proofing– we don’t want to bake HRGN and its limitations into the Avalon API, because one day we want to teach the window manager to go beyond hwnds and HRGNs)

Layered windows, on the other hand, come with some limitations, because Avalon is unlike your typical hwnd and uses DirectX rather than GDI to render.  And here’s where it really gets complicated.

• Avalon doesn’t hardware accelerate layered windows, all rendering is done in software.  We’re working on getting hardware acceleration for Longhorn, we’ll have to see what’s possible on earlier platforms.
• We currently have a number of timing issues – if you resize or move the window at the wrong time, you can get into the situation where the application stops painting itself.  You can also get rendering turds outside of the window, where the window used to be (these can be fixed by getting the background window to repaint, but not by moving the layered window over the turd).  We're working on fixing these issues, not clear we’ll be able to fix it completely for XP due to limitations in Win32, but we should be able to do it for Windows Server 2003 SP1 and Longhorn.
• We don’t really support transparency color keys or plan to, because Avalon can't guarantee we render the exact color you asked for, particularly when hardware accelerated.  There’s cases, for example, where we render 0x0000001 instead of 0x000000 – human eye sees both as black, but Win32 sees them very differently. 
• We don’t currently support per-pixel alpha, if you try today you get black instead of transparent.  We’re looking into this, right now it’s not clear which platforms we’ll be able to get it for.
• Layered windows on top of DirectX surfaces flicker when the DirectX application renders (basically, gdi hides the layered window, then lets DirectX draw, then puts the layered window back).  This is actually true of non-Avalon layered windows as well as Avalon layered windows, this will be fixed in Longhorn when using the Desktop Window Manager (DWM), but there's no way to fix in previous operating systems.
• Per-window alpha works fine.

So to recap, in English:

• Non-rectangular regions can be done by using Win32 APIs, either HRGNs or layered windows.  HRGNs work completely, layered windows we're trying to make work completely on Longhorn, layered windows will work with limitations on earlier platforms.
• Per-window alpha (transparency) works fine, again using Win32 APIs.
• Per-pixel alpha (transparency) doesn’t really work today, stay tuned for further developments.

As always, I'd appreciate your feedback so we can make sure we prioritize things appropriately.  And if you can figure out how to explain this one better, please let me know -- this was a very long answer to a short question!