UberDemo : .NET 3.0

Capturing a WebCam stream to a WMV file from within a WPF application

uberdemo — Fri, 28 Mar 2008 00:10:00 GMT

I was asked to help out with a demo, my assignment was to capture some video from a WebCam that was to be used for the Cirque du Soleil demo for MIX08. There were a few ways to do this, none which were obviously attractive. There are many 3^rd party tools that capture video and output to a file, but none I found were extendable and usable from within an application. Another thought was to use the DirectX API’s to do the capture, but this required some fairly involved interop. Windows Media Encoder was not an obvious choice, but to my delight ended up being perfect for the job. I was satisfied with the results. Windows Media Encoder Series 9 has type libraries that expose its COM API’s and can be added to a Visual Studio project directly as references. Visual Studio automatically generates Runtime Callable Wrappers for them.

This is a not a production quality sample, but I thought it still lhad some value getting it out there, since I spent quite some time finding out how to go about this.

A few things to mention up front

· The Windows Media Encoder Series 9 (WME9 from here on) has a bug on Windows Vista that requires a hotfix to be installed. It needs to be installed as an Administrator of the machine. You can run it as a non-Admin, but the hotfix will not install properly and will not report errors. If you find that anytime you access WME9 code (on Vista) and the application crashes without error, you can be sure that the hotfix is not installed properly.

· This sample shows at a basic level how to access audio level information via a Managed DirectX API (Microsoft.DirectX.DirectSound is the DLL referenced) and so requires installing some managed DLL’s. I happened to find these DLLs in the Microsoft LifeCam install, so that’s how I installed them, instead of installing the entire Managed DirectX SDK. There may be a faster/better way to install these DLL’s but I haven’t taken the time to look around for it. If you decide not to install this software, you will need to comment the relevant code.

· You will need a WebCam that functions on your PC to run this sample. This means that you need to install your WebCam’s software and ensure that it works outside of the context of this sample. I myself used the a Microsoft Vx6000 LifeCam, but I’ve tested this app with two other WebCams without issue. The application detects and lists the WebCam in a ComboBox on application launch.

· This application was built in Visual Studio 2008, and will also open properly in Expression Blend 2.0 Beta (February). A DesignTimeHelper class has been added to make it easier to see some of the application elements in the designer at design-time.

· To run this project in Visaul Studio you will need to uncheck the “Loader Lock” thrown checkbox due to the audio level code. To get to this dialog choose Debug -> Exceptions from the Visual Studio menu and expand the “Managed Debugging Assistants” node.

Installing and running the sample

1. Install WME9 from: http://www.microsoft.com/downloads/details.aspx?FamilyID=5691ba02-e496-465a-bba9-b2f1182cdf24&DisplayLang=en

2. [optional] Install the WME9 SDK from: http://www.microsoft.com/downloads/details.aspx?FamilyID=000a16f5-d62b-4303-bb22-f0c0861be25b&DisplayLang=en (The WME9 SDK ended up being very helpful, although I found that the samples in the “samples” directories were not as helpful as the actual content in the CHM file. There are complete samples hidden in the text of the CHM files that were enlightening.)

3. [required for Windows Vista only] Copy the WME9 hotfix locally from: http://download.microsoft.com/download/0/3/d/03d35c05-67da-40e0-9e45-3ea0ca6329a4/windowsmedia9-kb929182-intl.exe (Right-click and choose “Run as Administrator” to install)

4. Install the Microsoft LifeCam Software from: http://download.microsoft.com/download/1/9/5/195512A9-1C1E-4429-BFF0-613D8D92E122/LC14.exe

5. Copy the sample project bits from http://www.hugli.org/code/videoencoder/videoencoder.zip expand it and place the VideoEncoder directory in a directory of your choice. Typically I place content in a c:\users\public\demos directory, since this directory is accessible from all profiles without security issues.

6. Install any necessary software for your WebCam and plug it in.

7. Launch the project in Visual Studio 2008.

8. Modify the OUTPUT_PATH to be a directory that your identity has write access to. The path c:\users\public is the default and will work on Vista, however for XP you will need to modify this path.

9. Press F5. After the application launches you should see two ComboBox’s and a clickable image.

10. Choose your WebCam from the “Video Source” Combobox and use the default profile (384 Kbps.)

11. Click on the image and a video dialog will fade/animate in.

12. Press “Begin Recording.” There will be a slight delay and a video preview will appear on the video dialog. Note that there are also crosshairs, recording status and progress bars to indicate audio level that overlay the video.

13. Press the “Record” button and you will see timecode start advancing in the video overlay.

14. Press the “Save” button and the video strean will be closed and saved to disk in the directory specified in OUTPUT_PATH.

15. The video is then added to a WrapPanel as a MediaElement

16. End Sample.

How was sample done?

Now I’m going to cover the application and some of its hidden subtlties. I wrote this sample in a couple of days since the demo had to be put together in short order. If I had more time, I’d break out the audio related content into its own class and spend some time cleaning up the VideoEncoder class that wraps up all the encoding logic. Another thing that I would do is break out the VideoWindow code into its own UserControl. UserControls cannot draw outside of its own bounds so you would need to allow the UserControl to span the width/height of the entire application to allow for the video fadein animation. There are some problems in the code at present. The most troublesome is that sometimes I find that when capturing video the encoder captures only blank video, and I haven’t found the cause of it yet.

WME9 archiving

One thing that confused me about WME9 initially was that as soon as the Encoder was started the video started capturing to disk and it was only then that a a preview would show up in the preview drawing surface (both happened simultaneously). What I really wanted instead was to see the video preview *before* I began capturing the video, so that I could center my WebCam on my subject, and then and only then start the video capture to disk. Well, it turns out that when the Encoder is started, the video starts capturing (Archiving as they refer to it) immediately, by default. To change the behaviour to what I wanted, i.e. to see a preview before the video captured begins, I needed to set the Encoder.EnableAutoArchive property to false (from its default true), and then start the Encoder. At this point the preview begins displaying in the application, and then we can optionally start the video capture by clicking on the “Begin Recording” button with this call:

Encoder.Archive(WMENC_ARCHIVE_TYPE.WMENC_ARCHIVE_LOCAL, WMENC_ARCHIVE_OPERATION.WMENC_ARCHIVE_START);

Encoder.PrepareToEncode(true);

and end the video capture with this call:

Encoder.Archive(WMENC_ARCHIVE_TYPE.WMENC_ARCHIVE_LOCAL, WMENC_ARCHIVE_OPERATION.WMENC_ARCHIVE_STOP);

It’s also good to know that if you want to capture to an entirely new “archive” file, you will need to stop the encoder, redefine the capture file info and then restart the encoder.

Interop

To get the Video to display in a WPF application took some simple interop. It was interesting for me to find out that you can get a Handle from all WPF elements using IntPtr handle = ((System.Windows.Interop.HwndSource)myElement).Handle so that I could have the video draw on a specific element, but all elements returned the same Handle (the parent windows Handle.) This wasn’t very useful since the video ended up being drawn inside the entire Window no matter which element I chose. To workaround this, I instead created a WindowsFormsHost element and added it to the VisualTree. I then added a System.Windows.Forms.Panel as a child to that WindowsFormsHost (note it is important that the WindowsFormsHost has a child, since the WindowsFormHost element itself gives you the handle of the parent Window, which again is not what we want. The differentiating factor is that System.Windows.Forms controls have a Handle property which makes it very convenient to get the Handle. All System.Windows.Forms.Panel controls are Windows themselve’s. This makes it easy to grab a handle and draw within the bounds of the panel. It should be noted that having all controls be windows can lead to resource and performace problems for large numbers of children, this is not an issue in WPF. The Handle property returns a System.IntPtr which we then pass directly to the Preview.SetViewProperties method along with the video stream itself. Doing this draws on the panel.

When you draw directly on a surface using this technique, there is a side-effect in a WPF application; you no longer can see WPF elements residing on top of this drawing surface anywhere in the VisualTree. I call it a “black hole”, since it obscures or cuts a hole into any window-less elements you try to place on top of it, whether a child element or an element with a higher z-order. This is essentially because the OS can only address something with a handle (a Window in this case) and so it draws the video after all the other elements in the WPF window are drawn.

Video overlay

So how then was the video overlay done? It was accomplished by placing yet another Window on top of the video. The WPF Popup element just happens to be a Window with its own handle. There is a side-effect to this technique: if the parent window is moved, you will find that the Popup doesn’t move along with the parent. This can be remedied by subcribing to the Window.LocationChanged event and in that event modify the position of the Popup. Another side effect is that when the parent Window loses focus, the popup ends up on top of other applications. Again this can be remedied by subscribing to the proper focus events, both of these issues are beyond the scope of this writing.

Animation

Note that the video preview Grid (videoCaptureGrid which also contains the Popup) are not part of the animated videoWindow Grid. I found that trying to animate an element that contained the video rendered through interop performed poorly and drew improperly, so I simply hid videoCaptureGrid until the videoWindow Grid finished fading and animating in.

Possible Exercises

Try to get a Thumbnail of a video. I spent some time on trying to do this and ended up falling back to having a media element display the video for lack of time. Two potential ways to do this are: Use RenderTargetBitmap or grab the Thumbnail that Windows itself generates. The code for using RenderTargetBitmap is in the project in the RasterizeVisualAt96dpi method, but is not used at present. I found that it was tricky to grab the thumb of the video since the first frame(s) of the video tend to be black. Grabbing a video thumbnail from Windows will require the use of interop. I’m not sure if this is possible or feasible.
WME9 is able to capture the desktop of your computer. Create a WPF application that runs in your systray that captures your desktop to a WMV file.
Calibrate the audio levels. Right now the Audio levels are cosmetic and gratuitous. They seem to be a bit delayed and levels seem higher than they should appear. Experiment to get the indicators working properly.
Modify the Popup position when the application is moved or loses focus.

Building a recipe application using Vista and .NET 3.0 (Part III: Using Windows Vista and .NET 3.0 features)

uberdemo — Wed, 28 Feb 2007 22:19:00 GMT

Displaying a custom Thumbnail in the shell

The next thing that I wanted to do was to have a Thumbnail display in the shell for “Electronic Cookbook” files. My goal was to have each recipe file show an image of its recipe item. I’d never done this before and I didn’t know quite where to begin. I began an online search using the term “Thumbnail” and I discovered new API’s specific to Windows Vista that made creating custom thumbnails quite easy. In my search I also turned up a simple working sample in Mike Taulty’s Blog, which I based my sample on.

Vista shell enhancements

Mike’s blog was more than I had hoped for and was exactly what I needed. It also helped me to uncover the 3 ways that a file-type can take advantage of the new Vista shell.

Ø ThumbnailProvider: This allows the implementer to define a custom image that explorer displays. (calls to this run out-of-process, making it safe to program against with .NET)

Ø PreviewHandler: This allows the implementer to create a custom preview in the explorer. (Calls to this are made out-of-process, making it safe to program against with .NET)

Ø PropertyHandler: This enables a file-type to surface its properties (pre-defined and custom) in the explorer, readable and optionally writable in the property pages or in the preview pane. (right-click->properties) (Calls to this are made in-process with shell explorer process, so it is not .NET safe. See next paragraph.)

Ø See the Windows SDK C++ samples that show implementing these 3 features at: (\Program Files\Microsoft SDKs\Windows\v6.0\Samples\WinUI\Shell\AppShellIntegration).

.NET developers beware of PropertyHandlers

Incidentally, to save you from wasting valuable time, a PropertyHandler should not be implemented directly using the .NET Framework since the shell makes in-process calls to the PropertyHandler implementation. Much to my dismay, this was not mentioned in the MSDN docs (online or off) and so I found out the hard way after a week of work. Here’s the problem: The Explorer process does not check, and is not aware of, which version of the .NET framework it is loading. Therefore, there is no way to guarantee that the implementation will work, since it could be written in any version of the framework (.NET 1.0, 1.1, 2.0, 3.0 etc.). It actually works if there is only one version of the framework installed on the machine, but shipping an app that assumed this would be irresponsible. One potential workaround for this is to write a C++ PropertyHandler which then makes “out-of-process” calls to a .NET written PropertyHandler implementation. This was a bit more work than I had bargained for. The alternative, of course, is to write a PropertyHandler in pure C++ that parses the XML file and read/writes to the XML file. It looks as though I am going to need to do this, so I need to sharpen up my C++ skills.

XPS as an alternative for writing a PropertyHandler?

I was chatting in the hall with someone on our greater team, about the goals of this recipe application, and I learned that XPS (XML paper specification) files have 2 out of 3 of Vista shell capabilities built-in: the PropertyHandler and ThumbnailProvider. I went and talked to Doug Mahugh our resident XPS expert, and he showed me that XPS files can contain custom payloads. This means that XPS files can contain a print layout as well as proprietary data, and both are independent of each other and the proprietary data is preserved when the document is updated. The custom data will not surface in the XPS viewer. This can be very useful. My thought was to use XPS files to contain recipes, and get a PropertyHandler and ThumbnailProvider for free, but for my specific case this ended up not being an optimal solution since XPS files are already associated with the XPSViewer. Eventually I am going to write an application to open “Electronic Cookbook” files and I don't wish to launch the XPSViewer when opening ECB files. Still though, XPS files will be very useful for printing and archiving purposes. More on this later.

XPS & OPC

Incidentally, XPS conforms to Open Packaging Conventions (OPC specification.) XPS files are self-contained zip files that house content and relationships between that content. .NET 3.0 has nifty API’s that can access, manipulate, digitally sign and print these XPS files. Note that if you are manually manipulating an XPS file, you need to be aware that the contents are case sensitive (that threw me off). See more about XPS. Additionally Microsoft Office 2007 has implemented the Word, Excel and PowerPoint file-types as OPC implementations (e.g. .docx, .pptx, and .xlsx), and so they behave very similarly to XPS files.

In the next blog I will discuss the ECB ThumbnailProvider implementation.

Building a recipe application using Vista and .NET 3.0 (Part I: Design)

uberdemo — Wed, 31 Jan 2007 23:47:00 GMT

This is the first in a series of blog postings which explore designing and building a recipe application using Windows Vista and the .NET 3.0 framework. In this blog, I write about designing the application and the thought process that I went through with the design.

Playing Roles

Many times in the role that we are in, on our team, as demo creators, we wish to conjure up demos that are compelling to developers, CTOs, business decision makers and end-users alike. Obviously this is a virtually impossible feat since all these target audiences have different and sometimes opposing views. For instance, if we create a financial application, how would that appeal to someone in the medical profession? It really depends on the concepts used and whether any can be carried across to other professions.

Food

Not many people are in the profession of creating and using recipes, but it might be easier to see how some of these ideas can carry across to other segments since it deals with something that everyone knows a little about: food. We chose to create a recipe application. Additionally, it’s a topic that all our immediate team members are reasonably familiar with, so it definitely made this a fun and useful project.

What should it do?

Always the difficult question; It was our goal to create a robust recipe application that supported its own recipe (electronic cookbook [.ecb]) file-type. We also wanted the application to take advantage of relevant technologies available on the Vista platform. That is to say, we didn’t want to shoehorn in technologies that didn’t make sense. We start simple. We want a file type that:

- Contains the rich data necessary to re-hydrate a recipe.

- Is an easily serializable/deserializable file format.

- We want to be able to support keywords and ratings that can be aggregated by the shell.

- We want to have the shell to be able to display a thumbnail.

- We would like users to be able to add comments and alter recipes without altering the original content.

- The content should be searchable from the Windows shell (also called Explorer.)

- And of course, we want to create a very rich user experience (UX) in an application that consumes this file-type.

Brainstorming: Considerations in recipe conventions and design

As I quickly found out designing a recipe format wasn’t as easy as it sounded, for a couple reasons: there are no real standards (that I aware of) in place and so recipes are in random formats, plus there are so many ways to represent units of measure in recipes (i.e. really how much is a “pinch of salt”, and what is the unit for 1 egg”?), not to mention which system to use by default: English or Metric. In examining many recipes, here are some of the considerations I found, that needed to be taken into account:

- Some recipes that contain sub-recipes, or are part of a larger recipe, such as a full meal, or a sauce that is part of a meal.

- What measure to use and which ones to normalize on. “One can of tomato puree” is very ambiguous. “18 oz. of tomato puree” is concrete, yet one still doesn’t get a sense of how much it is. The application needs to be able to do some basic conversion.

- Some recipes contain several concrete “sections.” i.e. mixing the sauce and basting the ribs with the sauce.

- Be able to increase/decrease recipe quantities proportionately. If I find a recipe that serves 16, I might want to scale it back. Some flavors in cooking don’t scale linearly.

- Everyone wants to see a picture of what it “should” look like when it is done.

- Get a definition of what an ingredient is. I came across all kinds of terms I had never heard before.

- How about making it localizable.

And the list goes on

Wouldn’t it be cool if you could sort by what country a recipe came from, or a time period, and of course the genre; how about to be able to search friends’ favorite recipes or a large databases of recipes; or to be able to annotate recipes like you would to a recipe card? How about being able to find a recipe that contains ingredients that you have in your refrigerator, or generating a grocery list from the ingredients for items you don’t have in house? This is where things start to get interesting.

In the next blog posting I will cover the details of the XML format that I decided to use for the application.