In the Trenches : Developing on Win32

What is Open Metadata?

marcmill — Thu, 03 Nov 2005 03:01:00 GMT

In Windows Vista, you'll be able to register property handlers on your own file types. This is in fact what my PDC Talk was all about. These property handlers are used to expose your file's contents to other parts of the system. They use a standard schema to enable stacking, grouping, filtering, and other metadata-search operations across heterogeneously typed items. The Universal Search Engine will for example call the registered property handlers to enumerate all the properties on each file during indexing. So, indeed, the most important feature that a registered property handler provides is metadata reading. In this sense, the interface for property handlers, IPropertyStore, is a simplified version of IFilter.

But property handlers provide another important feature. They allow the system to write metadata to your file types. Although this introduces extra complexity for the file type designer, it enables many cool end-user features. For example, a user doesn't have to start up the file's associated application in order to change the Author property. Instead, the user can simply select the item from within Explorer and drag it into a different Author stack. Or, just type in the name of the new Author into the preview pane.

Property handlers which support open metadata can reason over properties which aren't intrinsic to the underlying file type. These handlers can accept any value for any property, persist that value, and then return the same value later. And by "any property", I include those properties whose schema is invented after the property hander is written. Quite a trick!

Of course, the platform provides serialization APIs and other services to make it easier to support open metadata (the IPropertyStoreCache implementation returned by SHCreatePropertyStoreOnMemory supports IPersistStream), but usually adding this support to your file format carries some nonzero cost. So why do it? Supporting Open Metadata means that users can organize their files of your type along dimensions that you may not have anticipated. This is a very powerful idea! Here are some example scenarios that come to mind:

The "rating" property in Windows Vista isn't intrinsically supported by most file formats, but it's a cool feature to be able to rate your Visio documents, for example, and then sort or group them by rating.
Imagine an IT manager at a law firm who deploys the "legal case number" and "client name" schema elements onto that firm's computers. The lawyers at the firm will be able to within Explorer (or the Common Item Dialog) assign case numbers and client names to the various documents associated with that case. This could include not only word processing and spreadsheet documents, but other file types like faxes and media files which represent, say, evidence or recordings of hearings.
The IT manager could then get fancy by creating and deploying custom Virtual Folders which perform interesting queries like:
Stack all my documents by their associated legal case number, then group those stacks by client name.
That IT manager will look like a super-star!
Imagine being able to attach a "priority" property to documents, like you can for messages in Outlook. Then you could quickly filter your documents based on their priority.

Open Metadata allows users to enable custom search and organize scenarios across heterogeneous sets of items.

PIDL Me This: The Explorer Storage Architecture

marcmill — Fri, 07 Oct 2005 04:22:00 GMT

The core Shell storage architecture has existed in some form since Windows 95. Since then, it's been evolving and will continue to as long as Windows is shipping. There are at least three notable places (that I can think of right now) where this storage platform is carried forward in Vista. First, more of the shell core functionality can now operate over non-filesystem places in the shell namespace, which is most evident in the new copy engine API (IFileOperation) and the new common file dialog API (IFileDialog and friends). Second, the addition of a rich shell namespace extension that can reason over the Windows search platform and abstractly distribute queries to multiple providers and scopes including remote queries to other machines. And finally, the addition of the shell property system which brokers property metadata for user interaction and multiplexes properties on items from their various providers.

I will almost certainly be blogging about these three and more in the future, so I'd like as a baseline to briefly discuss some of the high-level concepts of the shell storage architecture. This topic is discussed in greater detail elsewhere (here too), but I'm going use more modern terminology and paint only a conceptual and perhaps more idealistic picture than actually exists. After all, Explorer is both a platform and an application, and sometimes platform beauty has to succumb to the realities of shipping the application with the desired feature set.

The Namespace. The shell namespace is a tree of items, shell items to be precise. You can manipulate these items by using the IShellItem interface. The namespace is rooted at the root item, called the desktop folder. Shell items can have children that are other shell items. And you can enumerate the children of a shell item by binding to its IShellFolder handler and calling EnumObjects. You can ask a shell item for other services as well: you can bind to a stream on that item, you can read metadata for the item, you can get the icon or context menu on the item, you can enumerate and invoke verbs associated with the item, you can register for change notifications on the item, etc. You could build a whole shell-like application in fact by using this API set, and indeed the shell itself uses this API to deliver its user scenarios.

One of my favorite examples of a shell item is a ZIP archive file, because it is an example of an item that behaves both like a leaf node, in that it has streaming content, and a non-leaf node, in that it has child items which themselves can have children and streaming content, etc. In fact, not many people know this, but you can use the shell's namespace object model as a sort of native API for enumerating and extracting ZIP file content.

Junctions. The ZIP file is also an example of something called a junction, which is a place where one type of namespace handler delegates to another namespace handler by exposing an entrypoint to the other handler's namespace from within its own namespace. In the case of ZIP files, the file system namespace handler, which exposes a shell item namespace over the file system, junctions control to the ZIP namespace handler, which exposes a shell item namespace over the contents of a ZIP file.

Another example of a junction is the history folder. If you go into the history folder from the command prompt by saying:

> dir /a "%USERPROFILE%\Local Settings\History"
 Volume in drive C is CLAUDIA
 Volume Serial Number is 94C3-00F1

 Directory of C:\Documents and Settings\marcmill\Local Settings\History

08/10/2005  04:00 PM    <DIR>          .
08/10/2005  04:00 PM    <DIR>          ..
08/10/2005  04:00 PM               113 desktop.ini
10/06/2005  10:54 AM    <DIR>          History.IE5
               1 File(s)            113 bytes
               3 Dir(s)  25,113,985,024 bytes free

But if you view the same location in Explorer, you'll see something completely different:

That's because the History folder itself is another one of these junction points. And while the file system doesn't recognize the shell namespace junction points, Explorer does. That's what allows us to have content that appears only in the file system, in this case the history database itself, separated from the content that appears within Explorer.

Using this concept of junctions, the shell item namespace that ships with Windows provides extensibility points where third-party item namespace providers can plug in their own item namespaces. In fact, these same extensibility mechanisms are used internally to deliver features like the History folder, the control panel, and especially the new "virtual folders" feature in Windows Vista. In the case of virtual folders, the ".vfolder" files that you find in your profile directory are akin to ZIP files, in that they are junction points to a shell item namespace that operates over database query results. More on how this works in a later post.

File System Versus Item. Any shell item can be asked if it is really backed by a real file or folder on the file system, and if so, asked for that item's path. It is often tempting as a result to circumvent the shell storage abstraction by extracting the file system path from the shell item, and talking directly to the file system. After all, your code will work in the majority case where the underlying item is actually backed in some way by the file system. And in fact, sometimes this circumvention is required since the shell, for example, provides no abstraction over memory mapping the item's content from its storage.

However, when making a decision to only operate over file-backed items, you should be aware of the potential loss of generality. Do you want, for example, your thumbnail provider to be able to render thumbnails for items that appear inside ZIP files, or on storage devices that plug in to the shell namespace (but not the filesystem)? If you support the abstract storage model, then your code will have more reach.

More to come...

COMmunism: Sharing your Apartment

marcmill — Fri, 30 Sep 2005 00:14:00 GMT

The advent of neutral COM objects introduces the interesting consequence that your thread might have to share an apartment. This means that if you assume all asynchronous calls into your thread are on the same apartment, you probably have a subtle bug. Here’s an example that I ran into recently:

My STA thread runs a WndProc which receives asynchronous posted messages WM_DOWORK. The WndProc is associated with a thread-affine object that holds pointers to other STA objects and in response to WM_DOWORK, marshals one of these objects into the stream and returns that stream to the asynchronous caller. This is a common pattern that I’ve seen many times.
That very same thread calls an API which eventually CoCreate’s a neutral object. The neutral object runs in a new apartment hosted on the same thread, a neutral apartment. While running, the neutral object decides to itself call CoCreate, which ends up pumping messages. The message pumping therefore happens in the neutral apartment.
During this message loop, my thread reenters and processes a WM_DOWORK and in response does its job of marshaling one of its objects into a stream by calling CoMarshalInterThreadInterfaceInStream. COM, when deciding how to marshal the interface, looks at the current call context and sees that the object is in the neutral apartment. As a result, the object is marshaled from the neutral apartment instead of the STA.
Later, when the object is unmarshaled from another thread, a transition happens into the neutral apartment on the same thread, which results in the object’s pointer being accidentally smuggled across threads.

If you’re lucky, then some visible problem (like a concurrency issue) will result from this sort of problem.

If you’re really lucky, it will repro somewhat consistently.

The main problem here is that, while the assumption under STA is that apartment implies thread, we cannot assert the consequent and assume that thread implies apartment. So, any asynchronous entry into the thread has to somehow ensure the correct COM context before making calls into apartment-affine objects.

Luckily, COM provides an API for doing just this. One solution to the problem described above, then, is:

When the WndProc is created, it remembers its COM context by calling CoGetCallContext for IContextCallback.
Later, when the WndProc is called, it calls IContextCallback::ContextCallback to get back into the correct COM context. Then it proceeds to dispatch messages.

Q: But Marc, this is crazy, no one calls this API today.
A: You’re right.