The Great Flying Tortoise

Having trouble getting your property handler working with the indexer?  Here are some troubleshooting tips.

1.       Verity that your propdesc file(s) are in a global location

2.       Verify that you registered your propdesc file(s) using absolute paths

3.       Verify that the event log did not record any failures from registering your propdesc file.

4.       Verify that your property description is well formed, accurate, and was registered properly.

5.       Verify that your property descriptions are marked isColumn="true", isViewable="true", and isQueryable="true" as appropriate.

6.       Verify that your DLL is in a global location (e.g. not under your user profile)

7.       Verify that your DLL is registered under HKLM\Software\Classes

8.       Verify that your DLL is registered using full paths (or REG_EXPAND_SZ strings that expand to absolute paths using environment variables known by the System account)

9.       Verify that your property handler works under explorer

10.    We prefer that you use IInitializeWithStream.  If you must use IInitializeWithFile or IInitializeWithItem, verify that you specify DisableProcessIsolation

11.    Verify that the "Indexing Options" control panel lists your filetype as an indexed filetype

12.    Verify that the test file is in an indexed location

13.    Verify that the test file's has been modified since after you installed your property handler

Common Mistakes:

1.       Placing propdesc files or DLLs under a user directory

2.       Registering things with relative paths

3.       Registering things under HKCU instead of HKLM (remember that HKCR shows a merged view of the two hives)

4.       Forgetting DisableProcessIsolation for non-stream handlers

5.       The test file wasn't modified or wasn't in an indexed location

If all else fails, you can use Image File Execution Options to attach a debugger to searchindexer.exe and child processes (I use "<pathto>\ntsd.exe -server tcp:port=8001,icfenable -gG -o -snul").   Reboot, attach to the debugger, and type "sxeld mypropertyhandler".  The debugger will break when your DLL gets loaded.  Use of debuggers is an advanced topic that I won't get into right now.

A reader asked:

I've been reading up the issues with using .NET for property handlers. I know you can't using .NET Interop to create a property handler without getting into issues with the version of the .NET framework. I discovered though that .NET Assemblies can be registered and used as COM objects. Could one implement the required functions of IPropertyStore in c++ and then inside those functions make calls to a COM object that does all the real work?

 

I know you said that you feel property handlers should be written in c/c++ for performance reasons but would this get around the issue of the framework version issue?

This is a very good question, but the answer is "probably not". 

The reason is that property handlers cannot cause .NET code to get loaded in the caller's process which is not supported.  It has nothing to do with performance.  So you cannot get around the issue by writing a C++ stub that calls a .NET COM object in the same process.

The only thing that works is to write a C++ stub that calls a .NET class running in a different process altogether.  But that's a lot more work, introduces a lot of failure paths, may introduce subtle interactions, etc.

Raymond Chen digs into the reasons with this post:  http://blogs.msdn.com/1317290.aspx

 [Update 2007-06-04: I found another article (from 2006) which repeats this warning:  http://msdn.microsoft.com/msdnmag/issues/04/01/WindowsShell/default.aspx]

Viewed as a data flow component, a property handler has a single file stream input and outputs a one or more properties.  Unfortunately, writing one requires making many decisions before you even set eyes on the code.  Let's look at a few of these preparations.

First, pick a file type.  You'll be writing a property handler for this file type.

Now decide which properties you want to store.  Properties generally fall into a few categories.  Take a moment to consider which of the following you want to support:

• System defined properties (e.g. System.Author)
• Properties you define for your filetype (e.g. MyCompany.MyApp.MyProperty)
• Properties other people define and want to set on your file (e.g. SomeoneElse.Other.Property, aka "open metadata")

Next, decide where in the file you'll be storing the properties.  Hopefully the file's format allows for this already.  If it doesn't, you'll have to collaborate with other users of this file format to accomodate these properties.  Here are a few storage options:

• Compute the property from other data in the file (e.g. System.Document.WordCount)
• Read the property from a specific location in the file (e.g. the EXIF header in certain image formats)
• Read the property from a dynamic storage structure (e.g. suitable for storing SomeoneElse.Other.Property)

And let's not forget deciding which of your properties will be viewable, editable, searchable, etc.  Better start taking notes if you haven't already!

Once you know where to place your properties, you get to decide how to format them in your file.  Will you turn them into strings?  Or would a binary format be better?  Maybe you'll want to store some properties directly in the file format and others into a special dumping ground for open metadata.

Also quite important is choosing which API you will use to write to the file format.  Hopefully it supports stream inputs.  If it doesn't you'll have to flag your handler as "legacy".  Typically you'll end up writing a small layer mapping propert keys and values to whatever the format-specific API uses.

There is one easy decision in this mix.  Choose your programming language from the following list: C++, C.  Regrettably, shell extensions cannot use the CLR, so you may not use C# or any other .NET language.

As you can see, there are a lot of little decisions you get to make when writing a property handler.  There are undoubtedly more that I've missed, but you'll stumble upon those as you prepare your handler.  Additionally, there are many little options and flags you can use to tweak the way your property handler works and to accomplish more advanced tasks.  I am skipping those for now in favor of presenting them as a list later.

-Ben Karas

Knowing what you are trying to accomplish is the first step to using the property system.  Property handlers provide a specific set of services which support specific behaviors in the Windows shell.  So let's delve into the role of a property handler and how it relates to other components.

As a data flow diagram, a property handler is pretty simple:

The input is a file stream.  The output is a set of properties.  Yup, that simple. 

Notice that although property handlers interpret files, they don't open the files themselves.  The layer above it does!  There are good reasons for this which I'll discuss another time.  To prepare the property handler, the file system namespace creates the file stream, creates the property handler, and passes the stream to the property handler via the IInitializeWithStream interface.  Once the property stack is completed, the application is free to read and write properties through the stack.  If the layers above the property handler don't answer a question, they'll pass it down to the property handler.

It is also very important to understand what property handlers do not do.  Applying a bit of logic, we can reason out much of the role a property handler plays:

• A single property handler gets created per file.  Thus...
  • A property handler must be the authority on the layout of the file stream it is handed. 
  • It is not possible to "augment" the set of properties on a file.
• The property handler must glean its properties from the file stream itself.  Thus...
  • The property system, in general, cannot store properties on every file.  Not all files allow storing properties inside them.
  • Not all properties can get written to all file types.  Bummer.
• The caller is not guaranteed to ask the property handler for properties.  This means...
  • To be considerate, the property handler should intialize quickly, delaying heavy work until it is actually needed.
• The property handler is dealing with files.  This means...
  • For large files, if you have control over the file layout, consider making it stream-friendly. 
    • Reserve extra space in the property section.
    • Clump things together so that the handler doesn't have to read the whole disk to piece together data.
• Property handlers are an extensibility point of the file system namespace.  Thus...
  • Other namespaces may have different extensibility mechanisms (or none at all).
  • Other namespaces may be able to delegate to the file system namespace when it comes to properties.  (The search results do this)
  • Other namespaces may choose to reuse the same extensibility mechanism.  (ZIP folders do this).
• Property handlers just provide values.  Thus...
  • Although they provide the data, the windows shell controls the way it gets displayed.
  • It is not directly possible to customize the way the data is visualized(1)
  • When I say "property handlers" make the shell do this or that, I really mean that the shell will do "this or that" in the presense of a property handler that provides the right set of data.

Ok, so that's a big dose of reality for one post.  Despite our best wishes, property handlers are not magic.  They provide specific functionality to enable specific behaviors.  As a reminder, the upside of the role of a property handler is that it enables the great integration with the Windows user interface.

-Ben Karas

(1) Property descriptions contain hints about how to display a value (e.g. using a stars or a text box).  Property descriptions are system-wide and therefore a handler only can set the hints for properties it introduced to the system.

In this series, I will be presenting Windows Vista functionality with a focus on the the file system namespace.  Feel free to ask questions, especially if you want to know how things work on XP or in other namespaces, since I'll otherwise ignore those topics for now.

But first, let's get some terminology out of the way. 

The Windows Shell includes the Windows user interface and also reusable bits such as the save-as and folder browser dialogs.  The shell includes things like the start menu, task bar, desktop, folder views, details pane, properties dialog, and the search UI. 

The Indexer is a service that gathers properties and full text from files and items and answers search queries.

There are also various other applications that choose to use the property system to read or write properties to files.

All these applications connect to the property system via IShellItem2::GetPropertyStore.  This API creates a bunch of components and stacks them together.

The top layer I'm calling the Coercion Layer.  It's job is to ensure that values are of the correct type by weeding out bad ones, removing duplicates from vectors, and trimming spaces from strings.

The Shell Item Layer provides a set of properties, including PKEY_SFGAOFlags.  The shell item layer sits atop the shell namespace extensibility point.  As I mentioned earlier, I will focus on one in particular: the file system namespace.

The Shell File System Namespace provides properties about files such as PKEY_Size and PKEY_ItemType.  It sits atop the property handler extensibility point.

The Property Handler is at the bottom of the stack and provides a set of properties derived from the contents of a file.  Property handlers exist only in the file system namespace.

The way to really grok this stack is to understand that each layer wraps the one below it.  When asked for a property, any given layer chooses whether to return the value itself or to ask the layer beneath it.  The property stack is really just a chain of command.

-Ben Karas

A property handler is the piece of code that enables you to see the properties about your files as in the picture below.  Don't worry about the ambiguity in this sentence; I will be much more specific concerning what component does what in later posts.

Here I've created three .book files.  This is a file type I made up last weekend, so clearly Windows cannot know how to figure out all the data on the screen above without help.  That is where a property handler comes in.  It knows about the properties that apply to each .book file and helps Windows to display this information.

In addition to displaying properties, property handlers let users edit them within the standard Windows user interface:

Property handlers also let users search for files based on the property values.  I can search for "Haunting" and Windows will show "Blindness" in the result set.  And all this can happen from within the standard Windows user interface.

So, in summary, property handlers help your file types fit better into the Windows search and organize experience.  By exposing properties directly in the Windows UI, your users can seamlessly interact with your files without having to open them in applications all the time.  When they have property handlers, your file types feel less bolted onto Windows.

-Ben Karas

Graduate level courses are surreal.  Professors assign nearly impossible homework assignments and projects.  The exams make you break out in sweat just thinking about them.  At the same time, you are often able to just sketch out solutions since it is assumed you could carry out the details if you really wanted to; weeks pass where you just hold classroom discussions instead of doing "real" work.

The exams in an analaysis of algorithms course I took several years ago were particularly interesting.  Consisting of only one or two questions, they asked you to perform analysis on an algorithm you had probably never seen before.  Consequently, there wasn't much you could really do to prepare for the exam.

This all led to a strange incident on the first exam day.  The professor, who shall remain unnamed, was handing out the exam papers at the start of class.  As he did so, one of the students raised his hand and asked, almost seriously,  "Is this exam open book?".

The professor casually replied, "Um, sure, why not?"

:-)

Needless to say this didn't make the exam any easier.  Somehow, I miss those days.

-Ben

I can't bring myself to click the "compile" button on my property handler project yet... so here's a little piece of geeky joy:  Viruwormy.

-Ben Karas

A coworker recently turned me onto Pandora.com, a new music service backed by the Music Genome Project.  The concept is to create a radio station tailored to your exact likes and dislikes.  You tell it an artist you enjoy and it decodes what makes you enjoy that style of music. 

What I like about this service is that it is free and has a single non-annoying, unobtrusive ad that refreshes every few seconds.  Not bad!  This is especially nice with IE 7 tabs.  I can place Pandora on one tab and then continue surfing the net.  When it comes time to rate a new song, I just tab over and then tab back. 

-Ben Karas

ps.  I keep meaning to blog about how to write a property handler.  Honestly!  But I've discovered that blogging is hard if you want to talk about technical content.  Not only do I have to write a program ahead of time, but I have to research the nooks and crannies of the APIs, design a way to present it, etc.  Plus I've been working hard lately (writing documentation, working late, waiting for the Seattle buses to put their chains on, etc.).  Maybe I should just develop funny opinions which are quicker to blog?

Someone asked if I had a copy of the code I've been using in my blog so far.  Well, I didn't as of 5pm today, so I went back and collected the code into 3 projects:

1. propreader.exe <file>
    - Prints a list of all properties we know about the file
   
2. propwriter.exe <file> <property> <value>
    - Writes a single value
   
3. proplister.exe <file> <proplist>
    - Prints all the properties included in a proplist (e.g. System.PropList.PreviewDetails)

I've attached the code here (code.zip). 

Enjoy!

-Ben Karas

It is time that I talk a little about what to do if you want your application to run on XP.  There are three sets of APIs, each with subtle differences and caveats, and ultimately your choice requires deciding what platform your application must run on:

Windows XP; Windows XP with Windows Desktop Search 2.x:

Properties on XP are exposed through IPropertySetStorage.  The supported technique is to call IShellFolder::BindToStorage(IPropertySetStorage) for the item. 

Innate properties such as size and name are exposed through various methods on IShellFolder and IShellFolder2.

Caveats: Properties written with this API may or may not be readable on Vista.  Furthermore, on Vista, this technique only works for some namespaces and not others. 

Gotcha:  Many people mistakenly call StgOpenStorageEx.  Don't do that!  StgOpenStorageEx is only supported for specific formats like OLE Compound Documents or NTFS secondary stream storage.  StgOpenStorageEx doesn't know how to read the EXIF header from a .JPG image.  IShellFolder::BindToStorage knows how to do such things.

Best of all worlds:

If you really enjoy coding, you can detect if a given API is present by calling GetProcAddress or QueryInterface [Aside: please don't detect versions... just directly test if the API is present or not], select the best available API, and deliver an application that works better on Vista.  Hey, "Works better on Vista" sounds like a marketing slogan -- you could have fun coding and sell more software too!

-Ben Karas

Personally, I have trouble using the "I liked it" scale ala Netflix.  A movie that garners 5 stars one week only finds 4 another week.  In a way, this arises from ambiguity present even within the "I liked it" scale.  Did I like it because I was in the mood for a happy film?  Or it an enjoyable serious film?  Would I like to see it again?

What this scale needs are actions that go along with each rating:

•  - Loved it!  I watch it every weekend instead of sleeping.  I press my friends to come over and see it too.  I have an imaginary copy of this DVD on my bookshelf.  e.g. Seven Samurai or Rosencrantz and Guildenstern are Dead
•   - Liked it a lot.  I suggest the movie to friends.  I watch it occasionally when I'm in the mood, or not.  e.g. Tsotsi
•  - Worth renting.  I watch this one if my friends invite me over.  I find myself forced to rent it just once to see what it was all about, and often find it superficially satisfying.  e.g. Braveheart
•  - Blasé.  I didn't finish the movie.  These movies evoke nothing from me and I found it boring.  I would rather go watch a car wash.  I guide my friends away from these.  e.g. Finding Neverland
•  - Yech.  I have a physical hatred of this movie.  This is usually a personal dislike of an otherwise popular movie.  But I will not see this again even if friends are renting it.  e.g. Garden State
•  - Why?  I don't want to see this one.  Just not my style.  e.g. Click

You will probably point out that Garden State, Click, and Finding Neverland were all blockbuster movies.  I'm not saying they were bad.  I just want to make a statement about how I relate to them.  Again, ratings have multiple interpretations, which is a perennial frustration of mine.

How do you interpret your ratings?

The 5 star rating control is misused in most applications today.  How can I say that?  What's more natural than selecting 5 stars for a movie or song?  Actually, the intuitive nature of the rating star control is the cause of this subtle usability issue.

Ratings have multiple interpretations.  Does 5 stars mean I loved a movie?  Does it mean I would recommend it to my friends?  Does it mean Netflix should send more movies like that one?  Does it mean the cinematography/main actor/director alone was worth watching?  Does it mean I would watch the movie if I'm sad/happy/with friends?  5 stars cannot imply all these things.  And therein lies the trouble.

Intuitively, a 5-star control uses the "Liked it" scale.  I expect most people treat it this way:  5 stars means I loved a movie.  1 star means I hated it.  This, at least, works.

Meanwhile, software interprets these numbers.  Netflix tries to send me movies that resemble other 5-star movies.  iTunes tries to play 5-star songs more often than 3-star songs.  The act of imposing this correlation introduces a tension into my decision.  If I rate this movie as 5-stars, I now know that I'll get a bunch more like it, even if I didn't really want them.

An example helps:  Braveheart is not a 5 star movie in my book.  Yet I want to see more movies of that caliber.  So should I give it the 3-stars on the "I liked it" scale, or the 5-stars on the "Send me more" scale?  I liked Borat, but I usually hate that genre of movie.  So I could give that movie 4 stars or 2 stars depending on the interpretation.

What is the proper way to incorporate a ratings system into an application if the end-user interpretation may differ from how the application makes use of the rating?  I don't have a good answer other than to suggest that applications, including Netflix, should think very hard about how to incorporate the ratings system into their interface.  There is a balance to strike between simplicity in design and accomplishment and with complexity and nuanced meaning.

I will be on vacation until mid-December, so it is unlikely that I will make any more posts until then, or even maybe January.

 -Ben Karas

Coding to the Windows SDK 
Reading properties #7 - Summary

Writing properties #1 - Simple beginnings
Writing properties #2 - Filetype support?
Writing properties #3 - Which properties are writable?
Writing properties #4 - Which properties are writable?
Writing properties #5 - Property lists
Writing properties #6 - GPS_READWRITE omits read-only data sources
The deal with IPropertyStoreCapabilities
Writing properties #8 - Canonical Values
Gotcha: You must release property stores quickly

----

Writing properties is very simple at the core: you just call SetValue() and then Commit().  But then there are a bunch of details surrounding this task that make it a little more complicated.  To close this topic, I want to give one more tip:

The best way to determine if the file/filesystem/filetype itself is writable is to simply call IShellItem2::GetPropertyStore(GPS_READWRITE).  This performs the necessary ACL checks, attribute checks, etc.  The only downside is that the return value is non-actionable.  If you want to display a more specifical error message or offer corrective options, you'll have to program those yourself.

Thus, a property editing user interface will typically do the following:

1. Test writability by opening the file with GPS_READWRITE and then release that interface
2. Reopen the file with GPS_DEFAULT
3. Use a property list to populate the UI
4. Test writability using IPropertyDescription::GetTypeFlags
5. Test writability using IPropertyStoreCapabilities::IsPropertyWritable
6. Release the read-only property store

Then to save changes, it...

1. Opens the file with GPS_READWRITE
2. Calls IPropertyStore::SetValue for each change
3. Calls IPropertyStore::Commit to save the changes
4. Releases the read-write store

-Ben Karas

ps. There is an even simpler way to write properties using IFileOperation. I'll have to revisit that another time.