I have created installers in the past (using WiX) that add templates to Visual Studio 2005.  Recently, I started looking at adding support for Visual Studio 2008 to one of these setup packages.  This particular package installs some file associations in addition to project templates.  The file associations are used to add an entry to the menu that appears when right-clicking on files with a specific extension to open the file in Visual Studio 2005, and to automatically launch Visual Studio 2005 when double-clicking on files with a specific extension.

Visual Studio 2005 and 2008 can be installed and run side-by-side on the same system.  However, any given file extension can only have one "verb" associated with it for the default action when double-clicking on a file with that extension.  The combination of these two facts left me with a small dilemma - how should this file extension be registered if a user installs this Visual Studio add-in package on a system that has both VS 2005 and 2008 installed?

The ideal solution would be to have a stub program that is invoked when files with this extension are double-clicked, and allow the user to select the version of Visual Studio to run from within the stub program, or have the stub program examine the file contents and automatically choose what version of Visual Studio to use.  This is what Visual Studio itself does for file extensions that it registers.  However, I did not want to invest that much in a solution here.

Fortunately, Bob Arnson alerted me to an article in the Visual Studio help collection on MSDN called Managing Side-by-Side File Associations.  That article describes how to implement a "highest version wins" strategy for file associations (meaning that whenever the file association is registered on a system, the default action when opening files with the specified extension will be to open it in whatever the most recent version of Visual Studio was installed at the time that the file association was registered).

The algorithm is presented with example Windows Installer syntax that can be included in an MSI to accomplish this behavior.  At a high level, the algorithm looks like the following:

1. Add an AppSearch for each supported edition of Visual Studio that will resolve to the full path of devenv.exe
2. Add one type 51 custom action per supported edition of Visual Studio that will set a property named DEVENV_EXE_LATEST to the path of that version of devenv.exe that is set by the AppSearch results from step 1
3. Add each of the type 51 custom actions to the InstallExecuteSequence table and condition them such that they will only run if the appropriate version of Visual Studio is installed.  For example, if both VS 2003 and VS 2005 are installed, the conditions will cause only the custom action to set DEVENV_EXE_LATEST to the path of the VS 2005 devenv.exe will run
4. Add a registry entry to the registry table that sets a value like the following:
   HKEY_CLASSES_ROOT\<your extension>\Shell\Open\Command = "[DEVENV_EXE_LATEST]" "%1"

This algorithm is a bit complicated to explain in words and purely in Windows Installer constructs, so once I get the details worked out and tested for my scenario, I will write a follow-up blog with example WiX syntax that can be used to implement this type of logic in an MSI.