I posted an item on my blog last week with some details about how Windows Installer custom actions work behind the scenes.  Some of the feedback I got back after posting this item asked some good questions about debugging strategies for custom actions.  Based on that, I wanted to share a wrapper DLL written by Jungwook Bae, a developer on the Visual Studio and .NET Framework setup team.  His tool makes the process of implementing and debugging DLL custom actions much simpler.  You can download the tool itself by clicking here.  The following is more information about the wrapper DLL and how to use it:

CATest - a tool that will help developing custom action DLLs

One of the biggest pains when writing custom actions is whenever you modify the DLL, you have to stream the new DLL into the MSI and restart setup. Deferred custom actions are the most difficult because you have to wait until all the files are copied and registry values are written again, which can take a very long time for large products.

To get around this problem, you can use catest.dll as a wrapper for the custom action DLL you are writing. You can test your custom action DLL without streaming it into your MSI, and you can run it as many times you want without restarting setup.

How to use CATest:

1. Add catest.dll to the binary table of your MSI (by using Orca or some other tool)
2. Share out your build folder so that your DLL and PDBs are accessible from a test machine
3. Modify catest.ini to match your custom action name, path to DLL and EntryPoint.  You can debug the following 4 types of custom actions: Immediate, Deferred, Rollback, and Commit. You do not need to include all 4 sections in catest.ini; instead you can include only the ones you want to test.
4. Copy modified catest.ini to %windir% on the machine we are testing.
5. Modify the custom action records in the CustomAction table of your MSI to point to catest for each of the custom actions you want to debug. Each custom action you want to be able to debug should have "catest" as the Source value and the Target value, and should match the custom actions that you list in catest.ini.
6. Now you are ready to go, so run the setup. When prompted to attach a debugger, you can attach your favorite debugger and click OK. Then, you can load your symbols and open source files to step through the code. When the custom action finishes, a dialog will ask if you want to retry the custom action. You can update your code and rebuild the DLL, then choose to retry the custom action and debug it again within the same installation session. Note: you will have to reload your symbols if you rebuild and rerun the custom action.

<update date="3/24/2009"> Fixed broken link to download the debugging tool. </update>