There are times that you need to debug the startup code for an application, but something else is launching the application. Classic examples might be services or setup custom actions. Luckily, the OS team came up with a way to debug these problems: 'Image File Execution Options' debugging. Junfeng Zhang blogged about this a while back. Today, I wanted to talk in more detail about how to use Image File Execution Options with Visual Studio, and a bit more details about how it works.
Here is a sample registry script to do this:
Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\sample.exe]"Debugger"="vsjitdebugger.exe"
MSDN gives a good explanation for how to set this up.
The operating system implements this feature inside the user-mode part of CreateProcess. What happens is that after the operating system determines what executable is going to be started it checks the registry key for a debugger. If it finds one, it will launch the debugger instead of the application. The only way to skip this is to call CreateProcess with DEBUG_PROCESS. The debugger is then expected to launch the original application again, but this time under a debugger.
This has some very interesting implications:
Visual Studio 7.x had very basic support for image file execution options:
VS 2005 got rid of most of these. What is left:
In VS 2005, the other thing to be careful of is that debugger will guess what kind of code to debug by looking at the exe, so if you have a native exe which loads managed code, and you want to debug the managed code, make sure that you check the 'Manually choose the debug engines' checkbox when the Just-In-Time debugger dialog appears.