IFEO and Managed-debugging
Be careful of Image File Execution Options (IFEO) with managed debugging - it won't work like you expect.
IFEO lets you set some registry goo such that when you launch a target app (specified by a registry key name), a debugger (specified by a string named "debugger" under that registry key) is executed instead. The debugger then launches the target app under its control. (For more about IFEO: see MSDN for details, GreggM talks about debugger details. Junfeng talks about other IFEO tips; MSDN has some tips here; and Raymond Chen has more.)
MSDN warns this only works for native and interop-debugging. It does not work for managed-only debugging. Here's why...
First, look at a step-by-step walkthrough:
- You setup a registry key for "MyApp.exe" with a string "debugger"="MyDebugger.exe".
- You attempt to run MyApp.exe via Explorer.
- Explorer makes a call to CreateProcess("MyApp.exe", flags=NotDebugging)
- IFEO intercepts that CreateProcess call because flags=NotDebugging. It launches a debugger by concatenating the value in the "debugger" registry string with the parameters from CreateProcess. So it will actually launch: "MyDebugger.exe MyApp.exe" instead of launching MyApp.exe.
- It is expected that MyDebugger.exe will then use the command line args to launch a debuggee. Specifically, it will call CreateProcess("MyApp.exe", flags=Debugging).
- Since that CreateProcess call specifies it is debugging, the call is not intercepted by IFEO and MyApp.exe is created under the debugger as normal.
GreggM discusses a lot of interesting ramifications of this.
So what's the problem for managed-debugging?
Managed-debugging is not built on native-debugging. Managed-debugging has its own debugging channel that built on its own interprocess-communication protocol, which is completely separate from the OS facilities used by native-debugging. That means that launching the debuggee under the managed-debugger will do CreateProcess("MyApp,exe", flags=NotDebugging). This introduces infinite recursion with IFEO, because that will get intercepted by IFEO and relaunch the debugger. In other words, we'd loop forever between step 4 and step 5.
Interop-debugging is built on OS-facilities, and so looks like a native-debugger to the OS. This is why MSDN tells you to use interop-debugging with IFEO. Another option may be to disable IFEO after the debugger is launched, but before it lauches the debuggee.
