A first hand look from the .NET engineering teams
I recently received the following question from a customer:
“During our test runs (which might run for hours), if a process crashes, we’d like to create full memory dumps for later diagnosis. Can I configure the machine to do this automatically?”
I’ve actually gotten this question numerous times over the past couple of years. It comes in various forms but the general scenario is: A user wants to be able to diagnose application crashes and the environment does not lend itself to live debugging. The latter criterion could be for a number of reasons, but the most common I see are:
· We have an intermittent failure in production and want to gather a dump to debug the issue offline.
· I’m running a bunch of tests and when one crashes I don’t want to interrupt the whole run to diagnose the issue at failure time. Let’s just gather some information for triaging.
· Our issue isn’t reproducible under a debugger. E.g. a stress bug.
These essentially reduce to: You want to get as much data as you can, while minimizing the impact to the environment. Given these requirements, the solution I find meeting most people’s needs is to configure a just-in-time (JIT) debugger to launch, grab a dump, and exit whenever a process crashes.
Support for just-in-time debugging has been in the CLR since V1x and in the OS for as long as I can remember. The basic idea behind a JIT debugger is: when a process crashes, launch and attach a debugger so you can figure out why.
There are registry keys which provide this general ability for both managed and native code (support for the latter might actually be in Win.ini for 9x/Me). If your application is written in managed code (which is the case I’m often presented) you might ask, “My application is managed code, why do I care about native code?” Given that even the most simple managed applications run native code (e.g. the runtime itself), if your requirement is to gather data for any crash, you’ll need to set the keys for both types of code. In CLR V4 we’ve actually unified the key which controls the managed JIT debugger with the native one. However, that change does not change my guidance since, for the time being, we’ll be living in a world where V2 managed code exists alongside V4.
How do I configure the debugger?
1. Download and install the latest “Debugging Tools for Windows.”
a. If you’re running a 64-bit OS, you’ll want both the 32- and 64-bit versions*
b. You can either install the entire set of tools on the machine (it’s a quick, small install) or you can install to one machine and copy “cdb.exe” from the install directory to any number of target machines.
*Note: My sample .reg files below assume you install the 32-bit debugger to c:\debuggers\x86\ and the 64-bit version to c:\debuggers\x64\.
2. Create/set the following registry keys and values (If you’re working on a 64-bit version of Windows, you’ll need to set these keys under the Wow6432node as well.†):
a. Key: HKLM\Software\Microsoft\Windows NT\Current Version\AeDebug:
i. Value: “Debugger”
1. Type: String
2. Value data: <path to cdb> -pv -p %ld -c “.dump /u /ma <dump file path\name.dmp>;.kill;qd"
ii. Value: “Auto”
2. Value data: “1”
b. Key: HKLM\Software\Microsoft\.NETFramework
i. Value: “DbgManagedDebugger
2. Value data: <path to cdb> -pv -p %ld -c ".dump /u /ma <dump file path\name.dmp>;.kill;qd"
ii. Value: DbgJITDebugLaunchSetting
1. Type: DWORD (32-bit)
2. Value data: 2
†Note: You should set the keys to point to the appropriate “bitness” debugger. I.e. you want the OS/CLR to launch the 64-bit debugger for 64-bit process crashes and the 32-bit version for 32-bit crashes. Make sure your debugger paths are set accordingly.
The following sample .reg file will set cdb.exe to auto-launch and generate a crash dump for every process crash on the machine. Note the assumptions the file makes about debugger paths and the dump-file-placement path.
Windows Registry Editor Version 5.00
;This reg file installs just-in-time debuggers to capture a dump of all process;crashes for the machine.
;Assumes 32-bit debugger is cdb.exe and is installed to c:\debuggers\x86\.
;Assumes 64-bit debugger is cdb.exe and is installed to c:\debuggers\x64\.
;Assumes crash dumps can be written to c:\crash_dumps\.
;Make sure all users have write access to this directory.
"DbgManagedDebugger"="\"c:\\debuggers\\x64\\cdb.exe\" -pv -p %ld -c \".dump /u /ma c:\\crash_dumps\\crash.dmp;.kill;qd\""
"Debugger"="\"c:\\debuggers\\x64\\cdb.exe\" -pv -p %ld -c \".dump /u /ma c:\\crash_dumps\\crash.dmp;.kill;qd\""
;The following keys are only used on 64-bit versions of Windows (note Wow6432Node).
;They can be safely created with no side-effects on 32-bit versions of Windows.
;Alternatively, you can delete the remainder of this file if you’re running a
;32-bit version of Windows.
"Debugger"="\"c:\\debuggers\\x86\\cdb.exe\" -pv -p %ld -c \".dump /u /ma c:\\crash_dumps\\crash.dmp;.kill;qd\""
"DbgManagedDebugger"="\"c:\\debuggers\\x86\\cdb.exe\" -pv -p %ld -c \".dump /u /ma c:\\crash_dumps\\crash.dmp;.kill;qd\""
What do these keys do?
The “Debugger” and “DbgManagedDebugger” value data are basically command lines (printf format strings) that are run when a process crashes. The OS or CLR will substitute values for the format specifiers (e.g. in the case above it substitutes the crashing process ID for the “%ld”) and run the command in the user context of the crashing process. The command line I’ve supplied:
· launches cdb.exe, the debugger.
o Obviously, you must specify the correct path to the debugger.
· “-pv %ld” : Attaches non-invasively (just suspends the threads) to the crashing process (the OS or the CLR will actually fill in the PID for you).
· “.dump /u /ma <dump file path\name.dmp>”: Takes a full memory dump with a unique name (appends the date, time and process ID) and stores it in the path defined.
o The path and file name can be whatever you want them to be. Since the debugger is launched in the context of the crashing process, make sure the path points to a location all accounts can write to.
· “.kill”: Kills the target process (you’ve gotten the data you need).
· “qd”: Quits the debugger.
The “Auto” and “DbgJITDebugLaunchSetting” values set the policy for when to launch the debugger. As I wrote above, we want to get our data as quickly as possible and move on, so we don’t want to require any user intervention. For example, in the server scenario, there may be no one logged into the machine to click some “OK” button. The settings I described will automatically launch the registered debugger for all processes on the machine, without prompting (see Enabling JIT-attach Debugging for more details on the settings). Note that when these settings are in place, the debugger will be launched for crashes in all processes running on the machine and you will not have the option of submitting the crash through “Windows Error Reporting.” In future posts I’ll discuss ways of enabling both.
I don’t care about most processes on a machine. Can I just target specific ones?
The answer depends on the version of the OS and the version of CLR. Here are the rules:
· For native code: your OS must be Vista/Server 2008 or higher.
· For managed code: Your version of the CLR must be V4 (or higher, if you’re reading this from the future).
And here is how you configure:
1. Set the debugger keys (AeDebug\Debugger and .NETFramework\DbgManagedDebugger) just as you did in 2.a.i and 2.b.i, above.
2. Ensure AeDebug\Auto and .NETFramework\DbgJITDebugLaunchSetting are not set to auto-launch (again, see Enabling JIT-attach Debugging for more details on the settings).
a. Or you can delete them.
3. Create the following registry keys and values:
a. HKLM\Software\Microsoft\Windows\Windows Error Reporting\DebugApplications
i. Value: <Name of application executable> (e.g. “myapp.exe”)
2. Value data: 1
b. Repeat this for each application you want the debugger to be auto-launched.
You can set the DebugApplications key and values in HKCU if you prefer per-user control. When these settings are in effect the debugger will be launched for only the processes you specify and normal error handling will occur for all other processes (e.g. for most default configurations, a prompt to submit an error report to Microsoft).
The following sample .reg file will set cdb.exe to be auto-launched only for HelloWorld.exe. Replace HelloWorld.exe with the name(s) of the application(s) for which you’re interested in generating crash dumps.
;This reg file installs just-in-time debuggers to capture a dump of only the;processes listed under the [DebugApplications] key, below.
;For each application you want the debugger to be auto-launched, add a row below
;similar to “HelloWorld.exe"=dword:00000001 but replacing HelloWorld.exe with
;your application .exe name.
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\DebugApplications]
Can I do more than just capture a dump?
More information on the command-line switches for the Windows Debuggers can be found in the documentation included with the Debugging Tools for Windows. You’re not limited just to dumps. You can perform quite a bit of debugger automation if you choose.
So what happened?
Now that you have your dump, it’s time to figure out why your application crashed. For those of you familiar with dump debugging, at this point you’re about to fire up WinDbg + SOS (the managed debugging extension) and dive in. But wait! If your application runs on V4 of the CLR (.Net Framework 4.0) you can debug the dump in Visual Studio 2010. Our desire for the experience in VS 2010 is that it feels very similar to stopped-state live debugging (like you’ve hit a breakpoint). The fact that you’re often debugging optimized code can make this a bit trickier, but that’s a topic for another day.
For more information about how to debug crashes in managed code, a search for “managed dump debugging” yields quite a few helpful results. A great place to start is Tess Ferrandez’s blog (a support engineer at Microsoft). She has a number of great posts on the topic, including a post on dump debugging in VS 2010, and some labs/walkthroughs as well.
What is your advice to those unfortunates who want dumps on crashes, but don't want to disturb the customer's environment by setting machine-global registry keys? This is an often-discussed topic, the two common solutions being
1) using an unhandled exception handler to generate a dump — this is not always reliable, stack overflows being exceptionally tricky;
2) using a second small application which launches the main application as a debugger and listens for exceptions — this imposes a performance penalty, as all first-chance exceptions raise the debug event.
Don't forget ProcDump from http://technet.microsoft.com/en-us/sysinternals/dd996900.aspx; the -t flag allows a dump to get created when the process terminates.
@Anton: Good question. My blog post assumes a fair amount of control of the environment (setting the keys, accessing the dump files). If you don't have this type of control or don't want to exert it, I typically I recommend leveraging Windows Error Reporting. Software developers can register their applications and get crash dumps for failures (crashes and hangs). Additionally, it helps you prioritize issues by showing you how many "hits" there are for a particular crash. And finally, it allows you to set up responses, so you have a feedback loop with customers who hit a particular crash/bug. The nice thing is, for the common scenarios, you don't have to write any code in your product to take advantage of it.
More information is available here: http://www.microsoft.com/whdc/winlogo/maintain/StartWER.mspx.
If you'd like to discuss your particular scenario in more depth, feel free to contact me through http://blogs.msdn.com/clrteam/contact.aspx.
Hi, When I try to use cdb.exe with Winform application, it showed me a popup message for an exception and did not generate the dump.
I tried with sample console application and it worked fine. Do I have to perform some extra steps in order to get it to work?
@Ron: That's actually WinForms swallowing the exception and giving you an opportunity to continue the process (something I don't generally recommend). I.e. your exception doesn't really go unhandled. See the following forum thread for details on how get a dump for WinForms apps: social.msdn.microsoft.com/.../9dc0126a-3b39-46c6-bd15-44c130cbeb48.
Debug diag does also help create a dump when a process crashes. Hope I am not wrong. Please confirm.
Debug diag even doesn't cause performance, I hope i am rite. An expert told me.
Thanks for the post anywayz.
- Arun Kumar Allu
The sample *.reg files are not on this page. Somehow i see a table of some .NET user group. Could this article be corrupted?
Why not just use adplus.exe from the Windows Debugging Tools? It can run in the background without performance overhead and you can set it up to catch a dump on a crash or hang. The old version of adplus was a vbscript "adplus.vbs" but they've now updated it to an .exe. "adplus.exe".