Frequently when troubleshooting a problem we capture a memory dump.  Memory dumps are a great tool because they are a complete snapshot of what a process is doing at the time the dump is captured.  It can be a great forensic tool.  There are however an abundance of tools to capture memory dumps (*.dmp).  In talking with some new members on our team we were reviewing the dump capture techniques and the benefits/drawbacks.  If you have your own thoughts please feel free to comment.

Memory dumps are captured under two categories of problems: The first category is hang dump captured when the application is non-responsive or when you just want a snapshot of the current state of the application. The second category is a crash dump captured when exceptions occur. The debugging purpose, the actions we would instruct the debugger to take would be different for both types of dump.

For hang dumps, we would instruct debugger to attach to the process (if there is no debugger attached) and initiate the dumping immediately. For crash dumps, we would like to attach a debugger to a process first and then set up the instruction to monitor the type of exceptions we are interested in and generate a memory dump upon the exception is raised. We will discuss different ways to generate memory dumps.

First let's take a look at some of more specific scenarios when capturing a dump file (these are grouped for the most part into crash or hang based on what they relate to.  There are definitely a lot more variants but these are some basic ones that we can use to compare the different debuggers:

  1. Crash
    1. Create a dump file when an application crashes (an exception has occurred that causes the process to terminate)
    2. Creating a dump file from an application that fails during startup
  2. Hang
    1. Create a dump file when an application hangs (stops responding but does not actually crash)
    2. Creating a dump file while an application is running normally 
  3. First Chance - Creating a dump file when an application encounters any exception during the run of the application
  4. Smaller Dump - Shrinking an existing dump file to reduce the amount of information in the dump file

For each of these scenarios how do the debuggers measure up:

Debsuggers/Scenarios Crash Hang First Chance Smaller Dump
DebugDiag Yes Yes Yes No
ADPlus Yes Yes Yes No
CDB/Windbg Yes Yes Yes Yes
Dr. Watson Yes Yes No No
UserDump Yes Yes Yes No
NTSD Yes Yes Yes Yes
Visual Studio Yes Yes Yes No
Vista Crash Dump Features Yes Yes No No

Next let’s take a quick look at how you might use each one and then we will explore a bit more:

 

Crash

Hang

ADPlus

Adplus –crash –quiet -o c:\dumps –pn <processname.exe>

adplus -hang -quiet -o c:\dumps -pn <processName.exe

Dr. Watson

   

CDB and WinDbg

.dump /ma c:\w3wp.dmp

.dump /ma c:\w3wp.dmp

UserDump

see Additional Details below

Userdump PID

DebugDiag

Create a Crash Rule with Wizard

Right click on the process in Processes and select Full Memory Dump.

MiniDumpWriteDump

see Additional Details below

see Additional Details below

Visual Studio

Debug | Save Dump As...

Debug | Save Dump As…

There are many ways to capture a memory dump as you can see above. Depending on the nature of the problem, a hang or crash dump can be captured using different tools.

For quick and easy hang dump in production environment that installing tools might be an issue, using NTSD debugger that is distributed with Windows XP/Server 2003 can be easy.

However we frequently recommend to our customers to have the Debugging Tools for Windows and DebugDiag available for their servers.  ADPlus has a very low footprint and can be great to capture memory dumps with.  DebugDiag is super easy and great for setting up rules to generate dumps.  Unless there is a special case DebugDiag is the way to go.

Additional Details

Below are some details for most of the tools that we outlined above and when they might come in handy.

External Tools

DebugDiag.exe from Debug Diagnostic Tool

Where do I get it?

DebugDiag is another free downloadable tool from Microsoft. To download, search for “Debug Diagnostic Tool v1.1” from http://www.microsoft.com/downloads or specifically http://www.microsoft.com/downloads/details.aspx?FamilyID=28bd5941-c458-46f1-b24d-f60151d875a3&displaylang=en

How do I use it?

Hang - In “Processes” tab, select the process with right mouse, select “Create Full User Dump”. The memory dump will be located at under Logs\Misc folder under the installation folder of DebugDiag.

Crash - Select Rules tab. Click “Add Rule” button. Select “Crash” as the rule type. Click “Next” button. Select target type, for instance, “A Specific Process”. Click “Next” buttons to finish. DebugDiag will monitor crashes and when a memory dump is captured, it will show the number in the “UserDump” column of the Rules panel.

DebugDiag supports multiple crash rules to monitor multiple processes. It also supports many other configurations. For more information, please refer to the help file that is installed with DebugDiag.

Also see the DebugDiag blog for additional information – http://blogs.msdn.com/debugdiag/

Debugging Tools for Windows (WinDBG, CDB, ADPlus)

Where do I get it?

This set of tools is downloadable for free from http://www.microsoft.com/whdc/devtools/debugging/default.mspx

How do I use it?

Assuming it will be installed to C:\Debuggers folder, we will use that folder for examples.

Catch a hang dump using ADPLUS.vbs

Adplus.vbs is a vbsript that makes your life easy to use cdb debugger. To capture a hang dump to c:\dumps folder from a running instance of NOTEPAD.exe that has process ID 1806, type:

C:\debuggers>adplus -hang -quiet -o c:\dumps -p 1806

To capture hang dumps to c:\dumps for all instances of NOTEPAD.exe, type:

C:\debuggers>adplus -hang -quiet -o c:\dumps -pn notepad.exe

Catch a crash dump using ADPLUS.vbs

To catch a crash dump from a running instance of NOTEPAD with process ID 1806, type:

C:\debuggers>adplus -crash -quiet -o c:\dumps –p 1806

To catch crash dumps on all instances of NOTEPAD.exe, type:

C:\debuggers>adplus -crash -quiet -o c:\dumps -pn notepad.exe

Note that after starting adplus, it will quickly exit since all it does is setting up CDB debugger for the work. After adplus command window exits, you will notice CDB command window come up(minimized by default). Just wait till the CDB finishes. The command windows will close.

Catch a crash dump on any crashing process using CDB.exe

Since adplus utility can only catch crash dumps for targeted process, it cannot be used system wide to capture crash dump for any crashing process. If we want to overcome the limitation of Dr. Watson, we can use CDB debugger for this purpose. To do this, type:

C:\debuggers> cdb -iaec "-c \".dump /u /ma c:\dumps\av.dmp;q\""

This will configure CDB debugger as the default handler for crash by AeDebug registry key. You can verify the setting by browsing to registry key:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug

And see these two values:

Value name: Auto Value data: 1

Value name: Debugger Value data: “c:\debuggers\cdb.exe” -p %ld -e %ld -g -c “.dump /ma /u c:\av.dmp;q”

Except for a different debugger, this configuration is identical to the configuration described above for using NTSD.exe. Since NTSD.exe does not have the feature to configure AeDebug registry key, we have to do it manually in the previous scenario.

The advantage of using CDB as system wide crash handler is that it can capture crash dumps for any process on Windows XP/Server 2003/Vista.

There are a lot more features to discuss about the Debugging Tools for Windows and there are a lot of resources out there.

Before we leave this topic another cool feature of both CDB and WinDBG is that you can load a dump file into them for analysis and if you want to generate a smaller dump file to share the callstacks or something basic with another person via e-mail for instance.  You can generate a minidump using this command while debugging the dump:

.dump /m c:\smallerDump.dmp

This can be very helpful for providing a bit of information without having to transfer a huge DMP file.

UserDump.exe

Crash Dumps and hang dumps can be taken with the User Dump tool.  For all of the details and download information check out http://support.microsoft.com/kb/241215

Dumps and Development Time

Now when you are developing there are a couple of cases where you want to look at dump generation.

Visual Studio

Where do I get it?

You purchase it.

How do I use it?

To generate dump files you need to have the C++ tools installed which will enable you to do Native Debugging.  Once you enable Native Debugging then whenever you have “broken” into the debugger you can go to to the Debug menu and select Save Dump As… and that will allow you to save a dump file.  This can be very helpful on a developers machine when they encounter a problem in an application they are debugging and want to save it for later analysis.  Also, I have used this to capture dumps of Visual Studio when it is going weird.  Just pop open a second instance of VS and attach to the first and take a memory dump.

MiniDumpWriteDump

Where do I get it?

This is an API that you would have to build into your application to enable your application to generate a dump file.

How do I use it?

This is an API to generate dumps so it cannot be used directly but would have to be coded into a tool or application.  So customers will actually bake this into their Native Unhandled Exception filter.  MiniDumpWriteDump can have a dump taken using the following code:

#include <dbghelp.h>
#include <shellapi.h>
#include <shlobj.h>

int GenerateDump(EXCEPTION_POINTERS* pExceptionPointers)
{
    BOOL bMiniDumpSuccessful;
    WCHAR szPath[MAX_PATH]; 
    WCHAR szFileName[MAX_PATH]; 
    WCHAR* szAppName = L"AppName";
    WCHAR* szVersion = L"v1.0";
    DWORD dwBufferSize = MAX_PATH;
    HANDLE hDumpFile;
    SYSTEMTIME stLocalTime;
    MINIDUMP_EXCEPTION_INFORMATION ExpParam;

    GetLocalTime( &stLocalTime );
    GetTempPath( dwBufferSize, szPath );
    StringCchPrintf( szFileName, MAX_PATH, L"%s%s", szPath, szAppName );
    CreateDirectory( szFileName, NULL );

    StringCchPrintf( szFileName, MAX_PATH, L"%s%s\\%s-%04d%02d%02d-%02d%02d%02d-%ld-%ld.dmp", 
               szPath, szAppName, szVersion, 
               stLocalTime.wYear, stLocalTime.wMonth, stLocalTime.wDay, 
               stLocalTime.wHour, stLocalTime.wMinute, stLocalTime.wSecond, 
               GetCurrentProcessId(), GetCurrentThreadId());

    hDumpFile = CreateFile(szFileName, GENERIC_READ|GENERIC_WRITE, 
                FILE_SHARE_WRITE|FILE_SHARE_READ, 0, CREATE_ALWAYS, 0, 0);

    ExpParam.ThreadId = GetCurrentThreadId();
    ExpParam.ExceptionPointers = pExceptionPointers;
    ExpParam.ClientPointers = TRUE;

    bMiniDumpSuccessful = MiniDumpWriteDump(GetCurrentProcess(), GetCurrentProcessId(), 
                    hDumpFile, MiniDumpWithDataSegs, &ExpParam, NULL, NULL);

    return EXCEPTION_EXECUTE_HANDLER;
}

From http://msdn2.microsoft.com/en-us/library/bb204861.aspx

Existing tools for Windows XP/Server 2003

Dr. Watson

Where do I get it?

It is found at %SystemRoot%\system32\drwtsn32.exe and is distributed with Windows XP/Server 2003 and earlier versions.

How do I use it?

Configure Dr. Watson by typing:

c:\windows\system32>drwtsn32

You can configure the location of the memory dump (the default location is C:\Documents and Settings\All Users\Application Data\Microsoft\Dr Watson\user.dmp), the type of memory dump (for dump analysis purpose, select “Full” for Crash Dump Type. The default type is Mini). The problem with Dr. Watson is that it overwrites the user.dmp file. If you have more than one process crash or the same process crash due to different reasons, you have to make sure to move the user.dmp file before it is overwritten.

NTSD.exe

Where do I get it?

From %SystemRoot%\System32 folder and is a debugger distributed with Windows XP/Server 2003 and earlier versions.

How do I use it?

It’s a command line debugger and can be used to capture crash dump as well as hang dumps.

Crash Dump - To set up NTSD as system wide handler for crash, add two values under under AeDebug registry key:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug

Value name: Auto Value data: 1

Value name: Debugger Value data: “ntsd.exe” -p %ld -e %ld -g -c “.dump /ma /u c:\av.dmp;q”

Make sure to include the quotes in Value data.

Upon process crash, OS will invoke NTSD.exe which will in turn attach to the process and capture a memory dump by executing the .dump debugger command.

If you are curious what the command line means, the “-c” switch is for the debug command to be executed by NTSD debugger. The command “.dump” is for generating the memory dump. The switch “/ma” is for mini dump with all data streams. The “/u” switch is for the debugger to use a unique file name based on timestamp. The “;” delimits the first command. The second command “q” is for debugger to terminate the process and exit.

Hang Dump - NTSD is a full featured debugger. It can be used to capture a hang dump given the process ID. For example, if NOTEPAD.exe is running with process ID 1806 (process ID can be obtained from task manager), you can type this command to capture a hang dump:

C:\>ntsd -c “.dump /ma /u c:\hang.dmp;qd” -p 1806

If you don’t like to type the whole command line again and again, you can create a batch file: HangDump.bat with this command:

ntsd -c “.dump /ma /u c:\hang.dmp;qd” -p %1

To use the batch file, type:

C:\>HangDump 1806

Generating dumps with Vista

Vista does not ship with Dr. Watson or NTSD. Instead, Vista has “Problem Reports and Solutions” which is located under Control Panel->System and Maintenance.

Capture a hang dump in Vista

Inside Task Manager, select the process , right mouse click “Create Dump File”.

clip_image002

Capture a crash dump in Vista

For process crash, Windows Error Reporting (WER) will try to analyze the crash first and then display this message box(where BADAPP is the crashing application):

clip_image004

When clicking “Close program”, the process will exit. By default no crash dump file will be kept on the local system. However, if you wish to retain it, you can modify this registry value ForceQueue to 1, which is located at:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting

For dump files running in system context or elevated are located at:

%ALLUSERSPROFILE%\Microsoft\Windows\WER\[ReportQueue | ReportArchive]

Dump files from other processes are located at:

%LOCALAPPDATA%\Microsoft\Windows\WER\[ReportQueue | ReportArchive]

Starting with Vista SP1 and Windows Server 2008, more features for crash dumps will be supported. For more information, see Collecting User-Mode Dumps at http://msdn2.microsoft.com/en-us/library/bb787181(VS.85).aspx

Credits

A lot of people contributed thoughts comments and work to this so I just want to provide them with credit:

Linkai Yu who did the bulk of the work. (I will introduce you to him in a subsequent post!)
Michael Wiley, Pierre Donyegro, David Wu, Greg Varveris, Sean Thompson and Aaron Barth who contributed and helped review.