anandbms's WebLog

Walkthrough:Basics of Analyzing a Crash dump using Windbg

anandbms — Thu, 21 Apr 2005 00:54:00 GMT

Its been a while since I wrote anything. I got very busy at work and couldn’t get anything going for the blog and also I spent too much time playing Ninja Gaiden and Halo 2 on Xbox Live :). Today I am going to do a walkthrough on how to look at a dump file in windbg and some of the basic commands. Windbg is the coolest weapon in a Debugging Ninja’s hands ? I guess you can make out where I came up with this analogy. For this exercise I have used VC++.net 2003 and written an unmanaged console application which looks some thing like this
int _tmain(int argc, _TCHAR* argv[])
{
Test *temp = (Test *)malloc(sizeof(Test));
temp->y = 10;
temp = NULL;
free(temp);

temp->y = 30;

}

Sorry for choosing such a simple example. But this serves the purpose. I assume that you have the debugging tools for windows installed. The link is in the previous blog entry. When I run this application, it will crash with an access violation. Since this application crashes at start up, what you can do is go to the Debugging Folder that you had installed and you can start the application as follows

adplus –crash –sc c:\myfolder\Myapp.exe

This will launch the app under the debugger and will capture the crash when the error occurs. Run the debug build and make sure you have the pdb file also. Now when the crash occurs, we will get a folder something like “Crash_Mode….Date … Time…”. In this folder you will find the three dump files. The first one is the First Change AV.dmp, then the Process ShutDown.dmp and then the second Chance AV.dmp. The second change AV.dmp is the one we are interested in. Now launch windbg. This should be in the debugger folder or it should show up in start->Programs->Debugging tools for windows.

The first thing that we need to do is to set the symbol file path. Microsoft provides a symbol server that contains the public symbols for the system dlls. The way you would set it up is Go to File->Symbol File Path and then type the following

SRV*c:\websymbols*http://msdl.microsoft.com/download/symbols

Make sure you create a folder called websysmbols. This is the folder where the Microsoft Symbols will get downloaded. Once you set this up, click ok to this dialog and then Save the workspace so that every time you launch windbg it is already set. The go to File->Open Crash dump and then select the Second Chance Av.dmp. Once you open it, you will see a command window come up and see something like this

.....
(7f8.f7c): Access violation - code c0000005 (!!! second chance !!!)
eax=00000000 ebx=7ffde000 ecx=00000000 edx=00000000 esi=7c91d369 edi=0012fedc
eip=00411ca4 esp=0012fdec ebp=0012fedc iopl=0         nv up ei pl nz na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000             efl=00000202
*** WARNING: Unable to verify checksum for Win32Con.exe
Win32Con!+0xb4:
00411ca4 c7001e000000     mov dword ptr [eax],0x1e ds:0023:00000000=????????

In the command prompt you will see 0.000> and this is the thread ID. So basically this is the thread where the crash occurred. The first thing that you want to look at is the call stack. The command is “kb” Enter kb in the command window and you will see the following

0:000> kb
ChildEBP RetAddr Args to Child
WARNING: Stack unwind information not available. Following frames may be wrong.
0012fedc 00414d00 00000001 003218b8 00321918 Win32Con+0x11ca4
0012ffc0 7c816d4f 0012f7b4 7c91d369 7ffde000 Win32Con+0x14d00
0012fff0 00000000 00411384 00000000 78746341 kernel32!BaseProcessStart+0x23

My program name was Win32Con. If you see we see Win32Con!(some address). At this point it only displays addresses instead of function names and this is because we have only setup symbol path for the system dlls and not the application itself. So go back to the file->symbol File Path and then browse to the folder where you the symbol file for your app and then click the reload check box and click ok. Now when you type kb, you will see the following

0:000> kb
ChildEBP RetAddr Args to Child
0012fedc 00414d00 00000001 003218b8 00321918 Win32Con!main+0xb4 [c:\stubs\win32con\win32con.cpp @ 48]
0012ffc0 7c816d4f 0012f7b4 7c91d369 7ffde000 Win32Con!mainCRTStartup+0x170 [f:\vs70builds\3077\vc\crtbld\crt\src\crt0.c @ 259]
0012fff0 00000000 00411384 00000000 78746341 kernel32!BaseProcessStart

Now you can see the function where the crash occurred. So we now know that we crash in main. If you have the source file, you go to file->Source file path and point to it. Then Go to View->CallStack and in this window click source and it should take you right to the source line were the crash occurred. In this case it will take you to

Temp->y = 30.

You can look at local variables of a function by using the following command “dv” .When you do dv you get the following output

0:000> dv
           argc = 1
           argv = 0x003218b8
           var2 = class Test
           var1 = class Test
           temp = 0x00000000

You can clearly see that temp is NULL when we are trying to access the object which results in the AV. You can get a list of threads by doing the following “~*” . You can get the call stacks for all threads by using “~* kb”. To switch to a particular thread you can do
“~2s” and that will take to thread 2. If you want to examine the local variables for a different function than the one at the top, you need to figure out the frame number.

0:000> knL
# ChildEBP RetAddr
00 0012fedc 00414d00 Win32Con!main+0xb4
01 0012ffc0 7c816d4f Win32Con!mainCRTStartup+0x170
02 0012fff0 00000000 kernel32!BaseProcessStart+0x23

0:000> .frame 1
01 0012ffc0 7c816d4f Win32Con!mainCRTStartup+

Now the context changes to frame 1 and you can look at its local variables. You can get the parameters by using the view->callstack window and choosing the different options.

I shall leave the discussion about stack frames and how variables are pushed to the stack and how return addresses are stored and base pointers to another day. I have given you steps on how to setup windbg and setup symbol paths and look at crash dumps. The commands that I have listed are some of the basic ones that can get you started and the help that comes with windbg has a list of all the commands and explains them in detail. Reading a dump is like an Art and I am still trying to learn things. The familiarity with the product where the problem happens and knowledge of the technology would enable you to get more information from it. There is no cookie cutter methods to learn reading dumps. The more you look at it, the more easier it will become. At this point I shall stop here and hope there is not too much of a break between this post and my next post.

Collecting Information for Application Hang

anandbms — Tue, 08 Mar 2005 04:21:00 GMT

Application hangs typically occur in two scenarios.

1) When there is a deadlock

2) When there is an infinite loop.

Where there is infinite recusion involved, the application will most likely crash because the stack will overflow.

So when the hang occurs we would basically generate pdbs just like for crashes and then start the application and run it till the point where the problem occurs. Once the hang occurs, we would need to generate memory dumps at different intervals. After getting a set of dumps, then we would examine it to see if we are stuck at a certain point and that will give us an idea of the problem. So lets say the app hangs and stays there for around 10 mins. Now when we need to get around 4 dumps at intervals of 2 mins. So we would go to the folder where the debgging tools are installed and then run the following command

adplus -hang -pn "App.exe"

The help that is part of the debugging tools has information about the different command line options for adplus.

When we run the above command, we would get a folder "Hang_Mod_Date..Time.." which will have the dump file, but this time, there will be just one dmp file. You would then run the hang command this way for 4 times at regular intervals to get enough data so that we can load them up in the debugger and see where we are stuck.

I guess now I will start talking about using windbg and how to look at the dump files using various commands.

Collecting Information for Application Crash and Hang

anandbms — Fri, 04 Mar 2005 03:36:00 GMT

I am going to start of talking about the debugging tools that I use to generate memory dumps when customers have hang or crash. I am mainly going to talk about debugging 32bit apps for now. The debugging tools can be downloaded from the following link. One cool thing about these debugging tools is you can do xcopy deployment. Sometimes customers do not want to install anything on their production boxes. In such cases, you can install the debugging tools on a different machine and then copy the folder over to the target machine and the debugging tools will work just fine.

http://www.microsoft.com/whdc/devtools/debugging/installx86.mspx

The debugging tools come with tools such as windbg (I can’t imagine doing my job without it), adplus, cdb, gflags and a bunch of other tools.

I will talk about how to use them later as we discuss hangs and crashes.

Symbol Files:

=========

.One of the most important things before getting dumps is that you need to generate symbol files (pdb) for your application and any other dlls that you may have. Without symbol files the dump will not be of much use. When we try to look at call stacks, it will only show raw addresses and it will be almost impossible to get meaningful information from dumps. If you can get a debug version of the app the fail, that will give us more information because there will be less optimization in the code and it will be easier to review dumps. In VS.net, when you build a debug version of the app, the pdb file will be generated in the same folder as the exe or dll. For VS.net, in release mode, you need to make sure that Under Project->Properties, Configuration Properties->Build, “Generate Debug Information” should be set to true For VC 6.0, you need to make sure that the following options are set.

a) Go to Project->Settings->C/C++ tab and under “Debug Info” make sure that Program Database is selected.

b) Then go to the Link tab and under “Category” choose “Debug” and under “Debug Info” frame make sure that “Debug Info” is selected with “Microsoft Format” and uncheck “Separate Types”.

c) Now build the application and this will generate the corresponding symbol file.

You can follow the above steps in release mode as well and you can generate the symbol files.

For VB 6.0 applications go to Project->Properties and “Compile” tab. In compile tab, choose “Compile to Native Code”, select “No Optimization” and check “Create Symbolic Debug Info”. When you build your app now, it will generate the symbol file.

The other important thing is, once a symbol file is generated from a build, you would need to deploy that version of the application on the target machine. Even if you do not change the code and if you rebuild your application, the symbol files will not match up and the dump will not be useful. So whenever you compile your app and get a new symbol file, the application should be redeployed. This step is very important because, dumps will be very huge and if you transfer it from other customer sites, the last thing you want is a symbol mismatch.

You can use the “SymChk” tool that is part of the “Debugging tools” to verify that you have the correct symbols. The help that comes as part of the debugging tools has examples listed.

Application Crash:

=============

I am mainly going to be talking about user mode crashes. Application crash typically happens when we try to read from memory location that has already been freed or if we try to write to a memory location whose address is invalid. When a crash occurs if dr Watson is installed as the default post mortem debugger, it will generate a log file and also a dump file. Sometime we see the following error message as well.

"The instruction at the (address ), refernce memory at (address).The memory couldn't be written."

Some times the application might just disappear.

In these situations that data that we collect is typically user mode memory dumps of the process when the crash occurred and then we analyze it. In order to generate crash dumps, here the steps that you need to follow after having generated symbol files and deploying that version of the app on to target machines

1)Start your application.

2) Go to the folder where you installed the debugging tools through command prompt

3) Then type the following command

adplus -crash -pn YourApp.exe ( When you run adplus the first time, you might be prompted to make cscript your default script interpreter, please say yes to it, Also ignore the warning about NT symbol path)

4)Re-create the application crash and once the crash occurs there will be a folder generated in the folder where you installed debugging tools and it will look like

Crash_Mode__Date_03-03-2005__Time_10-39-52AM

In this folder, you will mostly find files with extension .dmp. and bunch of other folders and files.. In the case of crash, you will typically find a file with a long name and it will have “2nc_Chance_AV” as part of it. This is the dump file when the Second Chance Exceptions was thrown. Whenever an exception occurs, you get a first chance exception and if that is not handled, then you will get a second chance exception and if there is no handler, it will crash the application. Since crashes are unexpected, in most cases we will get a second chance exception.

So when you call Microsoft support to troubleshoot crashes, this is the data that we typically collect. So if you already have it when you call support, the time it takes to go through the process of getting crash dumps and symbol files will be eliminated and will greatly increase the speed at which we can analyze the problem.

I shall stop here for today. I will talk about the data required for Hangs at a later time. If any of you want to add to this or if you have any questions please feel free to post your comments.

This posting is provided "AS IS" with no warranties, and confers no rights. OR if you wish to include a script sample in your post please add "Use of included script samples are subject to the terms specified at http://www.microsoft.com/info/cpyright.htm

My First Post

anandbms — Thu, 03 Mar 2005 02:44:00 GMT

My name is Anand Balasubramanian and I am with the VC++/VC#.net support team at Microsoft. I will be posting regularly on technical topics on VC++, C# , .Net framework and Debugging. I hope to post regularly and will try my best to keep my blog active. If I do not have much posts, it is most likely because I got very busy at work or I might be on vacation. If you have any specific topics that you want to discuss about please feel free to send me a comment and I will try to discuss whatever I know about it and possible get more information.

Debugging Basics

anandbms — Thu, 03 Mar 2005 01:59:00 GMT

This month I will be posting mostly on debugging tools and techniques. I will be writing about some debugging tools and how to use them to generate information that can help in diagnosing different problems.