Hello, this is East again. This blog post is about a topic that we always skip over when discussing debugging; what and where are the tools for debugging. I will touch on the different types of debuggers, loading symbols and the basics of getting started with loading up a dump under your preferred debugger.
Microsoft currently offers 4 types of debugging tools. With these tools you can remote debug another machine over firewire or serial cable (USB also but may not work consistently), as well as debug usermode processes and dump files.
Command line debuggers:
1 ) kd.exe: kernel debugger – Used to review Crash dumps created by a blue screen crash event or a stop error. (kd –z <location of dump> –y <location of symbols>)
2 ) cdb.exe: User mode debugger for reviewing applications, processes, and process dumps (cdb –z <location of dump> –y <location of symbols> )
3 ) ntsd.exe: CDB and NTSD are virtually identical, except that NTSD spawns a new text window when it is started, whereas CDB inherits the Command Prompt window from which it was invoked. When I refer to "CDB", it applies to both CDB and NTSD.
Graphical User Interface Debugger:
4) Windbg.exe is a GUI based debugger. It can debug the same things as KD & CDB using the same commands. Windbg gives you the ability to have multiple windows open simultaneously to review source code or other selectable items under the view menu.
I like using windbg for all of my user and kernel debugging, while others I work with prefer kd for kernel debugging and cdb for user mode debugging.
There are 32bit and 64bit debuggers available.
NOTE: Some people use Visual Studio as well, but this blog post will not cover using Visual Studio as a debugger.
You can review applications that already have started on your machine using CDB or Windbg. You can have the problematic application launch under the debugger as well:
Cdb or Windbg
-p <pid> specifies the decimal process ID to attach to ( use tlist or the task manger to obtain the PID)
-psn <name> specifies the process to attach to by service name
<application to launch> -y <symbol path>
NOTE: windbg allows you to use menu options as well: select “Attach to a Process” on the File menu to debug a user-mode application that is currently running.
What are dumps?
Memory dumps are a record of what was in memory and the registers at the time of a crash. There are 3 types of memory dumps:
NOTE: The type of dump that will be written upon bugcheck can be configured by right clicking my computer ->properties ->Advanced tab ->Settings, in the Write debugging section you will use the first drop down box to select what type of memory dump you want. (See KB307973)
· Mini dump – is a subset of memory that is in use by the application creating the dump.
A mini memory Dump file is written to %SystemRoot%\Minidump\Memory.dmp by default and is usually less than a 1mb in size.
· Kernel only – This is used to review the machine’s kernel memory at the time of the crash.
· Full/Complete – This is the largest kernel mode dump file. It contains all information from kernel and user mode address spaces that was in physical memory at the time of the dump (about the same size as the physical memory on the box).
Kernel and Complete Memory Dumps are written to %SystemRoot%\Memory.dmp by default.
Note: You can configure the server to crash using certain keystrokes . This would be useful when troubleshooting a hung server or a timing issue, KB244139 explains how to configure your server for a manual crash.
You can also create dump files from an application or process, these are known as User-mode dumps. Additional information can be found on these types dump in the Debugging Tools for Windows help file.
How do I read a dump file?
In order to make fast progress with a memory dump file, it is best to load symbol files. Symbol files contains data that the debugger uses to interpret the application or driver code. They may contain:
- Global variable names
- Function names
Private Symbols would contain the above information and:
- Local variable names
- Source-line numbers
- Type information for variables, structures, etc.
Microsoft currently has two ways you can access symbols for the Operating System:
Service pack download site – You will need to create:
- Separate directories for Windows 2000 RTM, Windows 2000 SP1, Windows 2000 SP2, Windows XP RTM, etc.
- Separate directories for all of the above for free vs. checked build
- Separate directories for hotfix symbols
Public symbol server – uses a symbol store, which is a collection of symbol files. The symbol server uses the time stamp & file size to match up symbols to the active binary.After getting your symbol files together, you will need a way to tell the debugger where they are located and set up some other options.
To set the symbol path do one of the following:
- _NT_SYMBOL_PATH environment variable
- -y command line option
- .sympath (Set Symbol Path) debugger command
- WinDbg: File | Symbol File Path dialog, or CTRL+S
To set the executable Image Path (needed for minidumps only), do one of the foolowing:
- -i command line option
- .exepath debugger command
- WinDbg: File | Image File Path dialog, or CTRL+i
- Source Path
- .srcpath WinDbg: File | Source File Path dialog, or CTRL+P
If symbol errors appear when you begin, you can try the below commands to help narrow down some problems;
!sym noisy — gives verbose symbol information
AND
.reload — to reload all symbols
Also using the srv* in your symbol path tells the debugger to load and save symbols being used out to a specific directory:
srv*DownstreamStore*<symbol locations>
NOTE: You must always use .reload after you change the symbol path or fix a symbol error — the debugger doesn’t automatically reload your symbols!
Now that we are done with the overview, let’s configure our machine as a host computer to open memory a dump. I will be using Microsoft Public Symbol servers and I want to store current symbols locally to my host machine.
Using windbg I will set my current workspace symbols to: srv*c:\pubsymbols*http://msdl.microsoft.com/download/symbols
Click the menu option File ->Symbol File Path or Ctrl + S. This will bring up an empty box that will allow you to enter or browse to your symbol path.
If using kd you want to set an environment variable (_NT_SYMBOL_PATH) under “my computer properties -> advanced tab” to always start with your symbols set to: “srv*c:\pubsymbols*http://msdl.microsoft.com/download/symbols” or use this same path in your command line:
Kd –z <path to dump.file> -y srv*c:\pubsymbols*http://msdl.microsoft.com/download/symbols
NOTE: Windbg will append any workspace symbol path with the one set by the _NT_SYMBOL_PATH environment variable during loading of a memory dump.
Ok, now we know what debugger we want to use and we know our symbol locations. Let’s open our first kernel memory dump , located on <drive letter> <path to dump file>
Using windbg, I will load a dump file using menu options File ->Open crash Dump (ctrl + D) or drag the the dump file into the debugger; you can even start windbg at the command prompt. My command would look like this:
Windbg –z C:\training\case 7f\MEMORY051308.22.DMP
I did not use the –y for symbol path, as it is set already in my default workspace or in my environment variable.
When the debugger first loads a dump file it displays several lines of information before giving you a prompt to get started with your commands (by default):
Microsoft (R) Windows Debugger Version 6.9.0003.113 X86 ß debugger version
Copyright (c) Microsoft Corporation. All rights reserved. ß Copyright of the debugger creator
Loading Dump File [C:\training\case 7f\MEMORY051308.22.DMP] ß location of the dump file loading
Kernel Summary Dump File: Only kernel address space is available ß type of memory dump (mini, kernel, or full)
Symbol search path is: srv*c:\pubsymbols*http://msdl.microsoft.com/download/symbols ß Symbol path for this debug session
Executable search path is: ß points to the directory the executable files are located. For most situations this is not needed. For other situations please check the debugger help file.
The next 4 lines talk about The OS version, service packs and how many processors are on the box
1 -Windows Server 2003 Kernel Version 3790 (Service Pack 2) MP (8 procs) Free x86 compatible
2 - Product: Server, suite: Enterprise TerminalServer SingleUserTS
3 - Built by: 3790.srv03_sp2_gdr.070304-2240
4 - Kernel base = 0x80800000 PsLoadedModuleList = 0x808a6ea8
Next we would see when the machine crashed and how long it was up prior to this crash:
Debug session time: Wed May 14 01:27:36.768 2008 (GMT-4)
System Uptime: 0 days 16:32:51.921
After completing the above process, the debugger starts loading the dump file and parsing through the loaded symbols. Here you may notice some warnings for some user space processes which are not included in the kernel dump. This is ok.
WARNING: Process directory table base BFF0A080 doesn't match CR3 007AF000
WARNING: Process directory table base BFF0A080 doesn't match CR3 007AF000
Loading Kernel Symbols
...........................................................................................................................................
Loading User Symbols
PEB is paged out (Peb.Ldr = 7ffdf00c). Type ".hh dbgerr001" for details
Loading unloaded module list
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************
1- Use !analyze -v to get detailed debugging information.
2 - BugCheck 7F, {8, f773ffe0, 0, 0}
3 - *** ERROR: Module load completed but symbols could not be loaded for ql2300.sy
The three things I want to point out from above are:
1 - !analyze –v: This is the debugger command used to help analyze a dump file by reviewing information passed to KeBugCheck including specific parameters of that crash. It will analyze this information and provide a definition of the bugcheck, a stack showing all current function calls, and, when possible, the name of an offending driver or process that the debugger thinks is at fault. Please review the debugger help file for additional information in this area.
2 – The type of bugcheck that occurred on the machine.
3 – An error telling you about symbols missing or not available to help diagnose a particular driver or application. This can lead to a misdiagnostis if you’re not careful.
Once loading is completed you should be at a kd> prompt. This prompt shows you the current processor you are using (if the machine has more than one).
For this dump we are at processor 3 on an 8 proc machine:
3: kd>
To view the current crash stack location you can use the "K" command. There are multiple forms of this command, each one dumping the basic plus additional information. As functions are executed and call other functions, a call stack is created in stack memory. Here are two common commands to view the stack:
3: kd> k
ChildEBP RetAddr
00000000 baebf0ce nt!KiTrap08+0x75
b3a4bffc baebf737 storport!RaCallMiniportInterrupt+0x2
b3a4c008 8088d889 storport!RaidpAdapterInterruptRoutine+0x1d
b3a4c008 80a59d8e nt!KiInterruptDispatch+0x49
b3a4c09c 80a5c2fc hal!HalpGenerateInterrupt+0x1d2
b3a4c0c0 80a5c44d hal!HalpLowerIrqlHardwareInterrupts+0x108
b3a4c0d0 808256ed hal!KfLowerIrql+0x59
<snippet>
3: kd> kb
ChildEBP RetAddr Args to Child
00000000 baebf0ce 00000000 00000000 00000000 nt!KiTrap08+0x75
b3a4bffc baebf737 97bedb88 b3a4c02c 8088d889 storport!RaCallMiniportInterrupt+0x2
b3a4c008 8088d889 977b9e18 97bedad0 03010006 storport!RaidpAdapterInterruptRoutine+0x1d
b3a4c008 80a59d8e 977b9e18 97bedad0 03010006 nt!KiInterruptDispatch+0x49
b3a4c09c 80a5c2fc 97797004 97bedad0 00000102 hal!HalpGenerateInterrupt+0x1d2
b3a4c0c0 80a5c44d 00000101 977b9e02 b3a4c0d8 hal!HalpLowerIrqlHardwareInterrupts+0x108
b3a4c0d0 808256ed b3a4c0e8 baebf1c6 977b9bb0 hal!KfLowerIrql+0x59
<snippet>
Either one can be used depending on how much information you want to see and can use.
This completes the Basic of Debugging Windows, Part I. I will create a Part II using specific questions gathered from our readers.
Miscellaneous information:
To go further with this topic I would suggest starting with the debugger help file included with the Microsoft Debugging Tools.
ADPlus – An automated way to use the cdb.exe to capture/create a usermode dump when a process hangs or crashes. (more info - http://msdn.microsoft.com/en-us/library/cc265629.aspx or kb286350)
Public Symbols for Microsoft Operating Systems:
Microsoft Public Symbol server : srv * DownstreamStore * http://msdl.microsoft.com/download/symbols
example: srv*c:\mysyms*http://msdl.microsoft.com/download/symbols
Microsoft Symbol packages http://www.microsoft.com/whdc/devtools/debugging/symbolpkg.mspx#d
Use !Analyze-v to gather additional information about the bugcheck and a bucket-id for your dump file. The bucket-id can be submitted to Microsoft for review for similar crashes and resolutions. Try using the Microsoft Online Crash Analysis to submit your crash dump bucket-id for possible follow up from Microsoft or for Microsoft to look for trends: http://oca.microsoft.com/en/Welcome.aspx
For concepts, tools and information about the system architecture:
http://msdn.microsoft.com/en-us/default.aspx
Windows Internal 4th edition (by Mark E. Russinovich & David A. Solomon) the whole book or Chapter 14 - Crash Dump Analysis
Advanced Windows Debugging (by Mario Hewardt & Daniel Pravat )
http://technet.microsoft.com/en-us/default.aspx