Basics of Debugging Windows

Basics of Debugging Windows

  • Comments 8

Hello, this is East again. This blog post is about a topic that we always skip over when discussing debugging; what and where are the tools for debugging. I will touch on the different types of debuggers, loading symbols and the basics of getting started with loading up a dump under your preferred debugger.

Microsoft currently offers 4 types of debugging tools. With these tools you can remote debug another machine over firewire or serial cable (USB also but may not work consistently), as well as debug usermode processes and dump files.

Command line debuggers:

1 ) kd.exe: kernel debugger – Used to review Crash dumps created by a blue screen crash event or a stop error. (kd –z <location of dump> –y <location of symbols>)

2 ) cdb.exe: User mode debugger for reviewing applications,  processes, and process dumps  (cdb  –z <location of dump> –y <location of symbols> )

3 ) ntsd.exe: CDB and NTSD are virtually identical, except that NTSD spawns a new text window when it is started, whereas CDB inherits the Command Prompt window from which it was invoked.  When I refer to "CDB", it applies to both CDB and NTSD.

Graphical User Interface Debugger:

4) Windbg.exe is a GUI based debugger. It can debug the same things as KD & CDB using the same commands. Windbg gives you the ability to have multiple windows open simultaneously  to review source code or other selectable items under the view menu.

I like using windbg for all of my user and kernel debugging, while  others I work with prefer kd for kernel debugging and cdb for user mode debugging.

 

There are 32bit and 64bit debuggers available.

NOTE: Some people use Visual Studio as well, but this blog post will not cover using Visual Studio as a debugger.

 

You can review applications that already have started on your machine using CDB or Windbg. You can have the problematic application launch under the debugger as well:

Cdb or Windbg

-p <pid> specifies the decimal process ID to attach to ( use tlist or the task manger to obtain the PID)

-psn <name> specifies the process to attach to by service name

<application to launch> -y <symbol path>

NOTE: windbg allows you to use menu options as well: select “Attach to a Process” on the File menu to debug a user-mode application that is currently running.

 

What are dumps?

Memory dumps are a record of what was in memory and the registers at the time of a crash. There are 3 types of memory dumps:

NOTE: The type of dump that will be written upon bugcheck can be configured by right clicking my computer ->properties ->Advanced tab ->Settings, in the Write debugging section you will use the first drop down box to select what type of memory dump you want. (See KB307973)

·         Mini dump – is a subset of memory that is in use by the application creating the dump.

A mini memory Dump file is written to %SystemRoot%\Minidump\Memory.dmp by default and is usually less than a 1mb in size.

·         Kernel only – This is used to review the machine’s kernel memory at the time of the crash.

·         Full/Complete – This is the largest kernel mode dump file. It contains all information from kernel and user mode address spaces that was in physical memory at the time of the dump (about the same size as the physical memory on the box).

Kernel and Complete Memory Dumps are written to %SystemRoot%\Memory.dmp by default.

Note: You can configure the server to crash using certain keystrokes . This would be useful when troubleshooting a hung server or a timing issue, KB244139 explains how to configure your server for a manual crash.

You can also create dump files from an application or process, these are known as User-mode dumps.  Additional information can be found on these types dump in the Debugging Tools for Windows help file.

 

How do I read a dump file?

In order to make fast progress with a memory dump file, it is best to load symbol files. Symbol files contains data that the debugger uses to interpret the application or driver code. They may contain:

-          Global variable names

-          Function names

Private Symbols would contain the above information and:

-          Local variable names

-          Source-line numbers

-          Type information for variables, structures, etc.

 Microsoft currently has two ways you can access symbols for the Operating System:

Service pack download site – You will need to create:

-          Separate directories for Windows 2000 RTM, Windows 2000 SP1, Windows 2000 SP2, Windows XP RTM, etc.

-          Separate directories for all of the above for free vs. checked build

-          Separate directories for hotfix symbols

 

Public symbol server – uses a symbol store, which is a collection of symbol files. The symbol server uses the time stamp & file size to match up symbols to the active binary.After getting your symbol files together, you will need a way to tell the debugger where they are located and set up some other options.

To set the symbol path do one of the following:

-          _NT_SYMBOL_PATH environment variable

-          -y command line option

-          .sympath (Set Symbol Path) debugger command

-          WinDbg: File | Symbol File Path dialog, or CTRL+S

To set the executable Image Path (needed for minidumps only), do one of the foolowing:

-          -i command line option

-          .exepath debugger command

-          WinDbg: File | Image File Path dialog, or CTRL+i

-          Source Path

-          .srcpath WinDbg: File | Source File Path dialog, or CTRL+P

If symbol errors appear when you begin, you can try the below commands to help narrow down some problems;

!sym noisy — gives verbose symbol information

AND

.reload —  to reload all symbols

 

Also using the srv* in your symbol path tells the debugger to load and save symbols being used out to a specific directory:

srv*DownstreamStore*<symbol locations>

 

NOTE: You must always use .reload after you change the symbol path or fix a symbol error — the debugger doesn’t automatically reload your symbols!

 

Now that we are done with the overview, let’s configure our machine as a host computer to open memory a dump.  I will be using Microsoft Public Symbol servers and I want to store current symbols locally to my host machine.

Using windbg I will set my current workspace symbols to: srv*c:\pubsymbols*http://msdl.microsoft.com/download/symbols

Click the menu option File ->Symbol File Path or Ctrl + S. This will bring up an empty box that will allow you to enter or browse to your symbol path.

If using kd you want to set an environment variable (_NT_SYMBOL_PATH) under “my computer properties -> advanced tab” to always start with your symbols set to:  “srv*c:\pubsymbols*http://msdl.microsoft.com/download/symbols” or use this same path in your command line:

Kd –z <path to dump.file> -y srv*c:\pubsymbols*http://msdl.microsoft.com/download/symbols

 

NOTE: Windbg will append any workspace symbol path with the one set by the _NT_SYMBOL_PATH environment variable during loading of a memory dump.

Ok, now we know what debugger we want to use and we know our symbol locations. Let’s open our first kernel memory dump , located on <drive letter> <path to dump file>

Using windbg, I will load a dump file using menu options File ->Open crash Dump (ctrl + D) or drag the the dump file into the debugger; you can even  start windbg at the command prompt.  My command would look like this:

Windbg  –z C:\training\case 7f\MEMORY051308.22.DMP

I did not use the –y for symbol path, as it is set already in my default workspace or in my environment variable.

When the debugger first loads a dump file it displays several lines of information before giving you a prompt to get started with your commands (by default):

Microsoft (R) Windows Debugger Version 6.9.0003.113 X86  ß debugger version

Copyright (c) Microsoft Corporation. All rights reserved. ß Copyright of the debugger creator

Loading Dump File [C:\training\case 7f\MEMORY051308.22.DMP] ß location of the dump file loading

Kernel Summary Dump File: Only kernel address space is available ß type of memory dump (mini, kernel, or full)

Symbol search path is: srv*c:\pubsymbols*http://msdl.microsoft.com/download/symbols ß Symbol path for this debug session

Executable search path is:  ß points to the directory the executable files are located. For most situations this is not needed. For other situations please check the debugger help file.

 

The next 4 lines talk about The OS version, service packs and how many processors are on the box

1 -Windows Server 2003 Kernel Version 3790 (Service Pack 2) MP (8 procs) Free x86 compatible

2 - Product: Server, suite: Enterprise TerminalServer SingleUserTS

3 - Built by: 3790.srv03_sp2_gdr.070304-2240

4 - Kernel base = 0x80800000 PsLoadedModuleList = 0x808a6ea8

 

Next we would see when the machine crashed and how long it was up prior to this crash:

Debug session time: Wed May 14 01:27:36.768 2008 (GMT-4)

System Uptime: 0 days 16:32:51.921

 

After completing the above process, the debugger starts loading the dump file and parsing through the loaded symbols. Here you may notice some warnings for some user space processes which are not included in the kernel dump. This is ok.

WARNING: Process directory table base BFF0A080 doesn't match CR3 007AF000

WARNING: Process directory table base BFF0A080 doesn't match CR3 007AF000

Loading Kernel Symbols

...........................................................................................................................................

Loading User Symbols

PEB is paged out (Peb.Ldr = 7ffdf00c).  Type ".hh dbgerr001" for details

Loading unloaded module list

*******************************************************************************

*                                                                             *

*                        Bugcheck Analysis                                    *

*                                                                             *

*******************************************************************************

 

1- Use !analyze -v to get detailed debugging information.

2 - BugCheck 7F, {8, f773ffe0, 0, 0}

3 - *** ERROR: Module load completed but symbols could not be loaded for ql2300.sy

The three things I want to point out from above are:

1 - !analyze –v: This is the debugger command used to help analyze a dump file by reviewing information passed to KeBugCheck including specific parameters of that crash. It will analyze this information and provide a definition of the bugcheck, a stack showing all current function calls, and, when possible, the name of an offending driver or process that the debugger thinks is at fault.  Please review the debugger help file for additional information in this area.

2 – The type of bugcheck that occurred on the machine.

3 – An error telling you about symbols missing or not available to help diagnose a particular driver or application. This can lead to a misdiagnostis if you’re not careful.

Once loading is completed you should be at a kd> prompt. This prompt shows you the current processor you are using (if the machine has more than one).

For this dump we are at processor 3 on an 8 proc machine:

3: kd>

 

To view the current crash stack location you can use the "K" command. There are multiple forms of this command, each one dumping the basic plus additional information. As functions are executed and call other functions, a call stack is created in stack memory. Here are two common commands to view the stack:

 

3: kd> k

ChildEBP RetAddr

00000000 baebf0ce nt!KiTrap08+0x75

b3a4bffc baebf737 storport!RaCallMiniportInterrupt+0x2

b3a4c008 8088d889 storport!RaidpAdapterInterruptRoutine+0x1d

b3a4c008 80a59d8e nt!KiInterruptDispatch+0x49

b3a4c09c 80a5c2fc hal!HalpGenerateInterrupt+0x1d2

b3a4c0c0 80a5c44d hal!HalpLowerIrqlHardwareInterrupts+0x108

b3a4c0d0 808256ed hal!KfLowerIrql+0x59

<snippet>

 

3: kd> kb

ChildEBP RetAddr  Args to Child

00000000 baebf0ce 00000000 00000000 00000000 nt!KiTrap08+0x75

b3a4bffc baebf737 97bedb88 b3a4c02c 8088d889 storport!RaCallMiniportInterrupt+0x2

b3a4c008 8088d889 977b9e18 97bedad0 03010006 storport!RaidpAdapterInterruptRoutine+0x1d

b3a4c008 80a59d8e 977b9e18 97bedad0 03010006 nt!KiInterruptDispatch+0x49

b3a4c09c 80a5c2fc 97797004 97bedad0 00000102 hal!HalpGenerateInterrupt+0x1d2

b3a4c0c0 80a5c44d 00000101 977b9e02 b3a4c0d8 hal!HalpLowerIrqlHardwareInterrupts+0x108

b3a4c0d0 808256ed b3a4c0e8 baebf1c6 977b9bb0 hal!KfLowerIrql+0x59

<snippet>

 

Either one can be used depending on how much information you want to see and can use.

This completes the Basic of Debugging Windows, Part I. I will create a Part II using specific questions gathered from our readers.

 

Miscellaneous information:

To go further with this topic I would suggest starting with the debugger help file included with the Microsoft Debugging Tools. 

ADPlus – An automated way to use the cdb.exe to capture/create a usermode dump when a process hangs or crashes. (more info - http://msdn.microsoft.com/en-us/library/cc265629.aspx or kb286350)

Public Symbols for Microsoft Operating Systems:

Microsoft Public Symbol server : srv * DownstreamStore * http://msdl.microsoft.com/download/symbols

example: srv*c:\mysyms*http://msdl.microsoft.com/download/symbols

 Microsoft Symbol packages http://www.microsoft.com/whdc/devtools/debugging/symbolpkg.mspx#d

Use !Analyze-v to gather additional information about the bugcheck and a bucket-id for your dump file. The bucket-id can be submitted to Microsoft for review for similar crashes and resolutions. Try using the Microsoft Online Crash Analysis to submit your crash dump bucket-id for possible follow up from Microsoft or for Microsoft to look for trends: http://oca.microsoft.com/en/Welcome.aspx

For concepts, tools and information about the system architecture:

http://msdn.microsoft.com/en-us/default.aspx

Windows Internal 4th edition (by Mark E. Russinovich & David A. Solomon) the whole book or Chapter 14 - Crash Dump Analysis

Advanced Windows Debugging (by Mario Hewardt & Daniel Pravat )

http://technet.microsoft.com/en-us/default.aspx

Leave a Comment
  • Please add 5 and 4 and type the answer here:
  • Post
  • PingBack from http://hubsfunnywallpaper.cn/?p=2870

  • A good basic article! I have also a basic question, maybe it is a good place to ask it here: In minidumps when viewing the stack trace I sometimes saw lines similar to this:

    nt! ?? ::FNODOBFM::`string'+0x1d093

    I guess it is a call to a function of the kernel, which was not in the minidump, thus the debugger could not resolve it, but I'm not sure. Can you explain what this call means?

    Thanks,

    Zoltan

    [This is some type of symbol mangling, probably because the instruction pointer is pointing to the wrong address or some return address on the stack is incorrect.]
  • hello,every one here:

     Could you give me a clue about how to debug ocx with windbg?

     My ocx is reporting automation error.But I don't konw how to set breakpoint at this situation.

     I have tried sxe *.But didn't work.

     Thanks.

  • Can we find out more information about kdcom.dll? I would like to know what were the problems it tried to solve, what was the approach, how does the debugging architecture looks like on Windows, the debugging stack and other information. I expect that few if any questions will be answered due to proprietary reasons.

  • Nice article for those getting into Debugging. Do we have a part 2 coming up.

  • I appreciate articles for beginners on windows debuggin. I opened up windbg, throw a dump files in it, do !analyze -v and it states something wrong with a process or a driver. Then I do a stack dump with k or kb. And then, I don't know what to do next....

    Can you show us tips on how to continue?? Like, how do u know where to look at after you dump stack and which parameter to look at...

    Well, my problem is, after stack dump, what next can I do?

    [If you are looking at a dump of a bugcheck, the description will sometimes tell you what to look for next. Also, the help file has good information on what to do next, including techniques, tips and tricks. Also take a look at kernel_debugging_tutorial.doc that ships with the debuggers.]
  • Hi All, Debugging a dump from a hung server may not be something you do every day so you may want to

  • Hello ntdebuggers! We’ve seen a lot of interest in our Puzzlers , and we’ve also seen requests and interest

Page 1 of 1 (8 items)