• Ntdebugging Blog

    Basics of Debugging Windows

    • 8 Comments

    Hello, this is East again. This blog post is about a topic that we always skip over when discussing debugging; what and where are the tools for debugging. I will touch on the different types of debuggers, loading symbols and the basics of getting started with loading up a dump under your preferred debugger.

    Microsoft currently offers 4 types of debugging tools. With these tools you can remote debug another machine over firewire or serial cable (USB also but may not work consistently), as well as debug usermode processes and dump files.

    Command line debuggers:

    1 ) kd.exe: kernel debugger – Used to review Crash dumps created by a blue screen crash event or a stop error. (kd –z <location of dump> –y <location of symbols>)

    2 ) cdb.exe: User mode debugger for reviewing applications,  processes, and process dumps  (cdb  –z <location of dump> –y <location of symbols> )

    3 ) ntsd.exe: CDB and NTSD are virtually identical, except that NTSD spawns a new text window when it is started, whereas CDB inherits the Command Prompt window from which it was invoked.  When I refer to "CDB", it applies to both CDB and NTSD.

    Graphical User Interface Debugger:

    4) Windbg.exe is a GUI based debugger. It can debug the same things as KD & CDB using the same commands. Windbg gives you the ability to have multiple windows open simultaneously  to review source code or other selectable items under the view menu.

    I like using windbg for all of my user and kernel debugging, while  others I work with prefer kd for kernel debugging and cdb for user mode debugging.

     

    There are 32bit and 64bit debuggers available.

    NOTE: Some people use Visual Studio as well, but this blog post will not cover using Visual Studio as a debugger.

     

    You can review applications that already have started on your machine using CDB or Windbg. You can have the problematic application launch under the debugger as well:

    Cdb or Windbg

    -p <pid> specifies the decimal process ID to attach to ( use tlist or the task manger to obtain the PID)

    -psn <name> specifies the process to attach to by service name

    <application to launch> -y <symbol path>

    NOTE: windbg allows you to use menu options as well: select “Attach to a Process” on the File menu to debug a user-mode application that is currently running.

     

    What are dumps?

    Memory dumps are a record of what was in memory and the registers at the time of a crash. There are 3 types of memory dumps:

    NOTE: The type of dump that will be written upon bugcheck can be configured by right clicking my computer ->properties ->Advanced tab ->Settings, in the Write debugging section you will use the first drop down box to select what type of memory dump you want. (See KB307973)

    ·         Mini dump – is a subset of memory that is in use by the application creating the dump.

    A mini memory Dump file is written to %SystemRoot%\Minidump\Memory.dmp by default and is usually less than a 1mb in size.

    ·         Kernel only – This is used to review the machine’s kernel memory at the time of the crash.

    ·         Full/Complete – This is the largest kernel mode dump file. It contains all information from kernel and user mode address spaces that was in physical memory at the time of the dump (about the same size as the physical memory on the box).

    Kernel and Complete Memory Dumps are written to %SystemRoot%\Memory.dmp by default.

    Note: You can configure the server to crash using certain keystrokes . This would be useful when troubleshooting a hung server or a timing issue, KB244139 explains how to configure your server for a manual crash.

    You can also create dump files from an application or process, these are known as User-mode dumps.  Additional information can be found on these types dump in the Debugging Tools for Windows help file.

     

    How do I read a dump file?

    In order to make fast progress with a memory dump file, it is best to load symbol files. Symbol files contains data that the debugger uses to interpret the application or driver code. They may contain:

    -          Global variable names

    -          Function names

    Private Symbols would contain the above information and:

    -          Local variable names

    -          Source-line numbers

    -          Type information for variables, structures, etc.

     Microsoft currently has two ways you can access symbols for the Operating System:

    Service pack download site – You will need to create:

    -          Separate directories for Windows 2000 RTM, Windows 2000 SP1, Windows 2000 SP2, Windows XP RTM, etc.

    -          Separate directories for all of the above for free vs. checked build

    -          Separate directories for hotfix symbols

     

    Public symbol server – uses a symbol store, which is a collection of symbol files. The symbol server uses the time stamp & file size to match up symbols to the active binary.After getting your symbol files together, you will need a way to tell the debugger where they are located and set up some other options.

    To set the symbol path do one of the following:

    -          _NT_SYMBOL_PATH environment variable

    -          -y command line option

    -          .sympath (Set Symbol Path) debugger command

    -          WinDbg: File | Symbol File Path dialog, or CTRL+S

    To set the executable Image Path (needed for minidumps only), do one of the foolowing:

    -          -i command line option

    -          .exepath debugger command

    -          WinDbg: File | Image File Path dialog, or CTRL+i

    -          Source Path

    -          .srcpath WinDbg: File | Source File Path dialog, or CTRL+P

    If symbol errors appear when you begin, you can try the below commands to help narrow down some problems;

    !sym noisy — gives verbose symbol information

    AND

    .reload —  to reload all symbols

     

    Also using the srv* in your symbol path tells the debugger to load and save symbols being used out to a specific directory:

    srv*DownstreamStore*<symbol locations>

     

    NOTE: You must always use .reload after you change the symbol path or fix a symbol error — the debugger doesn’t automatically reload your symbols!

     

    Now that we are done with the overview, let’s configure our machine as a host computer to open memory a dump.  I will be using Microsoft Public Symbol servers and I want to store current symbols locally to my host machine.

    Using windbg I will set my current workspace symbols to: srv*c:\pubsymbols*http://msdl.microsoft.com/download/symbols

    Click the menu option File ->Symbol File Path or Ctrl + S. This will bring up an empty box that will allow you to enter or browse to your symbol path.

    If using kd you want to set an environment variable (_NT_SYMBOL_PATH) under “my computer properties -> advanced tab” to always start with your symbols set to:  “srv*c:\pubsymbols*http://msdl.microsoft.com/download/symbols” or use this same path in your command line:

    Kd –z <path to dump.file> -y srv*c:\pubsymbols*http://msdl.microsoft.com/download/symbols

     

    NOTE: Windbg will append any workspace symbol path with the one set by the _NT_SYMBOL_PATH environment variable during loading of a memory dump.

    Ok, now we know what debugger we want to use and we know our symbol locations. Let’s open our first kernel memory dump , located on <drive letter> <path to dump file>

    Using windbg, I will load a dump file using menu options File ->Open crash Dump (ctrl + D) or drag the the dump file into the debugger; you can even  start windbg at the command prompt.  My command would look like this:

    Windbg  –z C:\training\case 7f\MEMORY051308.22.DMP

    I did not use the –y for symbol path, as it is set already in my default workspace or in my environment variable.

    When the debugger first loads a dump file it displays several lines of information before giving you a prompt to get started with your commands (by default):

    Microsoft (R) Windows Debugger Version 6.9.0003.113 X86  ß debugger version

    Copyright (c) Microsoft Corporation. All rights reserved. ß Copyright of the debugger creator

    Loading Dump File [C:\training\case 7f\MEMORY051308.22.DMP] ß location of the dump file loading

    Kernel Summary Dump File: Only kernel address space is available ß type of memory dump (mini, kernel, or full)

    Symbol search path is: srv*c:\pubsymbols*http://msdl.microsoft.com/download/symbols ß Symbol path for this debug session

    Executable search path is:  ß points to the directory the executable files are located. For most situations this is not needed. For other situations please check the debugger help file.

     

    The next 4 lines talk about The OS version, service packs and how many processors are on the box

    1 -Windows Server 2003 Kernel Version 3790 (Service Pack 2) MP (8 procs) Free x86 compatible

    2 - Product: Server, suite: Enterprise TerminalServer SingleUserTS

    3 - Built by: 3790.srv03_sp2_gdr.070304-2240

    4 - Kernel base = 0x80800000 PsLoadedModuleList = 0x808a6ea8

     

    Next we would see when the machine crashed and how long it was up prior to this crash:

    Debug session time: Wed May 14 01:27:36.768 2008 (GMT-4)

    System Uptime: 0 days 16:32:51.921

     

    After completing the above process, the debugger starts loading the dump file and parsing through the loaded symbols. Here you may notice some warnings for some user space processes which are not included in the kernel dump. This is ok.

    WARNING: Process directory table base BFF0A080 doesn't match CR3 007AF000

    WARNING: Process directory table base BFF0A080 doesn't match CR3 007AF000

    Loading Kernel Symbols

    ...........................................................................................................................................

    Loading User Symbols

    PEB is paged out (Peb.Ldr = 7ffdf00c).  Type ".hh dbgerr001" for details

    Loading unloaded module list

    *******************************************************************************

    *                                                                             *

    *                        Bugcheck Analysis                                    *

    *                                                                             *

    *******************************************************************************

     

    1- Use !analyze -v to get detailed debugging information.

    2 - BugCheck 7F, {8, f773ffe0, 0, 0}

    3 - *** ERROR: Module load completed but symbols could not be loaded for ql2300.sy

    The three things I want to point out from above are:

    1 - !analyze –v: This is the debugger command used to help analyze a dump file by reviewing information passed to KeBugCheck including specific parameters of that crash. It will analyze this information and provide a definition of the bugcheck, a stack showing all current function calls, and, when possible, the name of an offending driver or process that the debugger thinks is at fault.  Please review the debugger help file for additional information in this area.

    2 – The type of bugcheck that occurred on the machine.

    3 – An error telling you about symbols missing or not available to help diagnose a particular driver or application. This can lead to a misdiagnostis if you’re not careful.

    Once loading is completed you should be at a kd> prompt. This prompt shows you the current processor you are using (if the machine has more than one).

    For this dump we are at processor 3 on an 8 proc machine:

    3: kd>

     

    To view the current crash stack location you can use the "K" command. There are multiple forms of this command, each one dumping the basic plus additional information. As functions are executed and call other functions, a call stack is created in stack memory. Here are two common commands to view the stack:

     

    3: kd> k

    ChildEBP RetAddr

    00000000 baebf0ce nt!KiTrap08+0x75

    b3a4bffc baebf737 storport!RaCallMiniportInterrupt+0x2

    b3a4c008 8088d889 storport!RaidpAdapterInterruptRoutine+0x1d

    b3a4c008 80a59d8e nt!KiInterruptDispatch+0x49

    b3a4c09c 80a5c2fc hal!HalpGenerateInterrupt+0x1d2

    b3a4c0c0 80a5c44d hal!HalpLowerIrqlHardwareInterrupts+0x108

    b3a4c0d0 808256ed hal!KfLowerIrql+0x59

    <snippet>

     

    3: kd> kb

    ChildEBP RetAddr  Args to Child

    00000000 baebf0ce 00000000 00000000 00000000 nt!KiTrap08+0x75

    b3a4bffc baebf737 97bedb88 b3a4c02c 8088d889 storport!RaCallMiniportInterrupt+0x2

    b3a4c008 8088d889 977b9e18 97bedad0 03010006 storport!RaidpAdapterInterruptRoutine+0x1d

    b3a4c008 80a59d8e 977b9e18 97bedad0 03010006 nt!KiInterruptDispatch+0x49

    b3a4c09c 80a5c2fc 97797004 97bedad0 00000102 hal!HalpGenerateInterrupt+0x1d2

    b3a4c0c0 80a5c44d 00000101 977b9e02 b3a4c0d8 hal!HalpLowerIrqlHardwareInterrupts+0x108

    b3a4c0d0 808256ed b3a4c0e8 baebf1c6 977b9bb0 hal!KfLowerIrql+0x59

    <snippet>

     

    Either one can be used depending on how much information you want to see and can use.

    This completes the Basic of Debugging Windows, Part I. I will create a Part II using specific questions gathered from our readers.

     

    Miscellaneous information:

    To go further with this topic I would suggest starting with the debugger help file included with the Microsoft Debugging Tools. 

    ADPlus – An automated way to use the cdb.exe to capture/create a usermode dump when a process hangs or crashes. (more info - http://msdn.microsoft.com/en-us/library/cc265629.aspx or kb286350)

    Public Symbols for Microsoft Operating Systems:

    Microsoft Public Symbol server : srv * DownstreamStore * http://msdl.microsoft.com/download/symbols

    example: srv*c:\mysyms*http://msdl.microsoft.com/download/symbols

     Microsoft Symbol packages http://www.microsoft.com/whdc/devtools/debugging/symbolpkg.mspx#d

    Use !Analyze-v to gather additional information about the bugcheck and a bucket-id for your dump file. The bucket-id can be submitted to Microsoft for review for similar crashes and resolutions. Try using the Microsoft Online Crash Analysis to submit your crash dump bucket-id for possible follow up from Microsoft or for Microsoft to look for trends: http://oca.microsoft.com/en/Welcome.aspx

    For concepts, tools and information about the system architecture:

    http://msdn.microsoft.com/en-us/default.aspx

    Windows Internal 4th edition (by Mark E. Russinovich & David A. Solomon) the whole book or Chapter 14 - Crash Dump Analysis

    Advanced Windows Debugging (by Mario Hewardt & Daniel Pravat )

    http://technet.microsoft.com/en-us/default.aspx

  • Ntdebugging Blog

    Transcript of Windows NT Debugging Blog Live Chat

    • 3 Comments

    For those of you that could not make the live chat on 8/13, here is the transcript of the chat session....

     

    Chat Topic: PGES-Windows NT Debugging Blog Live Chat
    Date: Wednesday, August 13, 2008

    Daniel (Moderator):
    Hello everyone-- thanks for coming to our chat on Platforms Global Escalation Services. The chat will officially get started at 1pm Eastern time. Only questions related to this topic will be addressed during this chat. Thanks!

    Daniel (Moderator):
    Hello everyone-- thanks for coming to our chat on Platforms Global Escalation Services. We'll get started in about 10 minutes.  You can start posting your questions now if you'd like and when the chat starts our Experts will begin answering them. Be sure to check the "Ask the Experts" box before you send your questions and please keep all questions on topic-- Thanks!

    Daniel (Moderator):

    Let's get started with our chat. Before we begin, though, I'd like to have our Experts introduce themselves and then they'll get started answering  your questions.

    Smoke [Windows Core] (Expert):
    Hi everyone, I'm an Escalation Engineer with the Window’s Core team.  I fix bugs for a living.

    Matthew [MSFT EE] (Expert):
    Hello, I am an Escalation Engineer with the Platforms Global Escalation Services (Windows Core) team.

    East - MSFT EE (Expert):
    I am East, an Escalation Engineer with the Microsoft Platforms Global Escalation Services. (Windows Core)



    Todd Webb - Msft (Expert):
    I am an Escalation Engineer with the Microsoft Platforms Global Escalation Services OEM hardware team...

    David (Expert):
    Hi, I'm an Escalation Engineer with Windows Core - reading code & debugging is my day-to-day.

    stheller (Expert):
    Hi, I'm a new Escalation Engineer with Platforms GES.

    Mr Ninja [MSFT EE] (Expert):
    Hi, I am an Escalation Engineer with Microsoft PGES.  I debug Windows for a living.

    Tate [MSFT EE] (Expert):
    Hi, I’m one of the EE’s on the Windows team.



    Jeff Dailey MSFT EE (Expert):
    Hi, my name is Jeff Dailey, I’m a Senior Escalation Engineer on the Microsoft Platforms Global Escalation Services team.



    Smoke [Windows Core] (Expert):
    Q: How can I track memory allocations through MmAllocateContiguousMemory?
    A: You could try poolhittag on MMCM or a breakpoint on MmAllocateContiguousMemory.  If you go with the break point, you can use a conditional breakpoint and dump the stack and anything else, then 'go' the system.  There will be a perf hit each time you break in.

    Tate [MSFT EE] (Expert):
    Q: For MmAllocatecontiguousMemory, will !poolused show the total amount used?
    A: !poolused 2 will show MmCm

    Matthew [MSFT EE] (Expert):
    Q: What's the best way to go about troubleshooting pool corruption dumps.
    A: Special Pool can be used to track down pool corruption problems.  http://msdn.microsoft.com/en-us/library/cc265889.aspx

    a-hstein (Expert):
    Greetings and sorry for the late message.  I am an intern in the GES group.

    Mr Ninja [MSFT EE] (Expert):
    Q: Could you explain the reasons why a memory dump analysis show an "illegal instruction" exception raised from a valid instruction?
    A: There are many reasons this could happen.  The instruction that was executed may not be what you see due to hardware problems such as a bit flip in the instruction when it was executed.  It is also possible for a hardware problem caused an exception to be raised on a valid instruction.  Sometimes software, or hardware, may trigger a jump to the middle of an instruction so that the instruction being executed is not what you think it is.I described a problem where we executed from the middle of an instruction in the blog http://blogs.msdn.com/ntdebugging/archive/2008/04/28/ntdebugging-puzzler-0x00000004-this-didn-t-puzzle-the-debug-ninja-how-about-you.aspx.

    Smoke [Windows Core] (Expert):
    Q: We use APC's to perform certain operations one of them is to have thread cleanup and exit.Is calling thread exit from an APC recommended ? This used to work fine, but with newer service packs we have threads exiting holding the heaplock!
    A: This sounds like a bad idea.  I would expect different ways that this could break (just like you have observed).

    David (Expert):
    Q: We use APC's to perform certain operations one of them is to have thread cleanup and exit.Is calling thread exit from an APC recommended ? This used to work fine, but with newer service packs we have threads exiting holding the heaplock!
    A: Part of the problem is that if ExitThread is called, any pending APCs on that thread's queue are lost.

    Matthew [MSFT EE] (Expert):
    Q: This question is in reference to special pool mentioned already. Is this article essentially the same as the MSDN reference?  http://support.microsoft.com/kb/188831/en-us
    A: The KB article documents enabling special pool via the registry, rather than verifier.  These are two different ways to accomplish the same thing.  Enabling it via the registry is sometimes preferred, since verifier enables additional checks beyond special pool.

    East - MSFT EE (Expert):
    Q: We use APC's to perform certain operations one of them is to have thread cleanup and exit.Is calling thread exit from an APC recommended ? This used to work fine, but with newer service packs we have threads exiting holding the heaplock!
    A: Would this help KB254956

     - If not we would need to follow-up with you for more information

    East - MSFT EE (Expert):
    Are there anything additional you want on the blog that we have not done?



    Jeff Dailey MSFT EE (Expert):
    Q: The final version of the Windows Internals Exam should be available before December 2008.  I’d like to thank all the community members that participated in the Beta.  Your feedback was very valuable.



    East - MSFT EE (Expert):
    Q: Are there anything additional you want on the blog that we have not done?



    Jeff Dailey MSFT EE (Expert):
    Q: When is the next Windows Internals exam scheduled? I would like to plan ahead.
    A: The final version of the Windows Internals Exam should be available before December 2008.  I’d like to thank all the community members that participated in the Beta.  Your feedback was very valuable.



    Matthew [MSFT EE] (Expert):
    Q: Will we get more puzzler on the blog?
    A: We’d like to do more puzzlers, but unfortunately they tend to take a lot of time, so I cannot say for sure when/if we’ll have more.



    Matthew [MSFT EE] (Expert):
    Q: How many of you in the audience are interested in more puzzlers on the ntdebugging blog?

    Smoke [Windows Core] (Expert):
    Q: Are you planning to write a book?
    A: Windows Internals is a great reference book that we all rely upon.  Additionally, you can check out: <http://www.amazon.com/Advanced-Debugging-Addison-Wesley-Microsoft-Technology/dp/0321374460>



    Tate [MSFT EE] (Expert):
    Q: As far as the blog is concerned I'm more a fan of the case studies type posts where you go through how you troubleshooted issues that you have enountered.
    A: So are we!!!

    Smoke [Windows Core] (Expert):
    Q: I'm very interested in puzzlers...
    A: Thanks for the feedback.  We will try to create some more in the future.

    Smoke [Windows Core] (Expert):
    Q: Debugging MPI apps - sometimes a crash happens on remote and the local smpd daemon will terminate the process being debugged. Using the debugger, is there a way to guard from TerminateProcess from the child? I guess that would break some security models.
    A: I'm not sure what MPI is, but this scenario sounds just like a service.  The service control manager will kill the service if it doesn't respond in a timely fashion.  With a service, there is a registry key to extend the timeout.  If such a mechanism isn't available for you, you should consider instrumentation/logging.

    East - MSFT EE (Expert):
    Q: I just skimmed over KB254956, we found APC to work. The issue here is that there are alertable waits in library modules like LSA/NDR/I_RPC calls where our APC fires which raises an user exception, gets handled and exits-thread exits holding the heap lock.
    A: We would need to discuss this further offline, how can I contact you?

    Matthew [MSFT EE] (Expert):
    Q: An award of puzzler like next edition of Windows Internals would definitely have my full attention. :)
    A: We'll consider it... thanks for the feedback!

    Jeff Dailey MSFT EE (Expert):
    Q: Have you ever found yourselves with an "unsolvable" case? :P
    A: No cases is unsolvable, nothing is truly random.  Some cases may take a very long time to resolve through multiple debugging passes, detailed code review, reverse engineering and multiple iterations of instrumentation.  In the end we find the problem.  



    Daniel (Moderator):

    Just a heads-up --we have about 15 minutes left in today's chat. Be sure to post your questions asap and our Experts will try to answer as many as possible before the chat ends. Thanks.

    Mr Ninja [MSFT EE] (Expert):
    Q: Tri-boot machine - XP, Server 2003 and Server 2000 with 2000 being the last one installed. After awhile, I got an error: "Windows 2000 could not start because the following file is missing or corrupt: \WINDOWS\SYSTEM32\CONFIG\SYSTEMd startup options for"..
    A: That is usually a known issue in Windows 2000 caused by the size of the system hive becoming too large.  We have several KB articles that describe this issue  KB269075, KB306038, KB323148, and KB277222 contain various resolutions you can try.  I have found that most often the steps in KB277222, using scrubber in a shutdown script, resolve this problem.  Starting with Windows 2003 we changed the boot architecture to prevent this problem, KB302594 describes this improvement.

    Tate [MSFT EE] (Expert):
    Q: Do you guys use USB debugging in Vista/2008? Why is that there is still one vendor that sells the debug dongle?
    A: Serial debugging works well enough most times.  Usually only if we have hardward that doesn't have a serial connection for some reason and only has USB or Firewire we try these alternates...

    East - MSFT EE (Expert):
    Q: I just skimmed over KB254956, we found APC to work. The issue here is that there are alertable waits in library modules like LSA/NDR/I_RPC calls where our APC fires which raises an user exception, gets handled and exits-thread exits holding the heap lock.
    A: On a better note it would be best to open a case with Microsoft Support - > <http://support.microsoft.com/> -> Need more help? -> Select a Product to start



    Jeff Dailey MSFT EE (Expert):
    Q: What companies are in attendance today?



    Graham (Expert):
    Q: There are lots of post mortum debuggers available, Dr Watson, NTSD, windbg, userdump, WER. Which ones do you usually recrommend your customers to use if you need to be sure to capture a dump from a crash?
    A: Userdump.exe is quite reliable for obtaining post-mortem dumps, and is easy to use.  It (along with ADPlus, which uses CDB) are good because they attach to the process and monitor exceptions, and can create dumps for times when a JIT debugger would not be able to create a thread in the process to obtain the dump.  Normally, I will set up drwtsn32 first, and if it cannot generate the dump, then I will go to userdump. 

    Smoke [Windows Core] (Expert):
    Q: How can I debug cases in which just I have the Minidump for CPU Hog? I tried !runaway and does not works
    A: The minidump alone may not be enough information.  You could try to look at the stacks and guess at what is using the CPU, but that require familiarity with the application.  You should capture a circular perfmon log with thread data.  Then get 3-5 dumps of the app.  From the perfmon log, you'll see what threads are active (and their activity profile).  From the dumps, you'll have a few snapshot of the process in motion.  Alternatively you could try a profiler like xperf.

    David (Expert):
    Q: Are there any free code coverage tools on Windows?
    A: This article describes how to obtain code coverage data:

    David (Expert):
    A: http://msdn.microsoft.com/en-us/library/ms182496.aspx

    stheller (Expert):
    http://www.microsoft.com/whdc/devtools/tools/prefast.mspx discusses the PREfast static source code analysis tool

    East - MSFT EE (Expert):
    Q: Are there any free code coverage tools on Windows?
    A: Please keep watching our blog site for the next chat - <http://blogs.msdn.com/ntdebugging>  or you can submit the question to the our blog site

    Daniel (Moderator):
    Well we're out of time for today's chat. Thank you very much to all of our guests who joined us today as well as to our Experts for answering so  many great questions. Have a great day!

     

  • Ntdebugging Blog

    New Facebook group: “Escalation Engineers”

    • 1 Comments

    Are you the final tier of escalation at a company or group that supports software?
    Are you fluent in assembly, C, C++, etc?

    Are you the voice of reason in critical situations?

    Do you spend more time debugging other people’s code than writing your own?

     

     

    If you answered the above questions with “yes”, then this new Facebook group is for you. "Escalation Engineers" http://www.facebook.com/group.php?gid=23477747996

    Jeff-

  • Ntdebugging Blog

    How to Access the User Mode Debugger from the Kernel Debugger

    • 4 Comments

    In certain cases you may want to use a user mode debugger to debug a process from within the kernel debugger.    It could be that you have an application that loads a kernel mode driver, and you want to be able to debug the user mode aspect of the application and then break into the kernel to follow the calls made to kernel.

    Here is how you do it!

    ·         Attach the kernel debugger via a serial cable (Null modem cable), USB cable or FireWire cable, and have your machine configured to be kernel debugged. The article located at  http://support.microsoft.com/kb/151981  is a good reference for pre-Vista systems.  To enable the debug options on Vista or Windows 2008 you must use bcdedit.exe because those OSes no longer use a boot.ini file. Here’s an example:

     

    bcdedit /debug {<guid>} <ON | OFF>
    bcdedit /dbgsettings SERIAL DEBUGPORT:1 BAUDRATE:115200

     

    ·         Add a new debugger key to the “Image File Execution Options” for your process.  In this case we will use notepad.exe as the target process. The new key will look like this:

     

    HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\notepad.exe

     

    ·         Add a string value under this key called “debugger” that contains the value “ntsd –d”. Here’s a screen shot of the registry changes for reference.

     

    ·         The –d option redirects the output of NTSD to the kernel debugger allowing remote control via the kernel debugger.

     

    ·         With the existence of this new key, the user mode debugger will automatically start and attach to your process when Notepad.exe starts.  Note: It’s important to remove the registry entry when you’re finished debugging.

     

    ·         You can now issue any standard NTSD Command via the kernel debugger.

     

    ·         When you are ready to break into the kernel and run under the kernel debugger simply type .breakin

     

     

    Jeff- 

  • Ntdebugging Blog

    Windows NT Debugging Blog Live Chat

    • 1 Comments

    Microsoft Platform Global Escalation Services is hosting our first live group debug chat session for the debugging community on August 13, 2008 at 10 AM PT.  We will be focusing on debugging techniques and any questions you may have about anything we’ve previously blogged about.  Also, we will try to cover some topics that were requested here.

     

    Details about the “PGES-Windows NT Debugging Blog Live Chat” can be found here: http://www.microsoft.com/communities/chats/default.mspx

  • Ntdebugging Blog

    How can I find out why the Cluster Resource Monitor dumped – Access Violation

    • 4 Comments

    Hello, my name is John Marlin, and I am a Support Escalation Engineer on the Microsoft Platform Cluster Services Support team.  I wanted to talk about the Windows 2003 Cluster Resource Monitor along with what happens when it crashes as well as how to debug it to find out why it crashed.

     

    We need to first understand what the Cluster Resource Monitor is and does.  Below is taken from the Microsoft MSDN site describing the Cluster Resource Monitor.

     

    A Resource Monitor provides a communication, monitoring, and processing layer between the Cluster service and one or more resources. Resource Monitors have the following characteristics:

    ·         A Resource Monitor always runs in a process separate from the Cluster service. If a resource fails, the Resource Monitor isolates the Cluster service from the effects. If the Cluster service fails, the Resource Monitor allows its resources to shut down gracefully.

    ·         To work with a resource, a Resource Monitor loads the resource DLL responsible for that resource type into its process.

    ·         When the Cluster service requests an operation on a resource, the Resource Monitor routes the request to the appropriate entry point function of the resource DLL responsible for the resource. The Resource Monitor performs default processing for some resource operations.

    ·         A Resource Monitor stores synchronized state data, allowing the Cluster service and resource DLLs to operate asynchronously, checking and updating resource status as needed.

    ·         A Resource Monitor periodically checks the operational status of all of its resources. For more information on this process, see Resource Failure.

     

    By default, the Cluster service creates one Resource Monitor per node.

     

    As the article states, everything currently running on the node is in the one Resource Monitor.  If the Resource Monitor crashes, the system will dump the Resource Monitor Process to a file called RESRCMON.DMP, and create a new instance of the process.  Because it must create a new one, all resources in the monitor are gone and needs to be restarted.  When this occurs, you would see the following entry in the Windows System Event Log. 

     

    Event ID:  1146

    Source:  ClusSvc

    Description:  The cluster resource monitor died unexpectedly, an attempt will be made to restart it

     

    After this, you could also see other resource failures (Event ID: 1069) as well as disk related events such as Lost Delayed Writes, etc.  You would see the disk related events because the disk(s) would be considered down and since there is data in the cache of the HBA, it has nowhere to write it.  Hence, lost delayed writes exist until the disk is brought back online.  For our examples here, we will ignore these disk related events as we will focus on why the Resource Monitor crashed.

     

    There are a couple reasons why a Resource Monitor would crash such as an Access Violation (0xC0000005) or a Deadlock (0xC0000194).  For this blog, we will talk about the Access Violation (0xC0000005).  An Access Violation will occur because a resource tried to do something it wasn’t supposed to or it is having an issue starting up.

     

    Along with the above System Event (Event ID: 1146) where the Resource Monitor died, you will see this in the Cluster Log file. 

     

    NOTE:

    The Cluster Log will convert times to Greenwich Mean Time (GMT), so you must ensure you do the proper GMT conversion of time to get to the location in the Cluster Log.

     

    00001d6c.00001b60::2008/03/04-05:28:46.114 ERR  [RM] Exception. Code = 0xc0000005, Address = 0x781449D1

    00001d6c.00001b60::2008/03/04-05:28:46.114 ERR  [RM] Exception parameters: 0, 0, 1003f, 0

    00001d6c.00001b60::2008/03/04-05:28:46.114 INFO [RM] GenerateMemoryDump: Start memory dump to file C:\WINDOWS\Cluster\resrcmon.dmp

     

    Now that we see this entry in the log, we should take a look at the Resource Monitor dump to see what caused the failure.  The first thing to examine is the register states, specifically the ESP (stack pointer) value.

     

    0:023> r

    eax=01bf0000 ebx=000f7b88 ecx=00000007 edx=7c8285ec esi=000f7b60 edi=000f7bb8

    eip=7c8285ec esp=01aed598 ebp=01aed5a8 iopl=0         nv up ei pl zr na pe nc

    cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00000246

    ntdll!KiFastSystemCallRet:

    7c8285ec c3              ret

     

    Starting at the stack pointer address 01aed598, we use the dds command to dump the raw stack.  We are looking for the value on the stack just below the routine resrcmon!GenerateMemoryDump.  It will take several iterations of the dds command to finally get to the value because the call was made much earlier in the stack.

     

    0:023> dds 01aed598

    01aed598  00740061 xpsp2res.dll

     xpsp2res+0x100061

    01aed59c  00720075 xpsp2res+0xe0075

    01aed5a0  00730065 xpsp2res+0xf0065

    01aed5a4  00610000

    *** pages removed ***

    01aedddc  0026afd8

    01aedde0  0026af28

    01aedde4  01aee034                                        <<-- pointer to Exception address stack

    01aedde8  0100e638 resrcmon!GenerateMemoryDump+0x180

    01aeddec  ffffffff

    01aeddf0  00001d6c

    01aeddf4  00000018

     

    Now that we have our value, we will use the kv command with the value 01aee034 to dump out the stack contents.

     

    0:023> kv=01aee034

    ChildEBP RetAddr  Args to Child             

    01aed628 7c826d2b 77e63eb3 000004e0 00080178 ntdll!KiFastSystemCallRet (FPO: [0,0,0])

    01aed62c 77e63eb3 000004e0 00080178 6d5b5af6 ntdll!ZwClose+0xc (FPO: [1,0,0])

    01aee034 0100e989 01aee300 01003024 00000000 kernel32!CloseHandle+0x59 (FPO: [Non-Fpo])

    01aee04c 01008b2c 01aee300 01003024 01aee300 resrcmon!GenerateExceptionReport+0x7e (FPO: [Non-Fpo])

    01aee060 76348d17 01aee300 01aee300 01aee080 resrcmon!RmpExceptionFilter+0x14 (FPO: [Non-Fpo])

    01aee070 7786d6d2 01aee300 77ecb7c0 01aee2d8 netshell!__CxxUnhandledExceptionFilter+0x4a (FPO: [Non-Fpo])

    01aee080 77e761b7 01aee300 00000000 00000000 netman!__CxxUnhandledExceptionFilter+0x4a (FPO: [Non-Fpo])

    01aee2d8 77e792a3 01aee300 77e61ac1 01aee308 kernel32!UnhandledExceptionFilter+0x12a (FPO: [Non-Fpo])

    01aee2e0 77e61ac1 01aee308 00000000 01aee308 kernel32!BaseThreadStart+0x4a (FPO: [SEH])

    01aee308 7c828752 01aee3ec 01aeffdc 01aee408 kernel32!_except_handler3+0x61 (FPO: [Uses EBP] [3,0,7])

    01aee32c 7c828723 01aee3ec 01aeffdc 01aee408 ntdll!ExecuteHandler2+0x26

    01aee3d4 7c82855e 01ace000 01aee408 01aee3ec ntdll!ExecuteHandler+0x24

    01aee3d4 781449d1 01ace000 01aee408 01aee3ec ntdll!KiUserExceptionDispatcher+0xe (FPO: [2,0,0]) (CONTEXT @ 01aee408)

    01aee6d0 10006d11 00000000 00f16914 01aeff58 msvcr80!wcslen+0x4 (FPO: [Non-Fpo])

    WARNING: Stack unwind information not available. Following frames may be wrong.

    01aee6f8 10001364 100096e4 00000000 744eecf8 JohnApp!Startup+0x5851

    01aeff78 781329bb 0009a6a8 75b03b60 00000000 JohnApp+0x1364

    01aeffb0 78132a47 00000000 77e64829 00f15d30 msvcr80!_endthreadex+0x3b (FPO: [Non-Fpo])

    01aeffb8 77e64829 00f15d30 00000000 00000000 msvcr80!_endthreadex+0xc7 (FPO: [Non-Fpo])

    01aeffec 00000000 781329e1 00f15d30 00000000 kernel32!BaseThreadStart+0x34 (FPO: [Non-Fpo])

     

    Based on the stack above, we have an exception at address 0x01aee300 which we will use to set the failing context.

     

    0:023> dc 0x01aee300

    01aee300  01aee3ec 01aee408 01aee32c 7c828752  ........,...R..|     <<-- Exception and Context Records

    01aee310  01aee3ec 01aeffdc 01aee408 01aee3c8  ................

    01aee320  01aeff6c 7c828766 01aeffdc 01aee3d4  l...f..|........

    01aee330  7c828723 01aee3ec 01aeffdc 01aee408  #..|............

    01aee340  01aee3c8 77e61a60 00000000 01aee3ec  ....`..w........

    01aee350  01aeffdc 7c8315c2 01aee3ec 01aeffdc  .......|........

    01aee360  01aee408 01aee3c8 77e61a60 00000000  ........`..w....

    01aee370  01aee3ec 00000000 00000000 00000000  ................

     

    The first DWORD is the Exception Record (0x01aee3ec) and the second DWORD is the Context Record (0x01aee408) that holds our true stack where the problem occurred.  From the Exception Record, we can see it is an Access Violation.

     

    0:023> .exr 0x01aee3ec

    ExceptionAddress: 781449d1 (msvcr80!wcslen+0x00000004)

       ExceptionCode: c0000005 (Access violation)

      ExceptionFlags: 00000000

    NumberParameters: 2

       Parameter[0]: 00000000

       Parameter[1]: 00000000

     

    So we need to jump into the saved context to get the thread that caused the Resource Monitor to crash.

     

    0:023> .cxr 0x01aee408

    eax=00000000 ebx=00f15d30 ecx=00000000 edx=00000000 esi=00000000 edi=00000000

    eip=781449d1 esp=01aee6d4 ebp=01aee6f8 iopl=0         nv up ei pl nz na po nc

    cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00010202

    msvcr80!wcslen+0x4:

    781449d1 668b08          mov     cx,word ptr [eax]        ds:0023:00000000=????

     

    0:023> kv

      *** Stack trace for last set context - .thread/.cxr resets it

    ChildEBP RetAddr  Args to Child             

    01aee6d0 10006d11 00000000 00f16914 01aeff58 msvcr80!wcslen+0x4 (FPO: [Non-Fpo])

    WARNING: Stack unwind information not available. Following frames may be wrong.

    01aee6f8 10001364 100096e4 00000000 744eecf8 JohnApp!Startup+0x5851

    01aeff78 781329bb 0009a6a8 75b03b60 00000000 JohnApp+0x1364

    01aeffb0 78132a47 00000000 77e64829 00f15d30 msvcr80!_endthreadex+0x3b (FPO: [Non-Fpo])

    01aeffb8 77e64829 00f15d30 00000000 00000000 msvcr80!_endthreadex+0xc7 (FPO: [Non-Fpo])

    01aeffec 00000000 781329e1 00f15d30 00000000 kernel32!BaseThreadStart+0x34 (FPO: [Non-Fpo])

     

    This stack reveals that JohnApp’s .DLL was in the process of a calling one of its threads to do something and caused this problem.  Now we can find out what the specific resource (in case there are multiple) that caused the problem.  In the case of an Access Violation dump, it is going to be a resource that failed or is in the process of coming online.  You can do a search through the threads using ~*kb to find a current resource trying to startup during a resrcmon termination.

     

    0:023> ~*kb

     

      10  Id: 1d6c.1394 Suspend: 0 Teb: 7fff4000 Unfrozen

    ChildEBP RetAddr  Args to Child             

    0179f78c 7c827d0b 77e61d1e 00000464 00000000 ntdll!KiFastSystemCallRet

    0179f790 77e61d1e 00000464 00000000 00000000 ntdll!NtWaitForSingleObject+0xc

    0179f800 77e61c8d 00000464 ffffffff 00000000 kernel32!WaitForSingleObjectEx+0xac

    0179f814 10002728 00000464 ffffffff 77e61d48 kernel32!WaitForSingleObject+0x12

    WARNING: Stack unwind information not available. Following frames may be wrong.

    0179f838 0100a352 0009a6a8 0179f888 0100864d JohnApp!Startup+0x1268

    0179f844 0100864d 000a88a0 0179f8a0 0179fa94 resrcmon!Resmon_Terminate+0x14          <<-- Resource we want

    0179f888 01009c3b 000a88a0 00000000 0179f8a4 resrcmon!RmpOfflineResource+0x2f1       <<-- Resource we want

    0179f89c 77c80193 00000003 02460246 00000001 resrcmon!s_RmTerminateResource+0x13

    0179f8b4 77ce33e1 01009c28 0179fa98 00000001 rpcrt4!Invoke+0x30

    0179fcb4 77ce35c4 00000000 00000000 000ef734 rpcrt4!NdrStubCall2+0x299

    0179fcd0 77c7ff7a 000ef734 0008f980 000ef734 rpcrt4!NdrServerCall2+0x19

    0179fd04 77c8042d 0100c24c 000ef734 0179fdec rpcrt4!DispatchToStubInCNoAvrf+0x38

    0179fd58 77c80353 00000005 00000000 01011458 rpcrt4!RPC_INTERFACE::DispatchToStubWorker+0x11f

    0179fd7c 77c7e0d4 000ef734 00000000 01011458 rpcrt4!RPC_INTERFACE::DispatchToStub+0xa3

    0179fdbc 77c7e080 000ef734 000ef6ec 00000000 rpcrt4!RPC_INTERFACE::DispatchToStubWithObject+0xc0

    0179fdfc 77c812f0 0008fb38 00083498 000efcf8 rpcrt4!LRPC_SCALL::DealWithRequestMessage+0x41e

    0179fe20 77c88678 000834d0 0179fe38 0008fb38 rpcrt4!LRPC_ADDRESS::DealWithLRPCRequest+0x127

    0179ff84 77c88792 0179ffac 77c8872d 00083498 rpcrt4!LRPC_ADDRESS::ReceiveLotsaCalls+0x430

    0179ff8c 77c8872d 00083498 00000000 00000000 rpcrt4!RecvLotsaCallsWrapper+0xd

    0179ffac 77c7b110 00086890 0179ffec 77e64829 rpcrt4!BaseCachedThreadRoutine+0x9d

     

    With the above information, and focusing on the resrcmon!RmpOfflineResource frame, the parameter 0x000a88a0 is our resource.  Using the DC command, you can confirm that this is the resource.

     

    0:023> dc 000a88a0

    000a88a0  63727352 00000001 000a7d90 000a9290  Rsrc.....}......

    000a88b0  000b37f0 000ad130 000a8318 000ad0f0  .7..0........... 

    000a88c0  00001388 0000ea60 10000000 0009a6a8  ....`...........

    000a88d0  00000000 00000000 00000000 00000000  ................

    000a88e0  00000000 00000001 10001540 10001910  ........@.......

    000a88f0  10001a00 10002280 100026b0 10002810  ....."...&...(..

    000a8900  100027c0 00000000 00000000 100028a0  .'...........(..

    000a8910  100029e0 00000003 00000000 0000000c  .)..............

     

    The DWORDS at offset 10, 14, 18, and 1C will reveal the information to confirm the resource such as its GUID in the registry, the .DLL being used, etc.

     

    0:005> du 0x000ad0f0                            <<-- Resource Displayed in Cluster Administrator

    000ad0f0  "Johns Resource"

     

    0:005> du 0x000a8318                            <<-- GUID in registry (HKLM\Cluster\Resources)

    000a8318  "35a73cba-6096-485e-a227-d4a8d06f"

    000a8358  "680a"

     

    0:005> du 0x000ad130                            <<-- Resource Type (HKLM\Cluster\ResourceTypes)

    000ad130  "Johns Customer Resource Type"

     

    0:005> du 0x000b37f0                            <<-- Specific DLL being Used

    000b37f0  "johnapp.dll"

     

    Now that you know the resource generating the access violation, you should consult with the vendor of this resource to find out what had happened.  It’s possible they can look at the threads with their symbols or get other information from the event logs, any dump it may create, etc.  It could be the resource has a known problem where an update is needed.

     

    The above steps will take you to where you want and I wanted to show those steps first.  In most cases, but not all, when the dump is first opened it could show the exception and the stack that we found above by simply entering the .ecxr command...

     

    0:023> .ecxr

    eax=00000000 ebx=00f15d30 ecx=00000000 edx=00000000 esi=00000000 edi=00000000

    eip=781449d1 esp=01aee6d4 ebp=01aee6f8 iopl=0         nv up ei pl nz na po nc

    cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00010202

    msvcr80!wcslen+0x4:

    781449d1 668b08          mov     cx,word ptr [eax]        ds:0023:00000000=????

     

    0:023> kb

      *** Stack trace for last set context - .thread/.cxr resets it

    ChildEBP RetAddr  Args to Child             

    01aee6d0 10006d11 00000000 00f16914 01aeff58 msvcr80!wcslen+0x4

    WARNING: Stack unwind information not available. Following frames may be wrong.

    01aee6f8 10001364 100096e4 00000000 744eecf8 JohnApp!Startup+0x5851

    01aeff78 781329bb 0009a6a8 75b03b60 00000000 JohnApp+0x1364

    01aeffb0 78132a47 00000000 77e64829 00f15d30 msvcr80!_callthreadstartex+0x1b

    01aeffb8 77e64829 00f15d30 00000000 00000000 msvcr80!_threadstartex+0x66

    01aeffec 00000000 781329e1 00f15d30 00000000 kernel32!BaseThreadStart+0x34

     

    You could also get to the same information using the above original steps (dds 01aed598) but stopping at the resrcmon!RmpExceptionFilter (Resource Monitor handles the exception) which has the exception as the first parameter.

     

    0:023> dds 01aed598

    01aed598  00740061 xpsp2res.dll

     xpsp2res+0x100061

    01aed59c  00720075 xpsp2res+0xe0075

    01aed5a0  00730065 xpsp2res+0xf0065

    01aed5a4  00610000

    *** pages removed ***

    01aedddc  0026afd8

    01aedde0  0026af28

    01aedde4  01aee034                                        <<-- pointer to Exception address stack

    01aedde8  0100e638 resrcmon!GenerateMemoryDump+0x180

    01aeddec  ffffffff

    01aeddf0  00001d6c

    01aeddf4  00000018

    *** pages removed ***

    01aee000  01005528 resrcmon!`string'+0xc

    01aee004  ffffffff

    01aee008  0100d27b resrcmon!ClRtlLogPrint+0x499

    01aee00c  0100e96c resrcmon!GenerateExceptionReport+0x61

    01aee010  00000001

    01aee014  01005bf4 resrcmon!`string'

    *** pages removed ***

    01aee048  7786d687 netman!__CxxUnhandledExceptionFilter

    01aee04c  01aee060                                        <<-- Frame 3 in kv=01aee034 above

    01aee050  01008b2c resrcmon!RmpExceptionFilter+0x14       <<-- Frame 4 in kv=01aee034 above

    01aee054  01aee300

    01aee058  01003024 resrcmon!`string'

    01aee05c  01aee300

     

Page 1 of 1 (6 items)