I was discussing debugging with some folk internally that didn’t really have much exposure to it, but wanted to learn.  I considered the items pretty basic and didn’t really dive into to much, but I had a few comments that the information was good.  One thing I found that was interesting was that I did some searching around, and couldn’t really find a good reference that summed up the items below.  I found some information on the individual items, but I knew what to go look for.  From the perspective of someone just getting started, it wasn’t obvious.  So, I figured I would share out what I put together as it may be helpful for someone else.

NOTE:  All examples below are using the public Debugger along with public symbols and extensions.  These are available to everyone.

Intro to the Intro

There are some concepts that go along with Debugging that are sometimes not addressed directly when we look at the topic of debugging itself.  These concepts are extremely helpful when we start going through dumps and understand how to connect the dots.  Foundational knowledge really helps in this complex topic.  I will try to add some public references to some items you can read up on, but this shouldn’t be where you stop.  If you really are interested in this topic, there is a wealth of information out there that can help with some of the background. 

A good place to start is the Windows Internals book by David Solomon and Mark Russinovich.  Specifically the chapter on Memory Management and how this works. When talking about Dumps and Debugging, we are working with the contents of Memory. Understanding how memory works is extremely helpful.  Note:  Volume 6 Part 1 was just recently release, but it looks like the Memory Management pieces for Volume 6 will be in Part 2. 

Having some development experience is also helpful.  While you may not need to look at Code directly in a dump, you are looking at the results of code.  The concept of Pointers is sometimes hard for someone to grasp that doesn’t necessarily have a programming background, but when dealing with a memory dump, or debugging, the understanding of how Pointers work can be very helpful.  Even when we are debugging a Managed (.NET) application instead of a Native (C++ or non-managed) Application.  Another good book is Windows via C/C++ by Jeffrey Rickter and Christophe Nasarre. 

The last book I will mention is Debugging Applications by John Robbins.  It should be obvious why I’m recommending it – having to do with Debugging and all.

The above books are not for the feint of heart, but do provide a lot of great information.  From a support perspective, these are two books you see on most people’s bookshelves.  I definitely recommend them for yours and will really help with regards to this topic.

Concepts

Here are some links to some core concepts that I will talk about below.  These can be general pointers to help explain some of the items I talk about below.

Process & Threads

Memory Management

What is a Dump?

A dump is basically the contents of Memory written out to a file.  The contents of this can vary depending on how the dump was generated.  A Kernel Dump is a dump of Windows itself, including all applications running on the system.  A user mode dump is a dump created for a specific process (i.e. an application like Notepad).  Think of a Memory dump as a snapshot of that application.  You can then poke around and see what was in Memory at that point in time.

Are there different kinds of dumps?

When I work with a dump, there are really two kinds that I come in contact with. 

A full dump is everything in memory for that process.  This includes any modules that are loaded, Handle Tables, Thread stacks and other information that is application specific.  A mini dump only includes selected parts of the process and can be set with options when the dump is created.  Have a look at the .dump command to see what some of those options are.

Applications in general can also capture dumps and include/exclude specific regions of memory.  SQL Server is a great example of this.  They have another dump option that we refer to as a Filtered Dump.  A filtered dump is really a Full dump but it excludes the memory region for the Buffer Pool.  When we want a full dump, if we included the Buffer Pool, this could be really huge.  Think of a 64GB box that SQL is running on and has been active for a while.  The Buffer Pool is probably pretty large.  If we did a Full Dump on SQL, that dump size would probably be close to 64GB, at least I would hope it would be (depending on what you configured Max Server Memory to).  However, if we grab a Filtered Dump, this dump may only be 1-3GB in size.  That’s a big difference!  And really, we don’t usually even care about what is in the Buffer Pool when looking at a dump.

You can also have a Crash Dump or a Hang dump.  A Crash dump is a dump triggered by an error (or Exception).  A Hang dump is a dump you manually invoke a dump.  This is great to see if what is going on if we are encountering a hang in the application, or you just want to poke around.

Working with Threads

Moving between threads is important in a dump.  When you open a dump, if it is a hang dump, it will usually open on thread 0, although this isn’t a guarantee.  A Crash dump will typically open on the Thread where the exception was thrown, which is helpful.  You can move around to other threads though.  You can use the ~ command to see what threads we have out there.

0:013> ~
   0  Id: cfc.1104 Suspend: 1 Teb: 7efdd000 Unfrozen
   1  Id: cfc.12a0 Suspend: 1 Teb: 7efd7000 Unfrozen
   2  Id: cfc.e50 Suspend: 1 Teb: 7efaf000 Unfrozen
   3  Id: cfc.12e8 Suspend: 1 Teb: 7efac000 Unfrozen
   4  Id: cfc.1344 Suspend: 1 Teb: 7efa9000 Unfrozen
   5  Id: cfc.798 Suspend: 1 Teb: 7efa6000 Unfrozen
   6  Id: cfc.1054 Suspend: 1 Teb: 7efa3000 Unfrozen
   7  Id: cfc.1214 Suspend: 1 Teb: 7ef3f000 Unfrozen
   8  Id: cfc.134c Suspend: 1 Teb: 7ef3c000 Unfrozen
   9  Id: cfc.fb4 Suspend: 1 Teb: 7ef39000 Unfrozen
  10  Id: cfc.13a4 Suspend: 1 Teb: 7ef33000 Unfrozen
  11  Id: cfc.113c Suspend: 1 Teb: 7ef36000 Unfrozen
  12  Id: cfc.11a4 Suspend: 1 Teb: 7ef30000 Unfrozen
. 13  Id: cfc.e40 Suspend: 1 Teb: 7efda000 Unfrozen

You’ll notice the “0:013>” on the command.  The 13 here indicates the current thread we are on from a context point of view.  So, if we a command like k, it is within the context of Thread 13.  We can then switch to a different thread by doing ~<thread #>s.

0:013> ~5s
eax=c0000034 ebx=00000001 ecx=00000000 edx=00000000 esi=00000000 edi=00000000
eip=77ce9bd5 esp=06b8f194 ebp=06b8f22c iopl=0         nv up ei pl nz na po nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202
ntdll!ZwWaitForMultipleObjects+0x15:
77ce9bd5 c21400          ret     14h

Whenever you switch to a new thread, you will see the register information. You will then be in the context of that thread (5 in this case).

image

 

Debuggers

There are different ways to debug a dump.  The main tool that I use to review a dump is WinDBG.  WinDBG is part of the Debugging Tools for Windows and is currently part of the Windows SDK.  Although, you can just choose to download the Debugging Tools from the options of the Windows SDK setup wizard.  This will install both the x64 and x86 version of WinDBG.  I always have both on my system.  If it is a 32bit process, I use the x86 version of WinDBG.  If it is a 64bit process, I use the x64 version of WinDBG. 

You can technically use the x64 version of WinDBG with a 32bit process and make use of the .effmach command, but this won’t work properly with managed dumps.  And, I’ve seen other issues with using this.  I just stick to the same bitness when matching Debugger with Process to be safe.

DebugDiag is a great tool to capture a dump, and also offers some Analyzers that may help to point something out, but doesn’t let you freely debug a dump.  CDB is also a debugger that comes with the Debugging Tools for Windows that you can use.  Also, KD for kernel debugging.

Also, don’t forget about Visual Studio.  VS is a great Debugger.  Maybe not with a raw dump, but can help with live debugging an application you may have and give you insight.

Symbols

http://msdn.microsoft.com/en-us/library/windows/desktop/ee416588(v=vs.85).aspx

For our purposes we are referring to PDB files that are generated when compiling an application.  There are older formats of debugging information, but we will focus on the PDB Format.  You may have PDB’s for your own application.  For Microsoft Application, we have a public symbol server (http://msdl.microsoft.com/download/symbols) that you can use to get public symbols.  You may be asking why I care about this.  Here is a thread stack with any symbols loaded.

0:054> k
Child-SP          RetAddr           Call Site
00000000`183cf118 000007fa`5376db54 ntdll!ZwSignalAndWaitForSingleObject+0xa
00000000`183cf120 00000000`00ae4b08 KERNELBASE!SignalObjectAndWait+0xc8
00000000`183cf1d0 00000000`00ae2d76 sqlservr+0x4b08
00000000`183cf920 00000000`00ae2700 sqlservr+0x2d76
00000000`183cf960 00000000`00aeae8d sqlservr+0x2700

Here is the same stack after getting symbols loaded for SQL Server from the public symbol server.

0:054> k
Child-SP          RetAddr           Call Site
00000000`183cf118 000007fa`5376db54 ntdll!NtSignalAndWaitForSingleObject+0xa
00000000`183cf120 00000000`00ae4b08 KERNELBASE!SignalObjectAndWait+0xc8
00000000`183cf1d0 00000000`00ae2d76 sqlservr!SOS_Scheduler::Switch+0x181
00000000`183cf920 00000000`00ae2700 sqlservr!SOS_Scheduler::SuspendNonPreemptive+0xca
00000000`183cf960 00000000`00aeae8d sqlservr!SOS_Scheduler::Suspend+0x2d

 

You may notice, after you install the debugger, that you don’t have any symbols.  You may also see the following notice when you open a dump.

Symbol search path is: *** Invalid ***
****************************************************************************
* Symbol loading may be unreliable without a symbol search path.           *
* Use .symfix to have the debugger choose a symbol path.                   *
* After setting your symbol path, use .reload to refresh symbol locations. *
****************************************************************************

It mentions the .symfix comment.  Here is what happens.  We can look at the Symbol path by using the .sympath command.  It will be empty.  Here is what we see when doing the .symfix.

0:054> .symfix
0:054> .sympath
Symbol search path is: srv*
Expanded Symbol search path is: cache*;SRV*http://msdl.microsoft.com/download/symbols

It will add in the default symbol path, which is the Microsoft Public Symbol Server.  You can then further modify it by issuing another .sympath command.  For example:

0:054> .sympath SRV*c:\symbols\public\*http://msdl.microsoft.com/download/symbols;c:\symbols\custom
Symbol search path is: SRV*c:\symbols\public\*http://msdl.microsoft.com/download/symbols;c:\symbols\custom
Expanded Symbol search path is: srv*c:\symbols\public\*http://msdl.microsoft.com/download/symbols;c:\symbols\custom

This means that I will cache any symbols from the public symbol server into c:\symbols\public and any Symbols I also want to use, I can stick in my custom directory.

You can also set an System Environment Variable with your Symbol path that other applications can use as well, such as Visual Studio.  It is _NT_SYMBOL_PATH.

There is also a difference between Private and Public symbols.  When generating symbols for your application, you can strip out certain information.  The Public symbols don’t contain all the information that internal (private) symbols provide.  The article I referenced above for symbols goes into this if you really want to understand it.

Debugger Extensions

Debugging Extensions can be really helpful to get information out of a dump with minimal effort as compared to finding it by hand.  Extensions essentially contain pre-made commands to get specific information.  The debugger comes with some extensions already.  You can see some of them by using the .chain command to see which ones are loaded.

0:054> .chain
Extension DLL search Path:
    C:\Program Files (x86)\Windows Kits\8.0\Debuggers\x64\WINXP;C:\Program Files (x86)\Windows Kits\8.0\Debuggers\x64\winext;C:\Program Files (x86)\Windows Kits\8.0\Debuggers\x64\winext\arcade;C:\Program Files (x86)\Windows Kits\8.0\Debuggers\x64\pri;C:\Program Files (x86)\Windows Kits\8.0\Debuggers\x64;C:\Program Files (x86)\Windows Kits\8.0\Debuggers\x64\winext\arcade;C:\Program Files\Microsoft Passport RPS\;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Microsoft SQL Server\100\Tools\Binn\;C:\Program Files\Microsoft SQL Server\100\Tools\Binn\;C:\Program Files\Microsoft SQL Server\100\DTS\Binn\;C:\Program Files (x86)\Microsoft SQL Server\100\Tools\Binn\VSShell\Common7\IDE\;C:\Program Files (x86)\Microsoft SQL Server\100\DTS\Binn\;C:\Program Files\Microsoft Network Monitor 3\
Extension DLL chain:
    dbghelp: image 6.2.8229.0, API 6.2.6, built Thu Feb 09 22:51:14 2012
        [path: C:\Program Files (x86)\Windows Kits\8.0\Debuggers\x64\dbghelp.dll]
    ext: image 6.2.8229.0, API 1.0.0, built Thu Feb 09 22:56:01 2012
        [path: C:\Program Files (x86)\Windows Kits\8.0\Debuggers\x64\winext\ext.dll]
    exts: image 6.2.8229.0, API 1.0.0, built Thu Feb 09 23:05:32 2012
        [path: C:\Program Files (x86)\Windows Kits\8.0\Debuggers\x64\WINXP\exts.dll]
    uext: image 6.2.8229.0, API 1.0.0, built Thu Feb 09 22:59:16 2012
        [path: C:\Program Files (x86)\Windows Kits\8.0\Debuggers\x64\winext\uext.dll]
    ntsdexts: image 6.2.8229.0, API 1.0.0, built Thu Feb 09 22:59:54 2012
        [path: C:\Program Files (x86)\Windows Kits\8.0\Debuggers\x64\WINXP\ntsdexts.dll]

The Debugger Extension that I use the most is the Managed Code Extension.  I use an internal version that takes advantage of private symbols, but there is a public version that ships with the .NET Framework called SOS.  You can load this Extension by using the following command.

0:069> .loadby sos mscorwks

We can see that it is loaded by looking at .chain again.

0:069> .chain
Extension DLL search Path:
    C:\Program Files (x86)\Windows Kits\8.0\Debuggers\x64\WINXP;C:\Program Files (x86)\Windows Kits\8.0\Debuggers\x64\winext;C:\Program Files (x86)\Windows Kits\8.0\Debuggers\x64\winext\arcade;C:\Program Files (x86)\Windows Kits\8.0\Debuggers\x64\pri;C:\Program Files (x86)\Windows Kits\8.0\Debuggers\x64;C:\Program Files (x86)\Windows Kits\8.0\Debuggers\x64\winext\arcade;C:\Program Files\Microsoft Passport RPS\;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Microsoft SQL Server\100\Tools\Binn\;C:\Program Files\Microsoft SQL Server\100\Tools\Binn\;C:\Program Files\Microsoft SQL Server\100\DTS\Binn\;C:\Program Files (x86)\Microsoft SQL Server\100\Tools\Binn\VSShell\Common7\IDE\;C:\Program Files (x86)\Microsoft SQL Server\100\DTS\Binn\;C:\Program Files\Microsoft Network Monitor 3\
Extension DLL chain:
    C:\Windows\Microsoft.NET\Framework64\v2.0.50727\sos: image 2.0.50727.4216, API 1.0.0, built Thu Jul 07 00:47:51 2011
        [path: C:\Windows\Microsoft.NET\Framework64\v2.0.50727\sos.dll]
    dbghelp: image 6.2.8229.0, API 6.2.6, built Thu Feb 09 22:51:14 2012
        [path: C:\Program Files (x86)\Windows Kits\8.0\Debuggers\x64\dbghelp.dll]
    ext: image 6.2.8229.0, API 1.0.0, built Thu Feb 09 22:56:01 2012
        [path: C:\Program Files (x86)\Windows Kits\8.0\Debuggers\x64\winext\ext.dll]
    exts: image 6.2.8229.0, API 1.0.0, built Thu Feb 09 23:05:32 2012
        [path: C:\Program Files (x86)\Windows Kits\8.0\Debuggers\x64\WINXP\exts.dll]
    uext: image 6.2.8229.0, API 1.0.0, built Thu Feb 09 22:59:16 2012
        [path: C:\Program Files (x86)\Windows Kits\8.0\Debuggers\x64\winext\uext.dll]
    ntsdexts: image 6.2.8229.0, API 1.0.0, built Thu Feb 09 22:59:54 2012
        [path: C:\Program Files (x86)\Windows Kits\8.0\Debuggers\x64\WINXP\ntsdexts.dll]

 

You can also run .help on an extension to see what some of the commands are that are available to you.  Assuming that that extension produces results.  Also note, that for any command on an extension, you need to prefix it with an exclamation mark (!) or a period (.).  Here is the beginning of the Help output for the SOS Extension.

0:069> !sos.help
-------------------------------------------------------------------------------
SOS is a debugger extension DLL designed to aid in the debugging of managed
programs. Functions are listed by category, then roughly in order of
importance. Shortcut names for popular functions are listed in parenthesis.
Type "!help <functionname>" for detailed info on that function.

Object Inspection                  Examining code and stacks
-----------------------------      -----------------------------
DumpObj (do)                       Threads
DumpArray (da)                     CLRStack
DumpStackObjects (dso)             IP2MD
DumpHeap                           U
DumpVC                             DumpStack
GCRoot                             EEStack
ObjSize                            GCInfo
FinalizeQueue                      EHInfo
PrintException (pe)                COMState
TraverseHeap                       BPMD

While there are some internal extensions that I can use, that doesn’t mean that you can’t get a lot out of a dump.  The reason the Extensions I use are internal is that they have a dependency on Internal Symbols.

Basic Commands

There are some common commands that you can use to poke around and discover what is in a dump.  I’ll talk about a few here, but if you really want to go through different commands that are available, check out the help file that comes with the debugger.  I know that sounds like I’m signaling defeat by reading the manual, but it seriously has some good info with some good examples.  I often refer back to the Debugger Help to look up a command.  If you don’t use it all the time, you sometimes have to go back and refresh your memory.  Especially when it comes to dumps.

k

The k command is used to generate a stack dump on a given thread.  There are also variants that you can use such as kp or kn, or even knp.  The debugger help outlines all of these modifiers you can use.  Lets look at the example I had above.

0:054> k
Child-SP          RetAddr           Call Site
00000000`183cf118 000007fa`5376db54 ntdll!NtSignalAndWaitForSingleObject+0xa
00000000`183cf120 00000000`00ae4b08 KERNELBASE!SignalObjectAndWait+0xc8
00000000`183cf1d0 00000000`00ae2d76 sqlservr!SOS_Scheduler::Switch+0x181
00000000`183cf920 00000000`00ae2700 sqlservr!SOS_Scheduler::SuspendNonPreemptive+0xca
00000000`183cf960 00000000`00aeae8d sqlservr!SOS_Scheduler::Suspend+0x2d
00000000`183cf990 00000000`00aeaf16 sqlservr!WorkDispatcher::DequeueTask+0x472
00000000`183cfa20 00000000`00c244fa sqlservr!SOS_Scheduler::ProcessTasks+0x76
00000000`183cfa90 00000000`00c247dd sqlservr!SchedulerManager::WorkerEntryPoint+0x2d2
00000000`183cfb70 00000000`0106c0cd sqlservr!SystemThread::RunWorker+0xcc
00000000`183cfbb0 00000000`00c253d2 sqlservr!SystemThreadDispatcher::ProcessWorker+0x2db
00000000`183cfc60 00000000`77c337d7 sqlservr!SchedulerManager::ThreadEntryPoint+0x173
00000000`183cfd00 00000000`77c33894 msvcr80!_callthreadstartex+0x17
[f:\dd\vctools\crt_bld\self_64_amd64\crt\src\threadex.c @ 348]
00000000`183cfd30 000007fa`5444298e msvcr80!_threadstartex+0x84 [f:\dd\vctools\crt_bld\self_64_amd64\crt\src\threadex.c @ 326]
00000000`183cfd60 000007fa`5660e229 kernel32!BaseThreadInitThunk+0x1a
00000000`183cfd90 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

You can see where the thread started and all of the calls that were made until we got to the Wait.  The Top is the most recent call, so it reads from the bottom and work up.  Notice also that the msvcr80 calls have source code information.  One thing I do to try and clean up stack calls is to use the upper case L with the k command to omit that. 

0:054> kL
Child-SP          RetAddr           Call Site
00000000`183cf118 000007fa`5376db54 ntdll!NtSignalAndWaitForSingleObject+0xa
00000000`183cf120 00000000`00ae4b08 KERNELBASE!SignalObjectAndWait+0xc8
00000000`183cf1d0 00000000`00ae2d76 sqlservr!SOS_Scheduler::Switch+0x181
00000000`183cf920 00000000`00ae2700 sqlservr!SOS_Scheduler::SuspendNonPreemptive+0xca
00000000`183cf960 00000000`00aeae8d sqlservr!SOS_Scheduler::Suspend+0x2d
00000000`183cf990 00000000`00aeaf16 sqlservr!WorkDispatcher::DequeueTask+0x472
00000000`183cfa20 00000000`00c244fa sqlservr!SOS_Scheduler::ProcessTasks+0x76
00000000`183cfa90 00000000`00c247dd sqlservr!SchedulerManager::WorkerEntryPoint+0x2d2
00000000`183cfb70 00000000`0106c0cd sqlservr!SystemThread::RunWorker+0xcc
00000000`183cfbb0 00000000`00c253d2 sqlservr!SystemThreadDispatcher::ProcessWorker+0x2db
00000000`183cfc60 00000000`77c337d7 sqlservr!SchedulerManager::ThreadEntryPoint+0x173
00000000`183cfd00 00000000`77c33894 msvcr80!_callthreadstartex+0x17
<—no source info with the kL command
00000000`183cfd30 000007fa`5444298e msvcr80!_threadstartex+0x84
00000000`183cfd60 000007fa`5660e229 kernel32!BaseThreadInitThunk+0x1a
00000000`183cfd90 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

db, dd, dq, da, dc and du

These commands all display data based on a given address or range. 

db = Display byte values and ASCII characters

dd = Display DWORD

dq = Display QWORD

da = Display ASCII characters

dc = Display DWORD and ASCII characters

du = Display Unicode characters

dt

This lets you dump a type (or structure) so you can see what it looks like.  This really leads into a larger discussion about how to debug in general and understand and working with structures.  In our example above, with the k command, SOS_Scheduler would be an example of a type or structure.  However, you need full, or private, symbols to look at type information.  It is not available with private symbols.  This can be helpful when debugging your own applications though.

!uniqstack

I love this command!  It does two things for me.  First, it can weed out the noise by only showing the unique stacks within the dump.  So, if a given stack is repeated on 40 threads, it only shows it once.  Also, I use it as a warm up to make sure I have all the Symbols I need downloaded because it goes through and processes the call stacks.

Debugging Managed Code

Debugging Managed Code can sometimes be much easier to do than native code.  Mainly due to the SOS debugging extension and what it provides.  On important point to make about Managed Debugging.  You really need a Full dump to take advantage of the SOS extension.  I’ve always kind of joked about Managed Debugging that you can just !do everything and you’ll get there eventually. Although, sometimes that’s not far from the truth.

!do (Dump Object)

!do is one of the main managed commands from SOS.

0:005> !do 02f487d4
Name: System.Data.SqlClient.SqlConnection
MethodTable: 6b66d824
EEClass: 6b58c404
Size: 56(0x38) bytes
(C:\Windows\assembly\GAC_32\System.Data\2.0.0.0__b77a5c561934e089\System.Data.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
70b50770  400018a        4        System.Object  0 instance 00000000 __identity
702d5700  40008de        8 ...ponentModel.ISite  0 instance 00000000 site
702ef9f0  40008df        c ....EventHandlerList  0 instance 00000000 events
70b50770  40008dd      108        System.Object  0   static 02cc8ea8 EventDisposed
6ba89940  4000bd6       10 ...hangeEventHandler  0 instance 00000000 _stateChangeEventHandler
6baaef30  400171d       14 ...t.SqlDebugContext  0 instance 00000000 _sdc
70b24620  400171e       30       System.Boolean  1 instance        0 _AsycCommandInProgress
6b670f4c  400171f       18 ...ent.SqlStatistics  0 instance 00000000 _statistics
70b24620  4001720       31       System.Boolean  1 instance        0 _collectstats
70b24620  4001721       32       System.Boolean  1 instance        0 _fireInfoMessageEventOnUserErrors
6b66f4d8  4001724       1c ...ConnectionOptions  0 instance 02f5a1b4 _userConnectionOptions
6b66f018  4001725       20 ...nnectionPoolGroup  0 instance 02f5a784 _poolGroup
6b66f614  4001726       24 ...onnectionInternal  0 instance 02f5b08c _innerConnection
70b52da0  4001727       28         System.Int32  1 instance        0 _closeCount
70b52da0  4001729       2c         System.Int32  1 instance        1 ObjectID
70b50770  400171c      798        System.Object  0   static 02f4880c EventInfoMessage
6b66ee84  4001722      79c ...ConnectionFactory  0   static 02f48818 _connectionFactory
70b5291c  4001723      7a0 ...eAccessPermission  0   static 02f4bf3c ExecutePermission
70b52da0  4001728      880         System.Int32  1   static        5 _objectTypeCount

You can then do another !do and get the connection string.

0:005> !do 02f5a1b4
Name: System.Data.SqlClient.SqlConnectionString
MethodTable: 6b66f48c
EEClass: 6b5a02dc
Size: 112(0x70) bytes
(C:\Windows\assembly\GAC_32\System.Data\2.0.0.0__b77a5c561934e089\System.Data.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
70b50b54  4000be0        4        System.String  0 instance 02f59f10 _usersConnectionString
70b531a8  4000be1        8 ...ections.Hashtable  0 instance 02f5a420 _parsetable
6b66f6e0  4000be2        c ...mon.NameValuePair  0 instance 02f5a50c KeyChain
70b24620  4000be3       14       System.Boolean  1 instance        0 HasPasswordKeyword
70b24620  4000be4       15       System.Boolean  1 instance        0 UseOdbcRules
70b52818  4000be5       10 ...ity.PermissionSet  0 instance 02f5a8f4 _permissionset
702e419c  4000bdc      1f0 ...Expressions.Regex  0   static 02f4cc84 ConnectionStringValidKeyRegex
702e419c  4000bdd      1f4 ...Expressions.Regex  0   static 02f54970 ConnectionStringValidValueRegex
702e419c  4000bde      1f8 ...Expressions.Regex  0   static 02f56310 ConnectionStringQuoteValueRegex
702e419c  4000bdf      1fc ...Expressions.Regex  0   static 02f57ce8 ConnectionStringQuoteOdbcValueRegex
70b24620  4001760       16       System.Boolean  1 instance        1 _integratedSecurity
70b24620  4001761       17       System.Boolean  1 instance        0 _async
70b24620  4001762       60       System.Boolean  1 instance        1 _connectionReset
70b24620  4001763       61       System.Boolean  1 instance        0 _contextConnection

You can see how you can start going around just !do’ing items to get what you want.  To pull in some of the display commands above, here is a cool trick.  If we look above at the _usersConnectionString, we can see that it is a System.String.  We could do a !do on that and get the following:

0:005> !do 02f59f10
Name: System.String
MethodTable: 70b50b54
EEClass: 7090d65c
Size: 530(0x212) bytes
(C:\Windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
String: server='localhost';Trusted_Connection=true;Application Name='Microsoft SQL Server Management Studio';Pooling=false;Packet Size=4096;multipleactiveresultsets=false
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
70b52da0  4000096        4         System.Int32  1 instance      257 m_arrayLength
70b52da0  4000097        8         System.Int32  1 instance      162 m_stringLength

You could also do a dc and see the text as well…  although not as cleanly, but is there.

0:005> dc 02f59f10
02f59f10  70b50b54 00000101 000000a2 00650073  T..p........s.e.
02f59f20  00760072 00720065 0027003d 006f006c  r.v.e.r.=.'.l.o.
02f59f30  00610063 0068006c 0073006f 00270074  c.a.l.h.o.s.t.'.
02f59f40  0054003b 00750072 00740073 00640065  ;.T.r.u.s.t.e.d.
02f59f50  0043005f 006e006f 0065006e 00740063  _.C.o.n.n.e.c.t.
02f59f60  006f0069 003d006e 00720074 00650075  i.o.n.=.t.r.u.e.
02f59f70  0041003b 00700070 0069006c 00610063  ;.A.p.p.l.i.c.a.
02f59f80  00690074 006e006f 004e0020 006d0061  t.i.o.n. .N.a.m.

That is because it is in memory, and these are just different options of looking at a given memory location.  The string is there though.  You see the periods in between the characters because it is actually a Unicode string and each character takes two bytes instead of one.  If you recall, we have a command for dumping Unicode – du.

0:005> du 02f59f10+c
02f59f1c  "server='localhost';Trusted_Conne"
02f59f5c  "ction=true;Application Name='Mic"
02f59f9c  "rosoft SQL Server Management Stu"
02f59fdc  "dio';Pooling=false;Packet Size=4"
02f5a01c  "096;multipleactiveresultsets=fal"
02f5a05c  "se"

You’ll notice the +c.  Had we just gone with the base address we would see the following with the du command.

0:005> du 02f59f10
02f59f10  ".炵ā"

That’s because the string doesn’t actually start at the base address.  There is some information regarding to the structure of System.String.  It starts a little bit further in – 12 bytes to be exact which is C in hex.  That’s where the +c comes from. We are saying grab me the unicode string at the base address+C which we are actually adding 12 bytes to it.  So, it would be the same if we did the following.

0:005> du 02f59f1c
02f59f1c  "server='localhost';Trusted_Conne"
02f59f5c  "ction=true;Application Name='Mic"
02f59f9c  "rosoft SQL Server Management Stu"
02f59fdc  "dio';Pooling=false;Packet Size=4"
02f5a01c  "096;multipleactiveresultsets=fal"
02f5a05c  "se"

You can being to see why using the SOS Extension makes things easier.

!dso (Dump Stack Objects)

This will list the managed objects for the given thread that you are currently on.  Assuming it is a managed thread.  This can be extremely helpful.  It has also helped me to figure out where we are within a given code path based on what objects were present.

0:005> !dso
OS Thread Id: 0xd74 (5)
ESP/REG  Object   Name
06b1f12c 0315e538 System.String    ErrorLogFile
06b1f144 03148088 Microsoft.Win32.SafeHandles.SafeWaitHandle
06b1f18c 0314805c Microsoft.SqlServer.Management.UI.VSIntegration.ObjectExplorer.Service+AsyncWmiBinding
06b1f624 02e94664 System.Runtime.CompilerServices.RuntimeHelpers+TryCode
06b1f628 02e94684 System.Runtime.CompilerServices.RuntimeHelpers+CleanupCode
06b1f62c 03146098 System.Threading.ExecutionContext+ExecutionContextRunData
06b1f668 03148238 System.Threading._ThreadPoolWaitCallback
06b1f680 03148238 System.Threading._ThreadPoolWaitCallback

You can also look at this post for another example, and this one.

!clrstack

This will dump out the Managed Stack for a given thread that you are on.  This is essentially the k command for managed code.

0:005> !clrstack
OS Thread Id: 0xd74 (5)
ESP       EIP    
06b1f0c0 77ce9bd5 [HelperMethodFrame_1OBJ: 06b1f0c0] System.Threading.WaitHandle.WaitOneNative(Microsoft.Win32.SafeHandles.SafeWaitHandle, UInt32, Boolean, Boolean)
06b1f16c 70ad689f System.Threading.WaitHandle.WaitOne(Int64, Boolean)
06b1f188 70ad6855 System.Threading.WaitHandle.WaitOne(Int32, Boolean)
06b1f19c 70ad681d System.Threading.WaitHandle.WaitOne()
06b1f1a4 6f107c90 Microsoft.SqlServer.Management.UI.VSIntegration.ObjectExplorer.Service+AsyncWmiBinding.RequestHandler(System.Object)
06b1f1b0 70ae9f7f System.Threading._ThreadPoolWaitCallback.WaitCallback_Context(System.Object)
06b1f1b8 70b157b1 System.Threading.ExecutionContext.runTryCode(System.Object)
06b1f5e8 71ee1b4c [HelperMethodFrame_PROTECTOBJ: 06b1f5e8] System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode, CleanupCode, System.Object)
06b1f650 70b156a7 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
06b1f66c 70b002f5 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
06b1f684 70aea4e3 System.Threading._ThreadPoolWaitCallback.PerformWaitCallbackInternal(System.Threading._ThreadPoolWaitCallback)
06b1f698 70aea379 System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(System.Object)
06b1f828 71ee1b4c [GCFrame: 06b1f828]

 

For learning more about Managed Debugging, I’ll point you to Tess’ blog.  That’s pretty much how I learned Managed debugging.  Great stuff over there!

Your Assignment

When I went through this internally, someone at the end asked if there was a Homework assignment.  I hadn’t thought about that, but I replied with the following:

1. Install a debugger

2. Capture a dump of Notepad

3. Make note of the number of threads you have

4. Output the stack of a thread

To help you along, you can use the following command to capture the dump from WinDBG (although you don’t have to use WinDBG to get the dump). after having attached to the Notepad Process:

.dump /ma c:\temp\mydump.dmp

This will capture a full dump.  You can see the command switches here:

.dump (Create Dump File)
http://msdn.microsoft.com/en-us/library/windows/hardware/ff562428(v=vs.85).aspx

Bonus points if you capture the dump using a different method.

 

Happy hunting!

Adam W. Saxton | Microsoft Escalation Services
http://twitter.com/awsaxton