I wrote this largely to get the commands right in my own head – as a result, this is more a list of commands and some semblance of a sequence you can run them in. If you’ve never used these tools before this will get you out of the gate on WinDbg and the debugger extension SOS.dll.

If you’re looking for scenario's and walkthroughs I can’t stress enough that Tess Ferrandez is the go-to-guy for this and does a much better job; including a superb hands-on lab and crash walkthroughs on her Blog. All the commands I've listed are also documented here

At the end I’ve put three quick sequences together – looking at an web site, reviewing an exception and getting the IL for  a particular method.

Symbols

Before beginning confirm you’ve setup your symbols and know where sos.dll is:
1) setup debug symbol and path – the easiest way is to configure a local store (for example c:\symbols) and use the public symbol server – create an environment variable _NT_SYMBOL_PATH and set it to SRV*c:\symbols*http://msdl.microsoft.com/download/symbols. If you have symbols in multiple places, add these locations separated by ‘;’. Note, if you’re using windows 7, windows symbols for your version are already installed to c:\windows\symbols.

If you set your symbol path to use the public server, or a symbols server, anything that requires you to resolve function names will cause the symbols to be downloaded – so some functions will take a few seconds to run, until you’ve built up your cache. WinDBG will set its status to busy while this happens.

2) If you’re debugging a 32 bit process or memory dump, copy the sos.dll from C:\Windows\Microsoft.NET\Framework\v2.0.50727 to the local directory for your windbg install (typically c:\program files[(x64)]\Debugging Tools for Windows). For 64 bit – use the framework64 version. It isn’t mandatory to copy the file here but makes .load a little quicker to run.

Attach to a running process

Pretty straight forwards – open windbg, hit F6, choose the process to connect to. You’ll be asked if you want to save the workspace – select no. Assuming we attached successfully, type ‘g’ – the debugger command for go. To break into the process again, hit ctrl-break. If all you have is a dump, open the dump from the file menu (open Crash Dump) – all of the following will work if a full user dump was taken (rather than a mini dump).

While the process is running, if any exceptions are being caught in the application, you’ll get ‘CLR exception - code e0434f4d (first chance)’ appearing in the console. See ‘Breaking When an Exception is thrown’ if you want to break on these.

If you get an error, typically ‘request not supported’. check you’re using the right ‘bitness’ of debugger for the process you’re connecting to or the right chip architecture for WinDbg (eg, IA64, x64, x86). You may also have to run windbg elevated.

Load SOS.Dll

Ctrl-Break into the process and load the SOS.dll by typing:

.Load sos.

If you didn’t copy sos into the same directory as windbg you’ll have to specify the full path. Note the command is prefixed with a ‘.’.  You’ll get a message if sos can’t be found. If you get a x08004005 error when calling any of the following commands, make sure you’ve copied the right version (either x64 or x86) into your WinDbg directory.

Two other commands that will be of use here:

.sympath <path to symbols> // use to set or get the path to symbols.
.reload // force a reload of any symbols

If the symbols are wrong often you’ll get a warning telling you to check them, and you’ll also get hex instead of the missing function names when resolving call stacks.

Help!

SOS – it’s a debugger extension, so it’s commands are prefixed by ‘!’. Get started by calling:

!Help.

If you get the message ‘no export help found’ then sos hasn’t been loaded – just try and reload it using .reload sos.

What’s currently running?

When you break into the process you’ll end up sitting on a specific thread, which is identified by the number to the left of the command prompt. You can get the native call stack for the thread you’re currently on using the WinDbg command:

kb

Any hex in the output where you can see names and check you’ve got the right symbols. You can get the managed call stack for the current thread using an SOS extension:

!CLRStack

if there’s no managed call stack, well, you’ll get an error telling you this. Otherwise you’ll get the managed stack for the current thread. Use –p and –l to return parameter and local variable information. Use –a to return both.

Get a list of the managed threads in the process using:

!Threads

From this you can get a list of ThreadIds (the column on the far left is the id to use with the next commands) and whether or not they’re gc enabled (true means managed code, false means unsafe code), how many locks are open for that thread, it’s apartment model and finally any exceptions. This last column will also indicate which one of the threads is the Finalizer. If a thread holds an exception, you’ll find the address at the far right. In all fairness, it’s a good place to start any exploration.

To jump onto a different thread use the ‘~’ to identify you’re issuing a thread command, then the command. For example:

~<<thread id>> s // switch context to the thread id, eg ~27 s // switch to thread id 27
~* e kb // for all threads in the process, execute kb (dump the stack)
~* e !CLRStack // you probably get the idea, but for completeness: for all threads, evaluate !clrstack

WinDbg allows to to shell out – you might be interested in parsing some of the output using the windows Batch command - FIND. The following evaluates all threads to see what has the given function on their stack:

.shell -i - -ci "~* e !clrstack" FIND “<<FunctionName>>” for example .shell -i - -ci "~* e !clrstack" FIND “DoStuff”

Note the syntax above is sensitive, the dash after ‘–i’ is not a typo.

Finally – you can get all the objects associated with a set of calls:

!DumpStackObjects

this will, er, Dump all the objects within the bounds of the current stack. If, for example you’ve got a function calling out to Sql – this will help you get the Command object associated with that call stack if you’ve switched to that thread.

Allocations

Get the allocations for this program using:

!dumpheap –stat

This will dump the types of objects and their sizes on the heap. The stat switch says give me the summary. The 4 columns in the output are Method Table, number of objects, total size and type. Typically you’ll see System.String and System.Object high on this list. This will also tell you which generation each object is currently in.

To get a look at any of these objects, for example, System.String we can ask for the address of all objects of a given type over a fixed size. For example:

!dumpheap –type System.String –min 200

This will return a table of results detailing the address, method table and size of the object. Following this is a second table with Method Table Count of Objects, Total Size and the class name. Getting the address of an object from the previous command means you can start to examine it and it’s fields. If that object is a string, frequently you’ll get xml or other goodies buried within the process.

!dumpobj <<address>>

Often a command might just return a pointer to that object, for example the following command which looks for handles with no parents (an increasing number of these may indicate a leak).

!gchandleleaks

You can resolve the address of the object using the WinDbg macro ‘poi’ (pointer of int):

!dumpobject poi(<<handle>>)

Once you’ve got the address of an object you can resolve what else might be referencing that object using:

!gcroot <<address of object>>

Here you’re looking either for explicit object references, or if objects are rooted in the domain (statics). It might also be useful to understand what the allocation graph is – call:

!traverseheap <<outputfile.log>>

This will produce a log file output that can be presented in the clr profiler available here and is very useful for evaluating object graphs, and viewing how objects are laid out in each heap. be warned though – even for process of a couple of 100Mb this can often take over an hour.

Some Commands You’re Gonna love

!aspxpages is one of a great place to start if you’re looking at a web process. It checks the heap for any existing HttpContext objects and attempts to resolve which thread either executed or is executing it, whether it’s still currently running and if the request completed or not, and how long it took and what the return code was – it’s a very quick indicator of health. Occasionally it will report that a thread is no longer running (it’s Id will be XXX) – this may not be the case – so have  a look at !threads to see if there is a suspect that may be running that request.

!dumpdatatables - returns a table of addresses of objects that are data tables, nextRowID (rowcount) and number of columns. No of Cells = nextRowId * RowCount.

!dumpobject {address of columns} will return the column names allowing you to establish how much data is in datatables.

Is there a large amount of concurrency in our process?

!syncblk

A high number under ‘monitor held’ may indicate a contention problem.

Note that the number of monitors held =  1 for the owner + 2*waiting number of waiting threads.

Check the help file, but .Net 2.0 uses thin locks, so you might also want to run i syncblk returns nothing.

!DumpHeap -thinlock

Breaking  when an Exception is Thrown

There are two ways to do this – the easiest is to use the debugger break into the process, select Debug –> Event Filters –> Enable CLR Exception. When an exception is thrown the debugger will break into the process and sit you on the thread that has the Exception. Call !Threads to which will return the type of exception and it’s address, you can then call:

!PrintException <<address>>

The second way is to use !StopOnException – see the help for this. I might come back an update this.

Possible Sequence of Events for troubleshooting a hanging web app

!aspxpages – look for requests that didn’t complete – find their thread id.
~<<threadid>> s – switch to that thread.
!clrstack get the command at the top of the stack
!DumpStackObjects – get the objects associated with this call stack.

Possible sequence of commands for getting the exception text:

Set filters to enable CLR Exception
g (go)
!threads to get the exception address and type
!PrintException <<address>> to get the exception text and any inner exception.

Get the IL for a method on the stack

!dumpstack – lists the return address for each call. Take the return address the line above the function you’re interested in – for example, I’d like to see DoStuff() so take the return address 000007ff00170223:

Child-SP         RetAddr          Call Site
000000000016e4f8 000007fefdc71203 ntdll!NtDelayExecution+0xa
000000000016e500 000007fef8de176d KERNELBASE!SleepEx+0xb3
000000000016e5a0 000007fef89a77b5 mscorwks!DllCanUnloadNowInternal+0xf37d
000000000016e620 000007fef8f8f2f9 mscorwks!CreateApplicationContext+0x391
000000000016e680 000007ff00170223 mscorwks!ReOpenMetaDataWithMemory+0x1ff59
000000000016e830 000007ff00170195 ExceptionalApplication!ExceptionalApplication.Program.DoStuff()+0x53
000000000016e880 000007fef8b5d502 ExceptionalApplication!ExceptionalApplication.Program.Main(System.String[])+0x25

We want the method descriptor, so call

!IP2MD 000007ff00170223

which returns the Method Descriptor:

0:000> !ip2md 000007ff00170223
MethodDesc: 000007ff000239b8
Method Name: ExceptionalApplication.Program.DoStuff()
Class: 000007ff00162230
MethodTable: 000007ff00023a48
mdToken: 06000002
Module: 000007ff000233d0
IsJitted: yes
CodeAddr: 000007ff001701d0

Now you can call !dumpil <<MethodDescriptor>> which get’s you the IL:

!dumpil 000007ff000239b8

That’s it for this post – I’d definitely encourage you to look at Tess’s blog and DumpAnalysis.Org for more.

Enjoy!

Originally posted by Ryan Simpson on 8 October 2009 here.