Kristoffer's tidbits : Windbg

Debugging memory usage in managed code using Windbg

Kristoffer — Tue, 09 Jan 2007 20:08:00 GMT

Windbg, is there anything it can't do?

CLR Profiler is great for getting an overview of memory allocations and usage for managed applications but it doesn't work for very large applications and can't be attached after the application has been running for a while to determine why after 12 hours memory use spikes. Windbg can be an acceptable alternative when you find yourself looking at an unexpected memory spike and you're curious where all that memory is being allocated.

Start Windbg, attach to the process, and load the sos dll (see previous post).

We start out by getting a summary of our memory allocations with !dumpheap

0:004> !dumpheap -stat total 8952 objects Statistics:
       MT    Count    TotalSize Class Name 
7b4779b8        1           12 System.Windows.Forms.OSFeature 
7b475ca8        1           12 System.Windows.Forms.FormCollection 
7b474f8c        1           12 System.Windows.Forms.Layout.DefaultLayout 
7b4749e0        1           12 System.Windows.Forms.ClientUtils+WeakRefCollection 
79128f18        1           12 System.Collections.Generic.GenericEqualityComparer`1[[System.Int16, mscorlib]] 
79128b94        1           12 System.Collections.Generic.ObjectEqualityComparer`1[[System.IntPtr, mscorlib]] 
79127ae4        1           12 System.Collections.Generic.ObjectEqualityComparer`1[[System.Object, mscorlib]] 
79114408        1           12 System.Security.Permissions.ReflectionPermission 
791142e8        1           12 System.Security.Permissions.FileDialogPermission 
----------------- snip ---------------------- 
790fd4ec       82         1968 System.Version 
7ae76b24      208         2496 System.Drawing.KnownColor 
791242ec       20         2952 System.Collections.Hashtable+bucket[] 
79114bf0      181         5068 System.Security.SecurityElement 
791036b0      229         5496 System.Collections.ArrayList 
00155a48       50        23424      Free 
79124228      306        47780 System.Object[] 
790fa3e0     6662       379600 System.String 
79124418        9      3156304 System.Byte[] Total 8952 objects

We've got about 3 megs worth of System.Byte[] allocated so let's focus in on this to see why these objects exist.

0:004> !dumpheap -type System.Byte[] 
 Address       MT     Size 
01241e5c 79124418       12      
0124aaf4 79124418       28      
0124ab48 79124418      140      
01251178 79124418      172      
0125129c 79124418       28      
01254f7c 79124418    10148      
02246da8 79124418  1048592      
02346db8 79124418  1048592      
02446dc8 79124418  1048592      
total 9 objects 
Statistics:       MT    Count    TotalSize Class Name 
79124418        9      3156304 System.Byte[] 
Total 9 objects

A few small Byte arrays but 3 of them are a meg each so let's find out why the first object is still alive (or if it is alive at all, a GC might not have happened to collect it yet).

0:004> !gcroot 02246da8 
Note: Roots found on stacks may be false positives. Run "!help gcroot" for more info. 
ebx:Root:012525d8(System.Windows.Forms.Application+ThreadContext)-> 
01251d58(WindbgDemo.Form1)-> 
01251edc(System.Collections.ArrayList)-> 
0125c278(System.Object[])-> 
02246da8(System.Byte[]) 
Scan Thread 0 OSTHread 1194 
Scan Thread 2 OSTHread 1720

The application has a reference to Form1 which has a reference to an ArrayList which has a reference to my Byte array. If a GC were to happen right now this object would be kept alive.

Instead of checking each Byte array by address individually we can use the .foreach token.

0:004> .foreach (obj {!dumpheap -type System.Byte[] -short}) {.echo obj;!gcroot obj} 
01241e5c 
Note: Roots found on stacks may be false positives. Run "!help gcroot" for more info. 
Scan Thread 0 OSTHread 1194 
Scan Thread 2 OSTHread 1720 
DOMAIN(00149768):HANDLE(Pinned):3e13fc:Root:02241010(System.Object[])-> 
01241e5c(System.Byte[]) 
0124aaf4 
Note: Roots found on stacks may be false positives. Run "!help gcroot" for more info. 
Scan Thread 0 OSTHread 1194 
Scan Thread 2 OSTHread 1720 
DOMAIN(00149768):HANDLE(Pinned):3e13fc:Root:02241010(System.Object[])-> 
0124a1c4(System.Environment+ResourceHelper)-> 
0124a358(System.Resources.ResourceManager)-> 
0124a3a0(System.Collections.Hashtable)-> 
0124a3d8(System.Collections.Hashtable+bucket[])-> 
0124a9f8(System.Resources.RuntimeResourceSet)-> 
0124aa5c(System.Resources.ResourceReader)-> 
0124aaac(System.IO.BinaryReader)-> 
0124aaf4(System.Byte[]) 
0124ab48 
Note: Roots found on stacks may be false positives. Run "!help gcroot" for more info. 
Scan Thread 0 OSTHread 1194 
Scan Thread 2 OSTHread 1720 
DOMAIN(00149768):HANDLE(Pinned):3e13fc:Root:02241010(System.Object[])-> 
0124a1c4(System.Environment+ResourceHelper)-> 
0124a358(System.Resources.ResourceManager)-> 
0124a3a0(System.Collections.Hashtable)-> 
0124a3d8(System.Collections.Hashtable+bucket[])-> 
0124a9f8(System.Resources.RuntimeResourceSet)-> 
0124aa5c(System.Resources.ResourceReader)-> 
0124aaac(System.IO.BinaryReader)-> 
0124ab48(System.Byte[]) 
--------------- remainder of output snipped --------------

By using the /ps parameter to .foreach (see Windbg help file for details) you can run gcroot on every nth item to get a sample of objects.

Debugging lock issues in managed code using Windbg

Kristoffer — Fri, 05 Jan 2007 19:35:00 GMT

Windbg can be helpful for locking lock issues in managed code that seem to only happen on client machines when the moon is full and the stars are aligned just right. Windbg is a minimal installation and with managed code doesn't require debug symbols to still be useful.

For my test scenario I have an application that hangs for 10 seconds at a time once a button is pressed so I start up windbg and break during the hang so I can inspect what's going on. First we check to see what the managed threads are currently running what:

0:000> ~*e!clrstack 
OS Thread Id: 0x123c (0) 
ESP       EIP      
0012eec8 7c82ed54 [GCFrame: 0012eec8]  0012ef98 7c82ed54 [HelperMethodFrame_1OBJ: 0012ef98] 
System.Threading.Monitor.Enter(System.Object) 
0012eff0 00cc0381 TestApp.Form1.button1_Click(System.Object, System.EventArgs) 
0012f02c 7b060a6b System.Windows.Forms.Control.OnClick(System.EventArgs) 
0012f03c 7b105379 System.Windows.Forms.Button.OnClick(System.EventArgs) 
0012f048 7b10547f System.Windows.Forms.Button.OnMouseUp(System.Windows.Forms.MouseEventArgs) 
0012f06c 7b0d02d2 System.Windows.Forms.Control.WmMouseUp(System.Windows.Forms.Message ByRef, System.Windows.Forms.MouseButtons, Int32) 
0012f0b8 7b072c74 System.Windows.Forms.Control.WndProc(System.Windows.Forms.Message ByRef) 
0012f0bc 7b0815a6 [InlinedCallFrame: 0012f0bc]  0012f158 7b0814c3 System.Windows.Forms.Button.WndProc(System.Windows.Forms.Message ByRef) 
0012f160 7b07a72d System.Windows.Forms.Control+ControlNativeWindow.OnMessage(System.Windows.Forms.Message ByRef) 
0012f164 7b07a706 System.Windows.Forms.Control+ControlNativeWindow.WndProc(System.Windows.Forms.Message ByRef) 
0012f178 7b07a515 System.Windows.Forms.NativeWindow.Callback(IntPtr, Int32, IntPtr, IntPtr) 
0012f324 0033216c [NDirectMethodFrameStandalone: 0012f324] System.Windows.Forms.UnsafeNativeMethods.DispatchMessageW(MSG ByRef) 
0012f334 7b084766 System.Windows.Forms.Application+ComponentManager.System.Windows.Forms.UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(Int32, Int32, Int32) 
0012f3d4 7b08432d System.Windows.Forms.Application+ThreadContext.RunMessageLoopInner(Int32, System.Windows.Forms.ApplicationContext) 
0012f440 7b08416b System.Windows.Forms.Application+ThreadContext.RunMessageLoop(Int32, System.Windows.Forms.ApplicationContext) 
0012f470 7b0c69fe System.Windows.Forms.Application.Run(System.Windows.Forms.Form) 
0012f480 00cc00a8 TestApp.Program.Main() 
0012f69c 79e88f63 [GCFrame: 0012f69c]  
OS Thread Id: 0x1408 (1) 
Unable to walk the managed stack. The current thread is likely not a  
managed thread. You can run !threads to get a list of managed threads in 
the process 
OS Thread Id: 0xb30 (2) 
Failed to start stack walk: 80004005 
OS Thread Id: 0x161c (3) 
Unable to walk the managed stack. The current thread is likely not a  
managed thread. You can run !threads to get a list of managed threads in 
the process 
OS Thread Id: 0x149c (4) 
ESP       EIP      
011ef82c 7c82ed54 [HelperMethodFrame: 011ef82c] System.Threading.Thread.SleepInternal(Int32) 
011ef880 793d80f5 System.Threading.Thread.Sleep(Int32) 
011ef884 00cc042d TestApp.Form1.ThreadProc() 
011ef8b4 793d7a7b System.Threading.ThreadHelper.ThreadStart_Context(System.Object) 
011ef8bc 793683dd System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) 
011ef8d4 793d7b5c System.Threading.ThreadHelper.ThreadStart() 
011efaf8 79e88f63 [GCFrame: 011efaf8]  
OS Thread Id: 0x7e0 (5) 
Unable to walk the managed stack. The current thread is likely not a  
managed thread. You can run !threads to get a list of managed threads in 
the process 
OS Thread Id: 0xf94 (6) 
Unable to walk the managed stack. The current thread is likely not a 
managed thread. You can run !threads to get a list of managed threads in 
the process 
OS Thread Id: 0x5e4 (7) 
Unable to walk the managed stack. The current thread is likely not a 
managed thread. You can run !threads to get a list of managed threads in 
the process

So we've got two managed threads: 0 (the message pumping thread) and 4. Thread 0 is currently in a call to System.Threading.Monitor.Enter which the lock statement wraps. Let's see what locks are held by the application.

0:000> !syncblk 
Index SyncBlock MonitorHeld Recursion Owning Thread Info  SyncBlock Owner    
11 001b415c            3         1 001be180  149c   4   012cfdfc TestApp.Form1 
----------------------------- 
Total           11 
CCW             0 
RCW             0 
ComClassFactory 0 
Free            0

So thread 4 (which is currently sleeping) holds a lock on the object at address 012cfdfc. Let's see what type of object this is:

0:000> !DumpObj -nofields 012cfdfc 
Name: TestApp.Form1 
MethodTable: 008f5ac4 
EEClass: 008f165c 
Size: 332(0x14c) bytes 
 (C:\Testing\WindowsApplication17\WindowsApplication17\bin\Debug\WindowsApplication17.exe)

Thread 4 has a lock on the Form. Looking at the stack objects for thread 0 we can see the Form object right before a synchronization context.

0:000> !dso 
OS Thread Id: 0x123c (0) 
ESP/REG  Object   Name 
0012edd0 012d05f4 System.Windows.Forms.WindowsFormsSynchronizationContext 
0012ef00 012cfdfc TestApp.Form1 
0012ef54 012f9f1c System.Threading.ThreadStart 
0012ef58 012f9f3c System.Threading.Thread 
0012ef64 012f9f1c System.Threading.ThreadStart 
0012ef6c 012f9f3c System.Threading.Thread 
0012efac 012cff60 Microsoft.Win32.SafeHandles.SafeWaitHandle 
0012efd4 012cfdfc TestApp.Form1 
0012eff0 012cfdfc TestApp.Form1 
0012eff4 012f9f3c System.Threading.Thread 
0012eff8 012cfdfc TestApp.Form1 
0012effc 012ea688 System.Windows.Forms.Button 
-------------- snip -----------------------
 0012f46c 012ec1e0 System.Windows.Forms.ApplicationContext 
0012f474 012cfdfc TestApp.Form1

We can confirm this by looking at the IL code for TestApp.Form1.button1_Click.

0:000> !dumpmt -md 008f5ac4 
EEClass: 008f165c 
Module: 008f2d5c 
Name: TestApp.Form1 
mdToken: 02000003 
 (C:\Testing\WindowsApplication17\WindowsApplication17\bin\Debug\WindowsApplication17.exe)
BaseSize: 0x14c 
ComponentSize: 0x0 
Number of IFaces in IFaceMap: 15 
Slots in VTable: 376 
-------------------------------------- 
MethodDesc Table 
   Entry MethodDesc      JIT Name 
7b05c298   7b4a5518   PreJIT System.Windows.Forms.Form.ToString() 
793539c0   7913bd50   PreJIT System.Object.Equals(System.Object) 
793539b0   7913bd68   PreJIT System.Object.GetHashCode() 
7a4a6510   7a75bf58   PreJIT System.ComponentModel.Component.Finalize() 
79361e50   7913dbd8   PreJIT System.MarshalByRefObject.GetLifetimeService() 
7b0662a4   7b4a5530   PreJIT System.Windows.Forms.Form.OnResizeEnd(System.EventArgs) 
------------------ snip -----------------
008f61d4   008f5a68      JIT TestApp.Form1.InitializeComponent() 
008f61b0   008f5a70      JIT TestApp.Form1..ctor() 
008f6cfc   008f5a78      JIT TestApp.Form1.ThreadProc() 
008f6a54   008f5a80      JIT TestApp.Form1.button1_Click(System.Object, System.EventArgs) 
0:000> !dumpil 008f5a80 
ilAddr = 00402274 
IL_0000: nop 
IL_0001: ldarg.0 
IL_0002: ldftn TestApp.Form1::ThreadProc
IL_0008: newobj System.Threading.ThreadStart::.ctor 
IL_000d: newobj System.Threading.Thread::.ctor 
IL_0012: stloc.0 
IL_0013: ldloc.0 
IL_0014: callvirt System.Threading.Thread::Start 
IL_0019: nop 
IL_001a: ldarg.0 
IL_001b: ldfld TestApp.Form1::threadStarted
IL_0020: callvirt System.Threading.WaitHandle::WaitOne 
IL_0025: pop 
IL_0026: ldarg.0 
IL_0027: dup 
IL_0028: stloc.1 
IL_0029: call System.Threading.Monitor::Enter 
IL_002e: nop 
.try
{
  IL_002f: nop 
  IL_0030: nop 
  IL_0031: leave.s IL_003b
} // end .try
.finally
{
  IL_0033: ldloc.1 
  IL_0034: call System.Threading.Monitor::Exit 
  IL_0039: nop 
  IL_003a: endfinally 
} // end .finally
IL_003b: nop 
IL_003c: ldloc.0 
IL_003d: callvirt System.Threading.Thread::Join 
IL_0042: nop 
IL_0043: ret 

Sure enough Form1 is passing itself to System.Threading.Monitor.Enter.

Debugging exceptions in managed code using Windbg

Kristoffer — Wed, 03 Jan 2007 19:04:27 GMT

By default Windbg will break for access violations (or null reference exceptions as they're called in managed code). To get Windbg to break for managed exceptions use the sxe command.

0:004> sxe clr
0:004> g

A little while later our program generates a CLR exception and pops into the debugger.

(17c.1194): CLR exception - code e0434f4d (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
eax=0012eea0 ebx=0014d578 ecx=00000000 edx=00000025 esi=0012ef2c edi=e0434f4d
eip=77e55e02 esp=0012ee9c ebp=0012eef0 iopl=0         nv up ei pl nz na po nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00000202
KERNEL32!RaiseException+0x53:
77e55e02 5e              pop     esi

This is a first chance exception so this may be expected and handled, as the message helpfully reminds you. Let's find out what caused this exception.

0:000> !PrintException
Exception object: 01289fb0
Exception type: System.ArgumentException
Message: Value does not fall within the expected range.
InnerException: <none>
StackTrace (generated):
<none>
StackTraceString: <none>
HResult: 80070057

We can also have a look at the managed stack trace to see where this exception happened.

0:000> !CLRStack
OS Thread Id: 0x1194 (0)
ESP       EIP      
0012ef78 77e55e02 [HelperMethodFrame: 0012ef78]
0012f01c 00cc0795 WindbgDemo.Form1.CauseOtherException()
0012f024 00cc075d WindbgDemo.Form1.btnOtherException_Click(System.Object, System.EventArgs)
0012f02c 7b060a6b System.Windows.Forms.Control.OnClick(System.EventArgs)
0012f03c 7b105379 System.Windows.Forms.Button.OnClick(System.EventArgs) 
0012f048 7b10547f System.Windows.Forms.Button.OnMouseUp(System.Windows.Forms.MouseEventArgs)
0012f06c 7b0d02d2 System.Windows.Forms.Control.WmMouseUp(System.Windows.Forms.Message ByRef, System.Windows.Forms.MouseButtons, Int32)
0012f0b8 7b072c74 System.Windows.Forms.Control.WndProc(System.Windows.Forms.Message ByRef)
0012f0bc 7b0815a6 [InlinedCallFrame: 0012f0bc] 
0012f158 7b0814c3 System.Windows.Forms.Button.WndProc(System.Windows.Forms.Message ByRef)
0012f160 7b07a72d System.Windows.Forms.Control+ControlNativeWindow.OnMessage(System.Windows.Forms.Message ByRef)
0012f164 7b07a706 System.Windows.Forms.Control+ControlNativeWindow.WndProc(System.Windows.Forms.Message ByRef)
0012f178 7b07a515 System.Windows.Forms.NativeWindow.Callback(IntPtr, Int32, IntPtr, IntPtr)
0012f324 0033216c [NDirectMethodFrameStandalone: 0012f324] System.Windows.Forms.UnsafeNativeMethods.DispatchMessageW(MSG ByRef)
0012f334 7b084766 System.Windows.Forms.Application+ComponentManager.System.Windows.Forms.UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(Int32, Int32, Int32)
0012f3d4 7b08432d System.Windows.Forms.Application+ThreadContext.RunMessageLoopInner(Int32, System.Windows.Forms.ApplicationContext)
0012f440 7b08416b System.Windows.Forms.Application+ThreadContext.RunMessageLoop(Int32, System.Windows.Forms.ApplicationContext)
0012f470 7b0c69fe System.Windows.Forms.Application.Run(System.Windows.Forms.Form)
0012f480 00cc0097 WindbgDemo.Program.Main()
0012f69c 79e88f63 [GCFrame: 0012f69c]

If we want to get a feel for what sorts of exceptions are generated by our program but not necessarily inspect each one we can tell Windbg to stop on exceptions, print out some information about the exception, and then continue running all without user intervention.

0:000> sxe -c "!pe;!clrstack;gc" clr

Setting a breakpoint in managed code using Windbg

Kristoffer — Tue, 02 Jan 2007 19:49:06 GMT

One of the great features of managed code is getting call stacks and proper class and member function names without debug symbols. We can examine the methods of a class and set breakpoints based on name.

To see the methods on an object in Windbg we first need to find its method table. We can do this with the !name2ee command which will look up information about a class.

0:004> !Name2EE *!WindbgDemo.Form1
Module: 790c2000 (mscorlib.dll)
--------------------------------------
Module: 00912380 sortkey.nlp)
--------------------------------------
Module: 00912010 (sorttbls.nlp)
--------------------------------------
Module: 008f2d5c (WindbgDemo.exe)
Token: 0x02000004
MethodTable: 008f5b5c
EEClass: 008f15dc
Name: WindbgDemo.Form1
--------------------------------------
Module: 7b442000 (System.Windows.Forms.dll)
--------------------------------------
Module: 7a714000 (System.dll)
--------------------------------------
Module: 7ae72000 (System.Drawing.dll)

With the method table address we can examine all the methods defined on the object.

0:004> !DumpMT -MD 008f5b5c
EEClass: 008f15dc
Module: 008f2d5c
Name: WindbgDemo.Form1
mdToken: 02000004  (C:\Testing\WindbgDemo\WindbgDemo\bin\Release\WindbgDemo.exe)
BaseSize: 0x16c
ComponentSize: 0x0
Number of IFaces in IFaceMap: 15
Slots in VTable: 383
--------------------------------------
MethodDesc Table
   Entry MethodDesc      JIT Name
7b05c298   7b4a5518   PreJIT System.Windows.Forms.Form.ToString()
793539c0   7913bd50   PreJIT System.Object.Equals(System.Object)
793539b0   7913bd68   PreJIT System.Object.GetHashCode()
7a4a6510   7a75bf58   PreJIT System.ComponentModel.Component.Finalize()
79361e50   7913dbd8   PreJIT System.MarshalByRefObject.GetLifetimeService()
79360770   7913dbe0   PreJIT System.MarshalByRefObject.InitializeLifetimeService()
--------------- snip ----------------
7b06621c   7b4a5528   PreJIT System.Windows.Forms.Form.OnResizeBegin(System.EventArgs)
7b0662a4   7b4a5530   PreJIT System.Windows.Forms.Form.OnResizeEnd(System.EventArgs)
008f62a0   008f5ac8      JIT WindbgDemo.Form1.InitializeComponent()
008f627c   008f5ad0      JIT WindbgDemo.Form1..ctor()
008f62bc   008f5ad8     NONE WindbgDemo.Form1.btnLockBlock_Click(System.Object, System.EventArgs)
008f6249   008f5ae0     NONE WindbgDemo.Form1.LockBlock()
008f62d0   008f5ae8     NONE WindbgDemo.Form1.btnException_Click(System.Object, System.EventArgs)
008f6251   008f5af0     NONE WindbgDemo.Form1.CauseException()
008f630c   008f5af8     NONE WindbgDemo.Form1.btnOtherException_Click(System.Object, System.EventArgs)
008f6259   008f5b00     NONE WindbgDemo.Form1.CauseOtherException()
008f62e4   008f5b08     NONE WindbgDemo.Form1.btnMemoryGrowth_Click(System.Object, System.EventArgs)
008f62f8   008f5b10     NONE WindbgDemo.Form1.btnBreakpoint_Click(System.Object, System.EventArgs)
008f6265   008f5b18     NONE WindbgDemo.Form1.Foo()

For the curious we can see whether the function is ngen'ed code or if it has been JITted yet. PreJIT means ngen code, JIT and NONE is whether the function has been JITted or not. From here we can set a breakpoint.

0:004> !bpmd -md 008f5b10 
MethodDesc = 008f5b10
Adding pending breakpoints...

It's a pending breakpoint because this code has not been jitted yet and thus has no place to put the debug breakpoint. You can also set breakpoints directly by name using !bpmd.

Getting started with Windbg and managed code

Kristoffer — Sat, 30 Dec 2006 00:51:10 GMT

Windbg (or wind bag as my friend calls it) is my one stop shop for almost everything debugging related. It's great for finding issues that only happen on a client machine and you don't want a full blown Visual Studio installation mucking up your repro.

With the sos extension Windbg becomes useful at debugging managed code as well. To get started with Windbg and sos fire up Windbg and attach to a managed process. I'm using .Net 2.0 for my development so I need to load the sos dll from the Framework directory (for 1.1 .Net use .load clr10\sos.dll instead) . I can do this with the .loadby command:

0:000> .loadby sos mscorwks

This tells Windbg to load the debugger extension sos.dll from the same directory that the current process has mscorwks.dll loaded from.
Next we hit Debug | Go (or F5) and the process is now happily running away.

Once the problem occurs go back to Windbg and press Debug | Break and we can now examine the process state in safety. If at any point you need help with the sos commands, !help will get you going.

Coming up next week: using Windbg to debug exceptions, deadlocks, and memory problems in managed code.