mvstanton's WebLog

Some new SOS functions

mvstanton — Wed, 12 Oct 2005 01:53:00 GMT

Hi all. Here is a short tutorial about !gchandleleaks and managed breakpoints in windbg. You need at least a Whidbey Beta2 build to run it. Soon Whidbey RTM will be out too (yay!)

WINDOWH.EXE (the code is below) - from a sample app which illustrates calling EnumWindows. I modified it to run in a loop, and forget to free a gc handle each time. Over time, this will cause the gc heap to grow and finally you'll run out of memory. For this document, the demo is running on an x64 machine.

Let the program run for a few minutes. Eventually it will become a large memory user in Task Manager. Break in:

Windbg.exe –pn windowh.exe

And let’s see how large the gc heap is:

(9f0.7c8): Break instruction exception - code 80000003 (first chance)

ntdll!DbgBreakPoint:

00000000`77efa850 cc int 3

0:003> !eeheap -gc

Number of GC Heaps: 1

generation 0 starts at 0x0000000010355a10

generation 1 starts at 0x0000000010354998

generation 2 starts at 0x0000000010351000

ephemeral segment allocation context: none

segment begin allocated size

000000000019f420 00000000027c0008 00000000027c0020 0x0000000000000018(24)

0000000000199890 0000064237482c80 00000642374f5668 0x00000000000729e8(469480)

0000000010350000 0000000010351000 000000001038da28 0x000000000003ca28(248360)

Large object heap starts at 0x0000000020351000

segment begin allocated size

0000000020350000 0000000020351000 000000002759bbc8 0x000000000724abc8(119843784)

Total Size 0x72f9ff0(120561648)

------------------------------

GC Heap Size 0x72f9ff0(120561648)

About 120 megabytes, mostly in the large object heap. You could let it run some more and verify the growth.

Why is the heap so big? We don’t have the source code (pretend), and know nothing about the program. But !dumpheap can help us:

0:003> !dumpheap -stat

total 11466 objects

Statistics:

MT Count TotalSize Class Name

0000064237554400 1 24 System.Text.EncoderExceptionFallback

0000064237553b30 1 24 System.Text.DecoderExceptionFallback

000006423757d308 1 32 System.Security.PermissionToken

0000064237522870 1 32 System.Version

0000064237518650 1 32 System.Guid

000006423756d4a0 1 40 System.IO.TextWriter+NullTextWriter

000006423755bd18 1 40 Microsoft.Win32.SafeHandles.SafeViewOfFileHandle

000006423755b998 1 40 Microsoft.Win32.SafeHandles.SafeFileMappingHandle

000006423755b8b8 1 40 Microsoft.Win32.SafeHandles.SafeFileHandle

000006423755aa20 1 40 Microsoft.Win32.Win32Native+InputRecord

0000064237554180 1 40 System.Text.InternalEncoderBestFitFallback

00000642375201c0 1 40 System.SharedStatics

00000642375e4608 1 48 System.Int64[]

000006423757d600 1 48 System.Security.PermissionListSet

000006423756d6f0 1 48 System.IO.TextWriter+SyncTextWriter

0000064237556678 1 48 System.Text.UTF8Encoding

0000064237553fd0 1 48 System.Text.EncoderNLS

00000642375538e0 1 48 System.Text.InternalDecoderBestFitFallback

000006423753cdf8 1 48 System.Reflection.Assembly

000006423751b810 1 48 System.OperatingSystem

000006423757d420 1 56 System.Security.PermissionTokenFactory

000006423756dfb8 1 56 System.IO.Stream+NullStream

000006423755cbb8 1 56 System.Security.Util.TokenBasedSet

00000642375568e0 1 56 System.Text.UTF8Encoding+UTF8Encoder

00000642375d5088 1 72 System.LogLevel[]

0000064237579f90 3 72 System.Security.Permissions.SecurityPermission

000006423756e348 1 72 System.IO.__ConsoleStream

00000642375103b0 1 80 System.AppDomainSetup

0000064237550fa0 3 96 System.Globalization.CultureTableItem

00000642375258f0 1 104 System.Threading.Thread

00000642375e4838 4 128 System.Int32[]

000006423757c6f0 2 128 System.Security.PermissionSet

00000642375559e8 1 136 System.Text.SBCSCodePageEncoding

0000064237550e80 1 136 System.Globalization.CultureTable

0000064237525f68 1 136 System.Threading.ThreadAbortException

000006423750da40 1 136 System.ExecutionEngineException

000006423750d7c0 1 136 System.StackOverflowException

000006423750d680 1 136 System.OutOfMemoryException

0000064237551540 2 176 System.Globalization.CultureTableRecord

00000642375d48d0 1 200 System.SwitchStructure[]

000006423750f7a8 1 200 System.AppDomain

0000064237571d78 2 208 System.IO.StreamWriter

000006423754b208 2 256 System.Globalization.CultureInfo

0000064237509990 12 288 System.Object

000006423751d220 8 320 System.RuntimeType

00000642375505d0 2 432 System.Globalization.NumberFormatInfo

0000064237534d40 7 616 System.Collections.Hashtable

0000064237518f60 27 648 System.Int32

00000642375db1b8 10 1328 System.Char[]

0000064280131980 25 1600 CallBack

00000642375b05c8 7 2016 System.Collections.Hashtable+bucket[]

00000642801315c0 234 7488 BloatedObject

0000000000186780 238 12888 Free

00000642375d4358 8 18040 System.Object[]

000006423750c758 2088 83520 System.Text.StringBuilder

000006423750b800 8509 612640 System.String

00000642375dba38 237 119814344 System.Byte[]

Total 11466 objects

This shows that 237 System.Byte[] objects make up the bulk of the problem. Why are we creating those? If we can list those objects, we might find a root to one of them and get a better understanding of why they are being created.

0:003> !dumpheap -type System.Byte[]

Address MT Size

0000000010351de0 00000642375dba38 24

0000000010353700 00000642375dba38 416

0000000010353b08 00000642375dba38 288

0000000020357048 00000642375dba38 512024

00000000203d4078 00000642375dba38 512024

0000000020451090 00000642375dba38 512024

…(a few hundred byte arrays of size 512024 here)

00000000274a1b68 00000642375dba38 512024

000000002751eb98 00000642375dba38 512024

total 237 objects

Statistics:

MT Count TotalSize Class Name

00000642375dba38 237 119814344 System.Byte[]

Total 237 objects

!GCRoot will search handles, stacks and the freachable queue for pointers that ultimately keep the object in question alive. I’ll try it on the last one in the list:

0:003> !gcroot 000000002751eb98

Note: Roots found on stacks may be false positives. Run "!help gcroot" for

more info.

Scan Thread 0 OSTHread 5f0

Scan Thread 2 OSTHread 318

DOMAIN(000000000016F260):HANDLE(Strong):1dc1e30:Root:000000001038ac40(BloatedObject)->000000002751eb98(System.Byte[])

So a GCHandle keeps a “BloatedObject” alive (strong hint, eh), which keeps this Byte array alive. Running !gcroot on other arrays reveals the same thing. I’m suspecting a handle leak. Another way that makes this more clear is to run !GCHandles:

0:003> !gchandles

GC Handle Statistics:

Strong Handles: 245

Pinned Handles: 4

Async Pinned Handles: 0

Ref Count Handles: 0

Weak Long Handles: 25

Weak Short Handles: 1

Other Handles: 0

Statistics:

MT Count TotalSize Class Name

0000064237579f90 1 24 System.Security.Permissions.SecurityPermission

0000064237509990 1 24 System.Object

00000642375201c0 1 40 System.SharedStatics

000006423757d600 1 48 System.Security.PermissionListSet

000006423753cdf8 1 48 System.Reflection.Assembly

000006423757c6f0 1 64 System.Security.PermissionSet

0000064237525f68 1 136 System.Threading.ThreadAbortException

000006423750da40 1 136 System.ExecutionEngineException

000006423750d7c0 1 136 System.StackOverflowException

000006423750d680 1 136 System.OutOfMemoryException

000006423750f7a8 1 200 System.AppDomain

00000642375258f0 2 208 System.Threading.Thread

0000064280131980 25 1600 CallBack

00000642801315c0 234 7488 BloatedObject

00000642375d4358 3 17440 System.Object[]

Total 275 objects

BloatedObject comes up with over 200 handles. So where are these handles stored, if they are stored at all? If a GCHandle is forgotten about before calling Free(), there is no way to release the memory it keeps alive. !GCHandleLeaks tries to help determine if you have a leak by a brute force search of memory for instances of any strong gc handles that exist. If it doesn’t find some, that is indicative of a leak. Let’s run it:

0:003> !gchandleleaks

-------------------------------------------------------------------------------

GCHandleLeaks will report any GCHandles that couldn't be found in memory.

Strong and Pinned GCHandles are reported at this time. You can safely abort the

memory scan with Control-C or Control-Break.

-------------------------------------------------------------------------------

Found 249 handles:

0000000001dc1200 0000000001dc1208 0000000001dc1210 0000000001dc1218

0000000001dc1220 0000000001dc1228 0000000001dc1230 0000000001dc1238

0000000001dc1240 0000000001dc1248 0000000001dc1250 0000000001dc1258

0000000001dc1260 0000000001dc1268 0000000001dc1270 0000000001dc1278

0000000001dc1280 0000000001dc1288 0000000001dc1290 0000000001dc1298

0000000001dc12a0 0000000001dc12a8 0000000001dc12b0 0000000001dc12b8

…(the list goes on)

Searching memory

Found 0000000001dc1ff8 at location 000000000012ce48

Found 0000000001dc1380 at location 0000000000145870

Found 0000000001dc1390 at location 0000000000145918

Found 0000000001dc1378 at location 00000000001557f8

Found 0000000001dc13f8 at location 000000000016fd50

Found 0000000001dc13f0 at location 0000000000182b70

Found 0000000001dc13b0 at location 000000000019eab8

Found 0000000001dc17f8 at location 000000000019fc08

Found 0000000001dc17e8 at location 00000000001a2148

…

------------------------------------------------------------------------------

Some handles were not found. If the number of not-found handles grows over the

lifetime of your application, you may have a GCHandle leak. This will cause

the GC Heap to grow larger as objects are being kept alive, referenced only

by the orphaned handle. If the number doesn't grow over time, note that there

may be some noise in this output, as an unmanaged application may be storing

the handle in a non-standard way, perhaps with some bits flipped. The memory

scan wouldn't be able to find those.

------------------------------------------------------------------------------

Didn't find 232 handles:

0000000001dc1200 0000000001dc1208 0000000001dc1210 0000000001dc1218

0000000001dc1220 0000000001dc1228 0000000001dc1230 0000000001dc1238

0000000001dc1240 0000000001dc1248 0000000001dc1250 0000000001dc1258

0000000001dc1260 0000000001dc1268 0000000001dc1270 0000000001dc1278

0000000001dc1280 0000000001dc1288 0000000001dc1290 0000000001dc1298

0000000001dc12a0 0000000001dc12a8 0000000001dc12b0 0000000001dc12b8

0000000001dc12c0 0000000001dc12c8 0000000001dc12d0 0000000001dc12d8

0000000001dc12e0 0000000001dc12e8 0000000001dc12f0 0000000001dc12f8

0000000001dc1300 0000000001dc1308 0000000001dc1310 0000000001dc1318

0000000001dc1320 0000000001dc1328 0000000001dc1330 0000000001dc1338

0000000001dc1340 0000000001dc1348 0000000001dc1350 0000000001dc1358

0000000001dc1360 0000000001dc1368 0000000001dc1370 0000000001dc1388

0000000001dc1800 0000000001dc1808 0000000001dc1810 0000000001dc1818

0000000001dc1820 0000000001dc1828 0000000001dc1830 0000000001dc1838

0000000001dc1840 0000000001dc1848 0000000001dc1850 0000000001dc1858

0000000001dc1860 0000000001dc1868 0000000001dc1870 0000000001dc1878

0000000001dc1880 0000000001dc1888 0000000001dc1890 0000000001dc1898

0000000001dc18a0 0000000001dc18a8 0000000001dc18b0 0000000001dc18b8

…

Let’s look at one of the handles that wasn’t found. We can view the object it keeps alive by dereferencing it. I’ll use the handy poi debugger command to reduce the number of pointers I have to type:

0:003> !dumpobj poi(0000000001dc18b8)

Name: BloatedObject

MethodTable: 00000642801315c0

EEClass: 0000064280163cd0

Size: 32(0x20) bytes

(C:\pub\eetwc\windowh.exe)

Fields:

MT Field Offset Type VT Attr Value Name

000006423756d248 4000001 8 System.IO.TextWriter 0 instance 0000000010353c28 tw

00000642375dba38 4000002 10 System.Byte[] 0 instance 0000000022e50080 ba

Naturally, it’s a BloatedObject. And the byte array is in field ba. You could run the new !DumpArray command on ba, but I don’t recommend it, it is too boring for such a large array! We’ll settle for !DumpObj (shorthand !do):

0:003> !do 22e50080

Name: System.Byte[]

MethodTable: 00000642375dba38

EEClass: 00000642376efd28

Size: 512024(0x7d018) bytes

Array: Rank 1, Number of elements 512000, Type Byte

Element Type: System.Byte

Fields:

None

Who is creating these objects? Let’s try and find a handle to that. One way would be to run ILDASM.EXE and dump the disassembly to a file, then search for instances of BloatedObject to see where they get created. I’ll try just setting a breakpoint on the constructor of BloatedObject (if one exists).

0:003> !name2ee windowh.exe BloatedObject..ctor

Module: 00000000001b0300 (windowh.exe)

Token: 0x0000000006000009

MethodDesc: 0000064280131508

Name: BloatedObject..ctor(System.IO.TextWriter)

JITTED Code Address: 00000642801706e0

We know the code is jitted, so we could just use the debugger’s built-in bp command at 642801706e0. But !bpmd works even if the code isn’t jitted yet, so I tend to use it for managed code.

0:003> !bpmd windowh.exe BloatedObject..ctor

Found 1 methods...

MethodDesc = 0000064280131508

Method is jitted, placing breakpoint at code addr 00000642801706e0

0:003> g

Breakpoint 0 hit

windowh!BloatedObject..ctor(System.IO.TextWriter):

00000642`801706e0 53 push rbx

The managed stack shows we are being called from App.Work():

0:000> !clrstack

OS Thread Id: 0x650 (0)

Child-SP RetAddr Call Site

000000000012e328 00000642801705aa BloatedObject..ctor(System.IO.TextWriter)

000000000012e330 0000064280170519 App.Work()

000000000012e390 000006426ec8839e App.Main()

Without digging too much further, I can tell by looking at App.Work() that a handle to the BloatedObject we just created is initialized:

0:000> !u 00000642801705aa

Normal JIT generated code

App.Work()

Begin 0000064280170570, size 12d

00000642`80170570 push rbx

00000642`80170571 push rsi

00000642`80170572 push rdi

00000642`80170573 sub rsp,0x40

00000642`80170577 xor eax,eax

00000642`80170579 mov [rsp+0x38],rax

00000642`8017057e xor eax,eax

00000642`80170580 mov [rsp+0x30],rax

00000642`80170585 call mscorlib_ni!System.Console.get_Out() (0000064236cd6a10)

00000642`8017058a mov rdi,rax

00000642`8017058d mov rcx,0x642801315c0 (MT: BloatedObject)

00000642`80170597 call mscorwks!JIT_TrialAllocSFastMP_InlineGetThread (000006426ec6d350)

00000642`8017059c mov rbx,rax

00000642`8017059f mov rdx,rdi

00000642`801705a2 mov rcx,rbx

00000642`801705a5 call CLRStub[EntryPointStub]@64280170270 (0000064280170270) (BloatedObject..ctor(System.IO.TextWriter), mdToken: 06000009)

>>> 00000642`801705aa mov edx,0x501

00000642`801705af mov ecx,0x1

00000642`801705b4 call mscorwks!JIT_GetSharedNonGCStaticBase_InlineGetAppDomain (000006426ec6d260)

00000642`801705b9 mov r8d,0x2

00000642`801705bf mov rdx,rbx // rdx = BloatedObject, arg1 for GCHandle ctor

00000642`801705c2 lea rcx,[rsp+0x28] // rsp+0x28 is the GCHandle

00000642`801705c7 call mscorlib_ni!System.Runtime.InteropServices.GCHandle..ctor(System.Object, System.Runtime.InteropServices.GCHandleType) (0000064236cd9370)

00000642`801705cc mov rax,[rsp+0x28]

00000642`801705d1 mov [rsp+0x38],rax

…

(I omitted code bytes in the output above for better formatting.)

We never see any more mention in the function of the GCHandle, but it does get passed to another function. Maybe that function is supposed to call Free()? I’ll leave that as an exercise for the reader.

--------------------------------

Here is the windowh.cs code:

using System;
using System.Runtime.InteropServices;
using System.IO;
using System.Text;
using System.Threading;

public delegate bool CallBack( int handle, IntPtr param );

public class LibWrap
{
   // Passes a managed object as an LPARAM type.
   // Declares a managed prototype for the unmanaged function.
   [ DllImport( "user32.dll" )]
   public static extern bool EnumWindows( CallBack cb, IntPtr param );

[DllImport("user32.dll", SetLastError=true, CharSet=CharSet.Auto)]
public static extern int GetWindowTextLength(IntPtr hWnd);

[DllImport("user32.dll", SetLastError=true, CharSet=CharSet.Auto)]
public static extern int GetWindowText(IntPtr hWnd, [Out] StringBuilder lpString, int nMaxCount);
}

public class BloatedObject
{
   public TextWriter tw;
   public Byte[] ba;
   public BloatedObject(TextWriter t)
   {
      tw = t;
      ba = new Byte[500*1024];
      // Do some "work"
      for(int i=0;i<(500*1024);i++)
      {
         ba[i] = (byte) (i%256);
      }
   }
}

public class App
{
public static int NamedWindowCount = 0;

   public static void Main()
   {
      while(true)
      {
         Work();
         Thread.Sleep(500);
      }
   }
   public static void Work()
   {
      TextWriter tw = System.Console.Out;
      BloatedObject bo = new BloatedObject(tw);
      GCHandle gch = GCHandle.Alloc( bo );
      CallBack cewp = new CallBack( CaptureEnumWindowsProc );

      // Platform invoke prevents the delegate from being garbage
      // collected before the call ends.

      NamedWindowCount = 0;
      LibWrap.EnumWindows( cewp, (IntPtr)gch );
      tw.Write ( App.NamedWindowCount + "\n");

      // Don't handles free themselves? I will assume so. :p
      // gch.Free();
   }

   public static string GetText(IntPtr hWnd)
   {
      // Allocate correct string length first
      int length       = LibWrap.GetWindowTextLength(hWnd);
      StringBuilder sb = new StringBuilder(length + 1);
      LibWrap.GetWindowText(hWnd, sb, sb.Capacity);
      return sb.ToString();
   }

   private static bool CaptureEnumWindowsProc( int handle, IntPtr param )
   {
      GCHandle gch = (GCHandle)param;
      BloatedObject bo = (BloatedObject)gch.Target;
      String name = App.GetText((IntPtr)handle);
      if (name.Length > 0)
      {
         App.NamedWindowCount++;
      }

      bo.tw.Write( "." );
      return true;
   }
}

Traversing the gc heap (and introducing PSSCOR.DLL)

mvstanton — Tue, 06 Apr 2004 00:58:00 GMT

We have an improved SOS.DLL with many bug fixes and enhancements. Tom Christian in Product Support maintains it, and gave me permission to post it here under the name PSSCOR.DLL.

[update June 2005 - For some time now, PSSCOR.DLL has been included with the Windows Debugger package, although it is renamed to SOS.DLL. I've removed this old link because you get it just by installing the debugger].

It works on V1.0 and V1.1 of the CLR. Load it in the same way you'd load sos.dll in the Windows Debugger, with “.load psscor.dll“. The good thing about PSSCOR.DLL is that we can fix bugs and enhance functions without going through a lengthy QFE process. If you've found bugs in SOS, it's likely that many were fixed already in PSSCOR.DLL.

The code examples below use psscor.dll to explore the gc heap. You'll want to use it in lieu of SOS, because some commands like !DumpMT and !EEHeap have additional useful output.

It's useful to know how objects are laid out in the gc heap. During garbage collection, valid objects are marked by recursively visiting objects starting from roots on stacks and in handles. But it's also important that the location of the objects sit in an organized way from the beginning to end of each heap segment. The psscor !DumpHeap command counts on this logical organization to walk the heap properly, and if it reports an error you can bet something is wrong with your heap (and will bite you later with a perplexing application violation). So to understand what !dumpheap is talking about, here is your guide to walking these objects by hand, hopping from one stone to another across a vast lake.

First you need a program. I have taken this program from Joel Pobar's Reflection Emit example, and inserted a PInvoke to DebugBreak so you can easily stop in the Windows Debugger. (You could use Visual Studio for these illustrations too, but the Windows Debugger "dd" command is quicker for viewing memory).

using System;
using System.Reflection;
using System.Reflection.Emit;
using System.Threading;
using System.Runtime.InteropServices;

public class EmitHelloWorld
{

[DllImport("kernel32")]
public static extern void DebugBreak();

      static void Main(string[] args)
      {
            // create a dynamic assembly and module
            AssemblyName assemblyName = new AssemblyName();
            assemblyName.Name = "HelloWorld";
            AssemblyBuilder assemblyBuilder = Thread.GetDomain().DefineDynamicAssembly(assemblyName, AssemblyBuilderAccess.RunAndSave);
            ModuleBuilder module;
            module = assemblyBuilder.DefineDynamicModule("HelloWorld.exe");

            // create a new type to hold our Main method
            TypeBuilder typeBuilder = module.DefineType("HelloWorldType", TypeAttributes.Public | TypeAttributes.Class);

            // create the Main(string[] args) method
            MethodBuilder methodbuilder = typeBuilder.DefineMethod("Main", MethodAttributes.HideBySig | MethodAttributes.Static | MethodAttributes.Public, typeof(void), new Type[] { typeof(string[]) });

            // generate the IL for the Main method
            ILGenerator ilGenerator = methodbuilder.GetILGenerator();
            ilGenerator.EmitWriteLine("hello, world");
            ilGenerator.Emit(OpCodes.Ret);

            // bake it
            Type helloWorldType = typeBuilder.CreateType();

            // run it
            helloWorldType.GetMethod("Main").Invoke(null, new string[] {null});

            DebugBreak();
            // set the entry point for the application and save it
            assemblyBuilder.SetEntryPoint(methodbuilder, PEFileKinds.ConsoleApplication);
            assemblyBuilder.Save("HelloWorld.exe");
      }
}

Save the program as example.cs, compile and run "cdb -g example.exe"

When you reach the breakpoint, load psscor and run "!eeheap -gc". It lists the heap segments that objects are stored in:

0:000> !eeheap -gc
Number of GC Heaps: 1
generation 0 starts at 0x00aa1b78
generation 1 starts at 0x00a5100c
generation 2 starts at 0x00a51000
ephemeral segment allocation context: none
segment    begin allocated     size
00a50000 00a51000 00ae4000 0x00093000(602112)
Large object heap starts at 0x01a51000
segment    begin allocated     size
01a50000 01a51000 01a54060 0x00003060(12384)
Total Size   0x96060(614496)
------------------------------
GC Heap Size   0x96060(614496)

This is a small gc heap, with only one normal object segment, and one large object segment (for objects over 80K). It's fine for our purposes. Normal-sized objects start at address 00a51000, and end at 00ae4000. In general we have this simple pattern:

       |---------|   segment.begin = 00a51000
       |object 1 |
       |_________|
       |object 2 |
       |_________|
       |   ...   |
       |_________|
       |object N |
       |_________| segment.allocated = 00ae4000

How large is each object? You can run !dumpobj to find out. The interesting thing is that each object has a 4 byte header, and the size of the header for object 2 is included in the size of object 1. Another point is that a special kind of object called a "Free" object lives in the heap. This is used to plug holes between valid objects. These Free objects are temporary, in that if a compacting gc occurs they'll disappear. Yun wrote a great article about how the heap could be unable to compact in the face of heavy pinning , and be filled with Free objects (http://blogs.msdn.com/yunjin/archive/2004/01/27/63642.aspx).

Let's start walking. (My heap may look different because it's a Whidbey debug build)

0:000> !dumpobj 00a51000
Free Object
Size 12(0xc) bytes
0:000> !dumpobj 00a51000+c
Free Object
Size 12(0xc) bytes
0:000> !dumpobj 00a51000+c+c
Free Object
Size 12(0xc) bytes
0:000> !dumpobj 00a51000+c+c+c
Name: System.OutOfMemoryException
MethodTable: 03077e9c
EEClass: 03064050
Size: 68(0x44) bytes
(C:\WINDOWS\Microsoft.NET\Framework\v2.0.x86dbg\mscorlib.dll)
Fields:
      MT    Field   Offset                 Type       Attr    Value Name
03076b7c 40000a5        4                CLASS   instance 00000000 _className
03076b7c 40000a6        8                CLASS   instance 00000000 _exceptionMe
thod
03076b7c 40000a7        c                CLASS   instance 00000000 _exceptionMe
thodString
03076b7c 40000a8       10                CLASS   instance 00000000 _message
03076b7c 40000a9       14                CLASS   instance 00000000 _data
03076b7c 40000aa       18                CLASS   instance 00000000 _innerExcept
ion
03076b7c 40000ab       1c                CLASS   instance 00000000 _helpURL
03076b7c 40000ac       20                CLASS   instance 00000000 _stackTrace
03076b7c 40000ad       24                CLASS   instance 00000000 _stackTraceS
tring
03076b7c 40000ae       28                CLASS   instance 00000000 _remoteStack
TraceString
03076b7c 40000af       30         System.Int32   instance        0 _remoteStack
Index
03076b7c 40000b0       34         System.Int32   instance -2147024882 _HResult
03076b7c 40000b1       2c                CLASS   instance 00000000 _source
03076b7c 40000b2       38        System.IntPtr   instance        0 _xptrs
03076b7c 40000b3       3c         System.Int32   instance -532459699 _xcode

Wow, it took some time to get to something interesting. You could continue like this until you get a buffer overflow due to all the "+c+44+68+12+..." You can also let !DumpHeap do this for you. It gives a rather sparse printout of the object pointers. Let's limit the output to the segment we care about (and note that Size is in decimal):

0:000> !dumpheap 00a51000 00ae4000
Address       MT     Size
00a51000 0015c260       12 Free
00a5100c 0015c260       12 Free
00a51018 0015c260       12 Free
00a51024 03077e9c       68
00a51068 030782cc       68
00a510ac 030786fc       68
00a510f0 03078b5c       68
00a51134 030f7b54       20
00a51148 0308b06c      108
00a511b4 030fa5bc       32
00a511d4 0305bbf8       28
00a511f0 030592e0       80
00a51240 0015c260       72 Free
...

How do we know the size of each object? Just look at the MethodTable, the first DWORD of the object. You can run !dumpmt on it:

0:000> !dumpmt 03077e9c
EEClass: 03064050
Module: 0016b118
Name: System.OutOfMemoryException
mdToken: 02000038 (C:\WINDOWS\Microsoft.NET\Framework\v2.0.x86dbg\mscorlib.dll)
BaseSize: 44
Number of IFaces in IFaceMap: 2
Slots in VTable: 21

BaseSize is in hex here. (We have a hard time deciding how we like to see these things!) How about arrays, how do we know their size? Let's list all the arrays in the segment to figure it out:

0:000> !dumpheap -type [] 00a51000 00ae4000
Address       MT     Size
00a511f0 030592e0       80
00a5129c 03115b68       56
00a51348 03135ca0       76
00a513a8 030592e0       16
00a51434 0313b1c0      144
00a51634 0313c234      100
00a51698 0313c620       56
00a51cc4 030592e0       16
00a51e8c 0313b1c0      144
00a52008 0313b1c0      144
00a52244 0313b1c0      144
00a52308 0313b1c0      144
00a523cc 0313b1c0      144
00a52620 0313b1c0      144
00a526e4 0313b1c0      144
00a52a14 031e23f8       36
00a52b7c 0313b1c0      144
00a52c0c 0315778c     1084
00a53048 0315778c     1628
00a536a4 0315778c      824
...

Picking one at random:

0:000> !dumpobj 00a52c0c
Name: System.Int32[]
MethodTable: 0315778c
EEClass: 03157708
Size: 1084(0x43c) bytes
Array: Rank 1, Type Int32
Element Type: System.Int32
Fields:
None

The formula for determining array size is:

MethodTable.BaseSize + (MethodTable.ComponentSize * Object.Components)

!dumpmt will tell you the first two:

0:000> !dumpmt 315778c
EEClass: 03157708
Module: 0016b118
Name: System.Int32[]
mdToken: 02000000 (C:\WINDOWS\Microsoft.NET\Framework\v2.0.x86dbg\mscorlib.dll)
BaseSize: 0xc
ComponentSize: 0x4
Number of IFaces in IFaceMap: 4
Slots in VTable: 25

and you can find the number of items in the array with:

0:000> dd 00a52c0c+4 l1
00a52c10 0000010C
0:000>

[I'm sure Josh Williams will come along and chide me for forgetting that on 64-bit pointers are 8 bytes, so I'd have to add 8 instead of 4 above. :p]. 0xc + (0x10C*0x4) = 0x43c, so our size is correct.

So we understand object sizes, and how they are arranged. There is one thing missing though, and this is the presence of zero-filled regions throughout the heap called Allocation Contexts. For efficiency, each managed thread can be given such a region to direct new allocations to. This allows multithreaded apps to allocate without expensive locking operations. There is also an Allocation Context for the heap segment that contains generations 0 and 1 (also called the Ephemeral Segment). The !dumpheap command is aware of these regions, and steps lightly over them. You can get the thread Allocation Context addresses with the !threads command:

0:000> !threads
ThreadCount: 2
UnstartedThread: 0
BackgroundThread: 1
PendingThread: 0
DeadThread: 0
                                 PreEmptive   GC Alloc               Lock
      ID OSID ThreadOBJ    State     GC       Context       Domain   Count APT Exception
0    1 16ac 00155da8      a020 Enabled 00ae2e1c:00ae3ff4 0014a890     0 MTA
2    2 169c 001648f8      b220 Enabled 00000000:00000000 0014a890     0 MTA (Finalizer)

Thread 0 (the main thread) has an allocation context, from 00ae2e1c to 00ae3ff4. If we look at that memory, we'll see all zeros:

0:000> dd 00ae2e1c
00ae2e1c 00000000 00000000 00000000 00000000
00ae2e2c 00000000 00000000 00000000 00000000
00ae2e3c 00000000 00000000 00000000 00000000
00ae2e4c ...

As for the Ephemeral Segment Allocation Context, we don't have one. Recalling !eeheap -gc output:

You might end up with a buffer overflow someday, and obliterate the MethodTable of an object right after your array of StrongBad fan club members. The next time a GC occurs, your program will crash. Let's simulate that dreadful occurrance and see how !dumpheap responds:

0:000> ed adf7f8 00650033 (I'm overwriting the MethodTable of the array we've been enjoying)
0:000> !dumpheap 00a51000 00ae4000
...
00adf7ac 03135ca0 76
object 00adf7f8: does not have valid MT
curr_object : 00adf7f8
Last good object: 00adf7ac

This allows you to become suspicious of the last good object, 00adf7ac. Of course we know he's alright, he's not responsible for what happened. But in the real world, an aggressive response is required! [imagine WWII air-raid siren here]

What is that last good object anyway?

0:000> !dumpobj adf7ac
Name: System.Byte[]
MethodTable: 03135ca0
EEClass: 03135c1c
Size: 76(0x4c) bytes
Array: Rank 1, Type Byte
Element Type: System.Byte
Fields:
None

Who cares about him? If I can find a root to this object on a stack, I may be close to code that would overwrite the next object:

0:000> !gcroot adf7ac
Note: Roots found on stacks may be false positives. Run "!help gcroot" for
more info.
Scan Thread 0 OSTHread 16ac
ESP:12ea9c:Root:00ad4914(System.Reflection.Emit.MethodBuilder)->00ad4448(System.
Reflection.Emit.TypeBuilder)->00adf52c(System.Reflection.Emit.MethodBuilder)->00
adf74c(System.Reflection.Emit.ILGenerator)->00adf7ac(System.Byte[])
Scan Thread 2 OSTHread 169c

Thread 0, eh? He's employed by an ILGenerator, eh? What kind of nefarious operations are going on in their shop! Okay I'll stop. But it's true, often the last good object is somehow responsible, and a PInvoke overrun is the reason why.

I've ignore the Large Object Heap Segment, but it is crawled in the same way. It has no pesky Allocation Contexts to muddy the water. Large Object segments are never compacted, it would take to long to move such objects around, as they are over 80K in size.

Have fun with PSSCOR.DLL.

Introduction

mvstanton — Thu, 04 Dec 2003 21:51:00 GMT

Howdy! I'm Michael Stanton, and I work on the CLR dev team. Specifically I lead the Quick Response Team, and we look at some of the most godawful stress failures you can think of from various labs inside Microsoft. We also look at customer issues in conjunction with Product Support Services (PSS). What I intend to share here is my bag of tips and tricks for debugging in managed processes. Primary tools are WINDBG (or it's cousins CDB and NTSD), and SOS.DLL, our team's debugger extension with a set of nifty CLR-specific commands.

Until I can get there, here is a tidbit not everyone knows about: in V1.1 of the CLR, you can load SOS.DLL in Visual Studio. First, enable unmanaged debugging in your solution properties. Then in the Immediate Window during a debugging session, just type “.load sos” [edited April 2004 to fix a bug in my steps]. Then type “!help” and you'll see a list of commands you can call, just as if you were debugging in WINDBG.

Happy hunting!