If broken it is, fix it you should

Using the powers of the debugger to solve the problems of the world - and a bag of chips    by Tess Ferrandez, ASP.NET Escalation Engineer (Microsoft)

!dumpheap –stat explained… (debugging .net leaks)

!dumpheap –stat explained… (debugging .net leaks)

Rate This
  • Comments 15


The most powerful command when debugging a managed memory leak is by far !dumpheap. It will show you all the objects on the managed heaps and using the different switches of !dumpheap you can display the output in virtually any way you want.

 

!dumpheap is a function of the sos.dll extension that comes with the framework installation (in the framework directory) and if you have the SDK installed you can find some basic help for it’s usage in C:\Program Files\Microsoft Visual Studio .NET 2003\SDK\v1.1\Tool Developers Guide\Samples\sos.

There are two categories of objects stored on the heap

   Objects that are rooted somewhere, i.e. something in the application has a pointer to them 
   Objects that have been created or un-rooted since the last garbage collection

A good trick if you want to know where you leak and want to avoid looking through a lot of data that will be garbage collected soon is to run a stress-test, then induce a GC by calling GC.Collect(3), take a memory dump, then stress it a bit more, induce a GC again and take another memory dump and compare the objects on the heap.   

The –stat switch (statistics) shows a summary, per type of the objects on the heap

0:000> !dumpheap -stat
0x79c489a0          1            12 System.Runtime.Remoting.Messaging.ClientContextTerminatorSink
0x79bf9aec          1            12 System.IO.TextReader/NullTextReader
0x79be7078          1            12 System.Runtime.Remoting.Proxies.ProxyAttribute
0x79bce8e0          1            12 System.Runtime.InteropServices.ComVisibleAttribute
0x79bce7c8          1            12 System.CLSCompliantAttribute
0x79bc08e0          1            12 System.Empty
0x0618ae68          1            12 System.Web.Configuration.CustomErrorsConfigHandler
0x061887f8          1            12 System.Web.UI.WebControls.UnitConverter
0x06180848          1            12 System.Drawing.ColorConverter
0x05dbfbc4          1            12 System.Data.Res

<… cut to save space …>

0x03f1236c        625     2,820,896 System.Char[]
0x04ad88f4    102,874     2,880,472 System.Web.UI.ControlCollection
0x0469bdf0    156,650     3,133,000 System.Collections.Specialized.HybridDictionary
0x04ad91bc    164,516     3,290,320 System.Web.UI.Triplet
0x03f134a8      7,582     3,799,704 System.Collections.Hashtable/bucket[]
0x04ade5e4     47,395     4,549,920 System.Web.UI.WebControls.Label
0x061826bc     58,197     4,888,548 System.Web.UI.DataBoundLiteralControl
0x04adff44    323,119     5,169,904 System.Web.UI.StateItem
0x0618788c     63,437     6,089,952 System.Web.UI.WebControls.TableCell
0x0469c5c4    309,132     6,182,640 System.Collections.Specialized.ListDictionary/DictionaryNode
0x0011cec0        305     6,240,720      Free
0x79ba2ee4    270,831     6,499,944 System.Collections.ArrayList
0x03f16d9c        222     7,703,284 System.DateTime[]
0x04add34c    105,502     8,018,152 System.Web.UI.LiteralControl
0x0615c6f4    558,019    11,160,380 System.Data.DataRowView
0x03f15d1c      3,783    15,447,528 System.Boolean[]
0x060bcc74    570,274    22,810,960 System.Data.DataRow
0x03f15fd4        702    50,930,472 System.Decimal[]
0x03f131e8     21,013    60,573,352 System.Int32[]
0x03f1209c    508,734    75,399,184 System.Object[]
0x79b94638  5,286,303   697,441,440 System.String
Total 9,712,896 objects, Total size: 1,032,127,612

The 1st column shows the method table for the type. If you dump out the actual data for an object you will find that the first DWORD is the method table and this contains links to information about the type, such as what its member variables are, what methods it implements etc. Essentially, with the method table you can uniquely identify the type.

The 2nd column lists the number of objects of this type on the heap, so in the example above we have 5,286,303 strings on the heap.

The 3rd column shows the total size of all the objects, so again, in the example above our 5,286,303 strings occupy a total of ~695 MB

However, total size is a truth with modification, what is shown in the 3rd column is the total size of the structures of the objects, not including member variables.

Lets take a System.Data.DataSet for example

0x060bbd2c        221        17,680 System.Data.DataSet

Here we can see that 221 datasets take up a total of 17,680 bytes, which means that on average they take up 80 bytes each. That would be a very, very small dataset if the size shown was the complete dataset.

If we dump out the contents of a dataset on the heap by first running !dumpheap –mt <methodtable> and then picking one out in the resulting list and running !dumpobj <address> on it we will see something like the following.

0:000> !dumpobj 0x3920ed4c
Name: System.Data.DataSet
MethodTable 0x060bbd2c
EEClass 0x060d2614
Size 80(0x50) bytes
GC Generation: 2
mdToken: 0x0200003b  (c:\windows\assembly\gac\system.data\1.0.5000.0__b77a5c561934e089\system.data.dll)
FieldDesc*: 0x060bb358
        MT      Field     Offset                 Type       Attr      Value Name
0x060b252c 0x4000583      0x4                CLASS   instance 0x00000000 site
0x060b252c 0x4000584      0x8                CLASS   instance 0x00000000 events
0x060b252c 0x4000582        0                CLASS     shared   static EventDisposed
    >> Domain:Value 0x001192a0:NotInit  0x0017fc40:NotInit  0x044b7b28:0x1c357cb8 <<
0x060bbd2c 0x40003d3      0xc                CLASS   instance 0x00000000 defaultViewManager
0x060bbd2c 0x40003d4     0x10                CLASS   instance 0x3920ee28 tableCollection
0x060bbd2c 0x40003d5     0x14                CLASS   instance 0x3920ed9c relationCollection
0x060bbd2c 0x40003d6     0x18                CLASS   instance 0x00000000 extendedProperties
0x060bbd2c 0x40003d7     0x1c                CLASS   instance 0x1c357c90 dataSetName
0x060bbd2c 0x40003d8     0x20                CLASS   instance 0x182d0224 _datasetPrefix
0x060bbd2c 0x40003d9     0x24                CLASS   instance 0x182d0224 namespaceURI
0x060bbd2c 0x40003da     0x40       System.Boolean   instance 0 caseSensitive
0x060bbd2c 0x40003db     0x28                CLASS   instance 0x14309a0c culture
0x060bbd2c 0x40003dc     0x41       System.Boolean   instance 1 enforceConstraints
0x060bbd2c 0x40003dd     0x42       System.Boolean   instance 0 fInReadXml
0x060bbd2c 0x40003de     0x43       System.Boolean   instance 0 fInLoadDiffgram
0x060bbd2c 0x40003df     0x44       System.Boolean   instance 0 fTopLevelTable
0x060bbd2c 0x40003e0     0x45       System.Boolean   instance 0 fInitInProgress
0x060bbd2c 0x40003e1     0x46       System.Boolean   instance 1 fEnableCascading
0x060bbd2c 0x40003e2     0x47       System.Boolean   instance 0 fIsSchemaLoading
0x060bbd2c 0x40003e3     0x2c                CLASS   instance 0x00000000 rowDiffId
0x060bbd2c 0x40003e4     0x48       System.Boolean   instance 0 fBoundToDocument
0x060bbd2c 0x40003e5     0x30                CLASS   instance 0x00000000 onPropertyChangingDelegate
0x060bbd2c 0x40003e6     0x34                CLASS   instance 0x00000000 onMergeFailed
0x060bbd2c 0x40003e7     0x38                CLASS   instance 0x00000000 onDataRowCreated
0x060bbd2c 0x40003e8     0x3c                CLASS   instance 0x00000000 onClearFunctionCalled
0x060bbd2c 0x40003e9        0                CLASS     shared   static zeroTables
    >> Domain:Value 0x0017fc40:NotInit  0x044b7b28:0x1c357c80 <<

So 80 bytes is how much is needed to store all the pointers to the member variables, but the actual data is contained in the table collection and it’s member variables.  If you need to get the actual size (including member variables for the object) you can run !objsize <address>.

Since the member variables are objects in themselves they will be listed separately in the !dumpheap –stat output, so the end sum: 1,032,127,612 bytes in this case, is the actual size of all data stored on the heaps.

Some types, like String, byte[], char[] contain the actual data in the structure which means that the total size shown by !dumpheap –stat will be the same or very close to the total size of the objects.  Both because of this and because these are very common object types you will see them as the biggest memory hoggers most of the time when you run !dumpheap –stat so usually when you are looking for a memory leak it is more interesting to look at some of the other more unique objects.

In this particular case we can see a huge amount of DataRows, so a good starting point here is to look at why there are so many DataRows. (a lot of the strings are probably entries in the datarows), and then continue looking at the number of DataSets etc.  If there was a large number of DataSets we would be interested in why there were so many… in this case as we can see there were 221 which means that there is a large number of DataRows per DataSet which means that the datasets are pretty big. (We could run !objsize on a few of them to verify)

I will talk in a later post about how to track down why/where an object is rooted (i.e. what is keeping it alive) but for now I want to concentrate on the !dumpheap output.

Another important thing to notice is the Free objects. These are not really objects but rather objects that have been marked as Free during a garbage collection but where the space is not yet compacted.   If you have a lot of free space on the heap you may have a problem with pinning (a lot is a very vague term, but if more than 30% of the managed heap is Free you should definitely look into it).  For some good explanations on this, please read Maoni’s blog http://blogs.msdn.com/maoni


Some other switches I use quite a bit are –mt   (dump the objects with a specific method table), -type  (dump the objects that match a specific pattern), -min  (dump any objects over a specific size) and –fix (dump all the objects between two heap addresses)

You can combine these in pretty much any way you want like this:
   !dumpheap –type String –min 85000 
which will dump out all Strings on the large object heap. (85000 bytes is the limit for when an object gets stored on the large object heap vs. on the normal .net heaps). 

The result will look something like this

0:067> !dumpheap -type String -min 85000
Using our cache to search the heap.
Address    MT           Size     Gen
0x212d0030 0x79b94638   86,812   -1 System.String dDwtMTE0NjcyOTQ4MTt0PDtsPGk8Mz
0x222d0030 0x79b94638   86,812   -1 System.String dDwtMTE0NjcyOTQ4MTt0PDtsPGk8Mz
0x232d0030 0x79b94638   86,688   -1 System.String dDwtMTE0NjcyOTQ4MTt0PDtsPGk8Mz
Statistics:
        MT      Count     TotalSize Class Name
0x79b94638          3       260,312 System.String
Total 3 objects, Total size: 260,312

So we had 3 strings on the large object heap, they were 86,812, 86,812 and 86,688 bytes respectively, which means that they contain around 40,000 characters each since a character is 2 bytes.

When strings are listed with !dumpheap the first few characters of the strings are listed for identification and a little piece of curiosa is that strings like the above that start with dDw are Base64 encoded strings. Most of the time when you see these they are viewstate strings so in this case the process is running some pages with pretty large viewstate.

Til next time...





  • Cool !!

    I can now understand how the things goes inside my programs with this tips ! I don't like GC "programming way" but if I need to use now this type of kind, at least I can understand it now !

    Thanks !!!





  • This must be a naive question - how and where does one run GC.Collect (as mentioned in the article) ? In the code ? Then how do i run sos after running GC.collect ?
  • Hi Charkravarthy,

    If it is an asp.net application you can have a dummy page that just calls GC.Collect, if it's a winforms app, perhaps a button or similar.  Then break in with windbg and load sos.dll in there.  

  • It always feels like issues come in clusters. One week we get tones of cache related cases, next week...
  • This is all good stuff, thanks, but....

    is it even possible to get the values of the items in collection? Take below as an example;

    0:024> !do 0x06358508
    Name: System.Collections.ArrayList
    MethodTable 0x79ba2ee4
    EEClass 0x79ba3020
    Size 24(0x18) bytes
    GC Generation: 1
    mdToken: 0x02000100  (c:\windows\microsoft.net\framework\v1.1.4322\mscorlib.dll)
    FieldDesc*: 0x79ba3084
           MT      Field     Offset                 Type       Attr      Value Name
    0x79ba2ee4 0x4000362      0x4                CLASS   instance 0x06358520 _items
    0x79ba2ee4 0x4000363      0xc         System.Int32   instance 7 _size
    0x79ba2ee4 0x4000364     0x10         System.Int32   instance 8 _version
    0x79ba2ee4 0x4000365      0x8                CLASS   instance 0x00000000 _syncRoot


    Now try and dump the items..

    0:024> !do 0x06358520
    Name: System.Object[]
    MethodTable 0x01b4209c
    EEClass 0x01b42018
    Size 80(0x50) bytes
    GC Generation: 1
    Array: Rank 1, Type CLASS
    Element Type: System.Object
    Content: 16 items

    ok 16 items, but how can you get their values?
  • Hi Howard,

    you get them with !do -v, or by running dd (dump dword) on the address of the object array and then !do on the individual dwords...  
  • Off topic ;-), how do you get windbg to format decimals with ','

    Thanks...
  • ahhh, by having regional settings for numbers set to Swedish:)    We use ,'s instead of .'s here by the north pole:)  You'll probably see .'s instead...
  • It always feels like issues come in clusters. One week we get tones of cache related cases, next week

  • PingBack from http://blogs.msdn.com/tess/pages/blog-post-index.aspx

  • W poniższym artykule wykorzystano materiały znajdujące się w następujących artykułach Tess: http://blogs.msdn.com/tess/archive/2005/11/25/i-have-a-memory-leak-what-do-i-do-defining-the-where.aspx

  • Does windbg cache the results of !dumpheap?

    I ran it 3 times and got the same results which is definately wrong because when i detached and re attached the debugger the results were massively different.

  • windbg doesnt cache the results but sos does.  however if you are live attaching and hit go in between and then run !dumpheap -stat again you should get new results.

  • !dumpheap –type String –min 85000, your last example

    The output under the Gen column is a -1. Is this GC generation. Are all LOH objects in generation  -1?

    Can you please explain what this column means?

  • Hi Cristan,

    Technically there is no gen -1 but I believe the version of SOS that I was using printed -1 for the generation if there was no actual generation, i.e. if it was on the LOH or if it had already been freed.

Page 1 of 1 (15 items)
Leave a Comment
  • Please add 4 and 3 and type the answer here:
  • Post