Problem:

You've written an ASP.NET application that is getting OutOfMemoryExceptions.

Cause:

Let's find out...

Resolution:

Use Windbg to take a look at the heap.


 

Is it a leak?

Take a look at the memory usage of you application using perfmon. If memory is slowly increasing and never released, then you have a leak. If it is going up and down like a rollercoaster, then you are most likely using huge amounts of memory for certain operations which is later garbage collected.

How do I troubleshoot this?

Below I'll try to give you a step by step description of how to troubleshoot this scenario

1. Get a memory dump

This is done using Windbg and Adplus. If you don't have Windbg installed, check out my previous post.

 

2. Open up the dump and load SOS

Open the dump in Windbg and load the SOS extension. You'll find it in the framework directory so if you're debugging an application for Framework 2.0 look in "C:\Windows\Microsoft.NET\Framework\v2.0.50727". Anyway, load SOS by typing:

.load [path]\sos

 

 

3. Run dumpheap

Execute the following command:

!dumpheap -stat

It will show statistics for the objects on the heap in a nice little summary divided in tho four columns.

  1. The method table of the object
  2. The number of objects of this type on the heap
  3. The total size of these objects in bytes
  4. The name of the object type

Be careful not to omit the -stat parameter. If you do then Windbg will dump the address of each object in the entire heap to your screen, which will be a lot of information to say the least.

Analyzing the information provided by !dumpheap

Here's a sample heap from one of my cases...

0:000> !dumpheap -stat
Statistics:
MT             Count       TotalSize Class Name
7a787cc4           1              12 System.IO.FileSystemWatcher+FSWAsyncResult
7a75904c           1              12 System.Diagnostics.OrdinalCaseInsensitiveComparer
7a7575cc           1              12 System.CodeDom.Compiler.CodeDomConfigurationHandler
7a7571a8           1              12 System.Net.DefaultCertPolicy
7a75661c           1              12 System.Diagnostics.TraceListenerCollection
7a755834           1              12 System.Diagnostics.PerformanceCounterCategoryType
........CONTINUED.........
68a66a88     227,559      12,743,304 System.Web.UI.WebControls.Literal
68a2f7fc     399,272      14,373,792 System.Web.UI.ControlCollection
68a92e2c     768,731      33,824,164 System.Web.UI.Control+OccasionalFields
68a884a0     641,952      38,517,120 System.Web.UI.LiteralControl
79124228     879,515      43,394,976 System.Object[]
790fa3e0   1,431,594     122,806,484 System.String

Total 10,389,625 objects, Total size: 463,313,540

So in this particular dump I have 1,431,594 strings whose total size is 122 MBytes, 879,515 Objects with a total size of 43 MBytes, etc.

Objects in heap may be larger than they appear

The TotalSize column isn't 100% true. Look at the LiteralControls that are on third place. They only use a total of 38 MBytes, or do they? The TotalSize refers to the object structure, but the member variables like strings, integers and other child objects are not included. It kind of makes sense, since otherwise the Total size would be completely off the scale. The LiteralControl objects contain a number of child objects and three of those are strings. Their respective size is listed with the System.String object.

Using !dumpobj

Let's take a closer look at one of the System.Web.UI.LiteralControls. We list all of the controls with the following command !dumpheap -type System.Web.UI.LiteralControl and quickly press Ctrl+Break before the screen fills with too many lines:

0:000> !dumpheap -type System.Web.UI.LiteralControl
 Address       MT Size Gen
023ea0a8 68a884a0   60   2 System.Web.UI.LiteralControl
023ea0e4 68a884a0   60   2 System.Web.UI.LiteralControl
023ea374 68a884a0   60   2 System.Web.UI.LiteralControl
023ea460 68a884a0   60   2 System.Web.UI.LiteralControl
023ea510 68a884a0   60   2 System.Web.UI.LiteralControl
023eab3c 68a884a0   60   2 System.Web.UI.LiteralControl
........CONTINUED........
023fe31c 68a884a0   60   2 System.Web.UI.LiteralControl
023fe414 68a884a0   60   2 System.Web.UI.LiteralControl
023fe4c4 68a884a0   60   2 System.Web.UI.LiteralControl
023fe500 68a884a0   60   2 System.Web.UI.LiteralControl

As you can see each LiteralControl is 60 bytes in size. Like I said before that's the size of the object structure alone, not it's referenced objects and properties. We now pick the address of one of the LiteralControls and execute the !dumpobj command (!do for short). This gives us the following result:

0:000> !do 023ea0a8
Name: System.Web.UI.LiteralControl
MethodTable: 68a884a0
EEClass: 68a88428
Size: 60(0x3c) bytes
GC Generation: 2
(C:\WINDOWS\assembly\GAC_32\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll)
Fields:
      MT    Field Offset                   Type   VT     Attr    Value Name
790fa3e0  4001fe0      4          System.String    0 instance 00000000 _id
790fa3e0  4001fe1      8          System.String    0 instance 00000000 _cachedUniqueID
68a2af44  4001fe2      c   ...em.Web.UI.Control    0 instance 023e8864 _parent
68a91070  4001fe3     2c           System.Int32    0 instance        0 _controlState
68a85ea0  4001fe4     10   ...m.Web.UI.StateBag    0 instance 00000000 _viewState
68a2af44  4001fe5     14   ...em.Web.UI.Control    0 instance 023e8864 _namingContainer
68a273d0  4001fe6     18     System.Web.UI.Page    0 instance 01df4514 _page
68a92e2c  4001fe7     1c   ...+OccasionalFields    0 instance 00000000 _occasionalFields
68a2b378  4001fe8     20   ...I.TemplateControl    0 instance 00000000 _templateControl
68a14528  4001fe9     24   ...m.Web.VirtualPath    0 instance 00000000 _templateSourceVirtualDirectory
68a8bb48  4001fea     28   ...rs.ControlAdapter    0 instance 00000000 _adapter
68a3a8f8  4001feb     30   ...SimpleBitVector32    1 instance 023ea0d8 flags
790f9c18  4001fda    c70          System.Object    0   shared   static EventDataBinding
>> Domain:Value 000f0d00:NotInit 0011a720:01df0028 <<
790f9c18  4001fdb    c74          System.Object    0   shared   static EventInit
>> Domain:Value 000f0d00:NotInit 0011a720:01df0034 <<
790f9c18  4001fdc    c78          System.Object    0   shared   static EventLoad
>> Domain:Value 000f0d00:NotInit 0011a720:01df0040 <<
790f9c18  4001fdd    c7c          System.Object    0   shared   static EventUnload
>> Domain:Value 000f0d00:NotInit 0011a720:01df004c <<
790f9c18  4001fde    c80          System.Object    0   shared   static EventPreRender
>> Domain:Value 000f0d00:NotInit 0011a720:01df0058 <<
790f9c18  4001fdf    c84          System.Object    0   shared   static EventDisposed
>> Domain:Value 000f0d00:NotInit 0011a720:01df0064 <<
79124228  4001fec    c88        System.Object[]    0   shared   static automaticIDs
>> Domain:Value 000f0d00:NotInit 0011a720:01df0070 <<
790fa3e0  4002211     34          System.String    0 instance 02238664 _text

Cool, now we can take a look at the specifics. For example the value of the text-property, which is located in the string down at the bottom with address 02238664. To get it's value, simply perform a !do on the address:

0:000> !do 02238664
Name: System.String
MethodTable: 790fa3e0
EEClass: 790fa340
Size: 158(0x9e) bytes
GC Generation: 2
(C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
String:      </td>
    </tr>
  </table>
<!-- end of content table -->
 
Fields:
MT Field Offset Type VT Attr Value Name
790fed1c 4000096 4 System.Int32 0 instance 71 m_arrayLength
790fed1c 4000097 8 System.Int32 0 instance 70 m_stringLength
790fbefc 4000098 c System.Char 0 instance 3c m_firstChar
790fa3e0 4000099 10 System.String 0 shared static Empty
>> Domain:Value 000f0d00:790d6584 0011a720:790d6584 <<
79124670 400009a 14 System.Char[] 0 shared static WhitespaceChars
>> Domain:Value 000f0d00:01d413b8 0011a720:01d44f80 <<

Okay, so we can see that the string contains some data to close off a table. We can also take a look at the other properties for the object and examine them if we wish. But there is another command that is really useful...

Using !objsize

Is there a way to get the total size of a specific System.Web.UI.LiteralControl? - Simple! We use the !objsize -command. !objsize looks at all the pointers within an object and calculate their total size. Below is the built in documentation for the !objsize -command:

0:000> !help objsize
-------------------------------------------------------------------------------
!ObjSize [<Object address>]
 
With no parameters, !ObjSize lists the size of all objects found on managed
threads. It also enumerates all GCHandles in the process, and totals the size
of any objects pointed to by those handles. In calculating object size,
!ObjSize includes the size of all child objects in addition to the parent.
 
For example, !DumpObj lists a size of 20 bytes for this Customer object:
 
0:000> !do a79d40
Name: Customer
MethodTable: 009038ec
EEClass: 03ee1b84
Size: 20(0x14) bytes
(C:\pub\unittest.exe)
Fields:
MT         Field Offset  Type              Attr    Value  Name
009038ec 4000008      4 CLASS          instance 00a79ce4  name
009038ec 4000009      8 CLASS          instance 00a79d2c  bank
009038ec 400000a      c System.Boolean instance        1 valid
 
but !ObjSize lists 152 bytes:
0:000> !ObjSize a79d40
sizeof(00a79d40) = 152 ( 0x98) bytes (Customer)
 
This is because a Customer points to a Bank, has a name, and the Bank points to
an Address string. You can use !ObjSize to identify any particularly large
objects, such as a managed cache in a web server.

 

Objects in heap may also be smaller than they appear

So what do we get if we run !objsize on our LiteralControl? This is really interesting, because what happens is; the debugger gets really busy for quite some time and eventually we get this:

0:000> !objsize 023ea0a8
sizeof(023ea0a8) = 456918136 ( 0x1b3c0478) bytes (System.Web.UI.LiteralControl)

456 MBytes! How is that possible? Well if you scroll up to where we ran the !do command on the LiteralControl, you'll see that the control holds a reference to the page. The page in turn has a reference to the cache, and before long we'll have referenced almost the entire heap.

Summary

Hopefully this is enough to give you a quick glimpse of what is possible with three relatively simple commands from the sos extension. The commands were:

  1. !dumpheap
  2. !dumpobj
  3. !objsize

Over and out

/ Johan