@TessFerrandez
I was helping a colleague out with an OOM (OutOfMemory) situation he was dealing with.
Problem description:
Their applications memory usage would grow over time until they finally ended up with an out of memory exception.
First debug:
They had gotten a memory dump when memory usage was really high 1.4 GB using debug diag and I opened it up in windbg.exe, loaded up sos (.loadby sos mscorwks) and ran !dumpheap -stat to get the content of the GC heaps.
The output above, showing the most memory consuming objects, tells us that there are pretty much two types of objects that consume most of the memory. Data related items and UI related items.
Notes about finding lots of UI objects on the heap
If you have followed my blog, you have probably noticed that I have spoken about this particular pattern before, with many UI related items, and that it is usually due to storing user controls in session or cache, and/or using static controls that where you set up event handlers in the page class to handle events for these static controls.
Basically, anytime you store a control or a UI item in session scope, cache or static variables, you will also hold a reference to the page that it was created on, as well as any controls it has and any data that might be databound to any of the controls on the pages. In other words, until the control goes out of scope, i.e. is removed from cache or session state, these objects will not be available for garbage collection.
Here are a few posts that I have written on the topic:
ASP.NET Memory: Thou shalt not store UI objects in cache or session scope ASP.NET Quiz Answers: Does Page.Cache leak memory? .NET Memory Leak Case Study: The Event Handlers That Made The Memory Baloon
Next debugging actions
When you see a lot of UI objects on the heap like this, the next step is to figure out why they are sticking around. One of the things I will always do first in these cases is to look at aspx and ascx pages on the heap and !gcroot them to see where they are rooted, i.e. what is keeping them in memory.
In this case there were 1230 aspx pages on the heap, in reality there should be approximately one per currently executing request, so this tells us that they are definitely staying longer than they should since I couldn't find a single thread executing a request when I printed out all the callstacks with ~* e !clrstack.
I gcrooted one of the pages to see why it is sticking around and found that it was stored in a static HybridDictionary...
And if I run !objsize on this Hybrid dictionary I find that it holds on to about 941 MB of data so this is certainly very interesting...
I have to add a small caveat here... if you !objsize something that contains an aspx page, your objsize will include the size of the cache since the page has an indirect reference to the cache.
In this case though the cache is very small so most of the memory held up by this HybridDictionary is the pages itself.
Ok, so now we know that our issue is due to the fact that we have a lot of pages in memory, and they are sticking around in a static HybridDictionary. The next step is to find out what this hybrid dictionary is and who is populating it with pages, and why...
To do this I dump out the HashTable+bucket[] that contains the page and search for the page address (6d67cb28) and the result was the entry displayed below
[132] 3eac5444 Name: System.Collections.Hashtable+bucket MethodTable 791021d8 EEClass: 79102154 Size: 20(0x14) bytes (C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll) Fields: MT Field Offset Type VT Attr Value Name 790fd0f0 4000937 0 System.Object 0 instance 6d67cb28 key 790fd0f0 4000938 4 System.Object 0 instance 6d67d388 val 79102290 4000939 8 System.Int32 1 instance 27189591 hash_coll ...
The page seems to be stored as the Key of this entry, so someone is populating a HybridDictionary with a key/value pair, where key=<the page>...
The value in this case is a ListDictionary
and if we print out the first entry of the listdictionary (the head node) we find that the value is Ajax.NET.Prototype...
0:028> !do 6d67d3a4 Name: System.Collections.Specialized.ListDictionary+DictionaryNode MethodTable: 7a7582d8 EEClass: 7a7c4220 Size: 20(0x14) bytes GC Generation: 2 (C:\WINDOWS\assembly\GAC_MSIL\System\2.0.0.0__b77a5c561934e089\System.dll) Fields: MT Field Offset Type VT Attr Value Name 790fd0f0 4001167 4 System.Object 0 instance 027773d8 key 790fd0f0 4001168 8 System.Object 0 instance 6d67d2e8 value 7a7582d8 4001169 c ...ry+DictionaryNode 0 instance 6d67d44c next
0:028> !do 027773d8 Name: System.String MethodTable: 790fd8c4 EEClass: 790fd824 Size: 54(0x36) bytes GC Generation: 2 (C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll) String: Ajax.NET.prototype Fields: MT Field Offset Type VT Attr Value Name 79102290 4000096 4 System.Int32 1 instance 19 m_arrayLength 79102290 4000097 8 System.Int32 1 instance 18 m_stringLength 790ff328 4000098 c System.Char 1 instance 41 m_firstChar 790fd8c4 4000099 10 System.String 0 shared static Empty >> Domain:Value 000d5d68:790d884c 000fa020:790d884c << 7912dd40 400009a 14 System.Char[] 0 shared static WhitespaceChars >> Domain:Value 000d5d68:026203f4 000fa020:0262429c <<
To figure out where this comes from, I saved out all the modules to disc using !for_each_module
!savemodule ${@#Base} f:\blog\modules\${@#ModuleName}.dll
as described in http://blogs.msdn.com/tess/archive/2008/01/10/using-reflector-to-search-through-code-and-resolve-net-issues.aspx.
I loaded them all up in reflector, and did a string search for Ajax.NET.Prototype, which returned a method containing this string(AjaxPro.Utility.RegisterCommonAjax, in AjaxPro.dll)
Armed with this, I searched the internet for "AjaxPro memory leak HybridDictionary" and found http://www.ajaxpro.info/changes.txt which shows a list of changes in the AjaxPro library. Of particular interest is a fix made in version 6.4.27.1
Version 6.4.27.1 (beta) - Fixed null values to DBNull.Value for System.Data.DataTable. - Fixed memory leak with HybridDictionary for JavaScript include rendering.
And the solution here was to upgrade to the latest version of AjaxPro.
About AjaxPro
Normally I don't write about 3rd party products, especially issues with 3rd party products, but I know that AjaxPro is a nice AJAX library that is used by quite a few of our customers so I hope this post is of general interest, as this issue is resolved in a later version of AjaxPro. I also hope that the post can be of use for finding other similar issues outside of AjaxPro.
I should also mention that I am posting this with the permission of Michael Schwartz who developed AjaxPro, and I must say that I was extremely impressed with the openness in his response when I asked if it was ok to post about this. His comment was, "feel free to write about it, I love to hear critics if there are any... to improve developement and/or fix bugs"
AjaxPro is now open source and can be downloaded from codeplex here http://www.codeplex.com/AjaxPro/. If you are interested in this you should also visit Michaels blog at http://weblogs.asp.net/mschwarz/
Laters,
Tess
Hello, I'm using AjaxPro, too, and I am very happy with the library. I have another question: is it possible to get a dump of a running system without interacting/stopping the users?
yes, you can take a memory dump with adplus (debugging tools for windows) in hang mode (adplus -pn w3wp.exe -hang) or a full dump with debug diag 1.1. It will stop the process momentarily but usually not long enough for the users to notice.
Awesome! You've used some many tools that I've never heard of before. I do have some sites that tend to grow in memory usage more than others, and it's a complete mystery. I hope I'll be able to reproduce some of the steps you showed. Thanks.
In some cases you may end up with the following exception when working with RadControls for ASP.NET Ajax