Identifying Memory Leak With Process Explorer And Windbg

Your ASP.NET web application behaves strange. It gets stuck, it stop serving incoming requests, or gets recycled. You observe strange application pool recycles events in event log.
Chances you are dealing with memory leak in your ASP.NET web application. Lately I've been involved with few of such situations. Here is the method and the resources that helped me to extremely quickly identify the root cause of the memory leaks.

Related Books
_________________________________

I am super grateful to the resources published by Tess and Rico that discuss the process of memory leak identification in more detail:

Customer Case Study

The customer complained his application gets less responsive as more subsequent request come in. The customer reported the high CPU utilization. At some point the application gets stuck and the only way to get it back is..... iisreset. The customer felt there is a memory leak in the application.

Analysis - Summary of steps

Step 1 - Use Process Explorer To Confirm The Memory Leak
Step 2 - Take Memory Dump Using adplus.vbs
Step 3 - Use Windbg to identify the leaking Type
Step 4 - Use Windbg to identify to identify who's holding on the Type
Step 5 - Find the culprit in the source code

Step 1 - Use Process Explorer To Confirm The Memory Leak

I am using Process Explorer to quickly identify whether we are dealing with memory leak. In our case it showed we are. This is the memory consumption that Process Explorer reveals in very convincing way - you can clearly see how the memory consumption climbs up until the application hangs/restarts:

Porcess Explorer Memory Leak

Next question is who's consuming the memory? Windbg to the rescue.

Step 2 - Take Memory Dump Using adplus.vbs

Use Windows Debugging Tools package to take a memory dump of the leaking managed process. Download the tools and extract on the server where the application runs. Open command prompt and change the directory to the folder that contains the tools, then type the following command:

adplus.vbs -quiet -hang -p <<PROCESSID>>

where <<PROCESSID>> is the process of w3wp.exe PID (process ID) that runs the leaking application. The result should be another folder inside of tools folder with very lengthy name. Open that folder- you should see a big size file with DMP extension. This is the memory dump you have just taken.

Step 3 - Use Windbg to identify the leaking Type

To analyze the dump file you will going to use Windbg.exe. The tool is located in the Debugging Tools for Windows folder, same folder that contains adplus.vbs that was used in the previous exercise. Open the tool, Windbg.exe. Press Ctrl+D, locate your DMP file and open it.

Now you need to load SOS.DLL extension to analyze managed memory dump. Type the following command according ro what .Net framework you are analyzing:

.load clr10\sos - for Net 1.X

.loadby sos mscorwks.dll - for Net 2.0 and up.

Type the following command to identify the Type that consumes most of the memory:

!dumpheap -stat

The result should be similar to the following:

.....
654088b4 92811 13736028 System.Data.DataColumn
000dec48 44 13900264 Free
654359c8 31704 32845344 System.Data.RBTree`1+Node[[System.Int32, mscorlib]][]
7912d8f8 320769 47859900 System.Object[]
790fd8c4 827939 186834312 System.String

Total 3351270 objects

Most consuming is that System.String type that is responsible for almost 190 MB of allocated memory.

Step 4 - Use Windbg to identify to identify who's holding on the Type

Next question your should ask is who's holding on it so that it's not collected. For that purpose type the following command in Windbg but be prepared to hit Ctrl+Break to stop it as you do not want to list all 827939 strings, you are looking for a few to get its addresses:

!dumpheap -type System.String

The output should be similar to this:

057f9364 790fd8c4 4116
057fa4b8 790fd8c4 4116
057fb52c 790fd8c4 4116
057fc68c 790fd8c4 4116
057fd700 790fd8c4 4116
0583fef8 790fd8c4 4116

Next you want to start trying to identify who's holding on the objects. That can be done using the following command (I replaced the real Namespaces with XXXXXXXX to keep customer's privacy):

0:000> !gcroot 0583fef8
Note: Roots found on stacks may be false positives. Run "!help gcroot" for
more info.
…
Scan Thread 27 OSTHread e40
DOMAIN(00102980):HANDLE(WeakLn):945e00:Root:19312e44(System.Web.Hosting.ISAPIAsyncCompletionCallback)->
190c2950(System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6)->
190c3138(System.Web.HttpContext)->
190c340c(System.Web.AspNetSynchronizationContext)->
0cbb80d0(ASP.global_asax)->
01b3a5b4(System.Web.HttpApplicationState)->
01b3a610(System.Collections.ArrayList)->
0a4e4c10(System.Object[])->
05846388(System.Collections.Specialized.NameObjectCollectionBase+NameObjectEntry)->
058331c8(XXXXXXXX.Web.UI.WebControls.XXXXXXXX.XXXXXXXX)->
058333c0(XXXXXXXX.Web.UI.WebControls.XXXXXXXX.Statuses)->
058333cc(System.Object[])->
0583fe20(XXXXXXXX.Web.UI.WebControls.XXXXXXXX.Status)->
0583fee0(System.Object[])->
05840f0c(XXXXXXXX.Web.UI.WebControls.XXXXXXXX.Style)->
0583fef8(System.String)

Looks like the code using endlessly Application object sticking strings into it. Once there it's never collected until the end of life of the Application which caused only by recycle.

Step 5 - Find the culprit in the source code

Next is to look in the source code all the usages of assigning strings to Application object. After quick look up we identified the following code that is called recursively:

HttpApplicationState app = HttpContext.Current.Application;
for (int i=0; i < m_XXXXXX.Count; i++)
{
    string sProcessId = m_XXXXXX[i].SelectSingleNode("@XXXXXX").Value;
    app.Set (GetPrefix() + XXXXX, new XXXXXX(m_XXXXXX[i].OuterXml));
}