Real world problems; Real world solutions : Debugging

manually Debugging native memory leaks

Sudeep Gopalakrishnan — Fri, 22 May 2009 11:01:23 GMT

So in my previous posts, I explained how you can use DebugDiag tool to capture high memory dumps with leak tracking enabled and also how to use the inbuilt memory analysis scripts to get a report of memory usage. In this post, I discuss how you can do things manually using Debugging tools for Windows or Windbg. Again, I have tried to provide a generic approach, but with an example. It doesn’t apply to each & every situation.

So I have a memory dump which is about 500 MB in size and was captured when web applications started throwing out of memory errors. The first thing to do find out is where most of the memory is. I discussed this a bit in one of my earlier blog posts.

        1: 0:000> !address -summary

       2:  

       3: -------------------- Usage SUMMARY --------------------------

       4:     TotSize (      KB)   Pct(Tots) Pct(Busy)   Usage

       5:     1806000 (   24600) : 01.17%    01.19%    : RegionUsageIsVAD

       6:     14f3000 (   21452) : 01.02%    00.00%    : RegionUsageFree

       7:     23e9000 (   36772) : 01.75%    01.77%    : RegionUsageImage

       8:     2200000 (   34816) : 01.66%    01.68%    : RegionUsageStack

       9:       88000 (     544) : 00.03%    00.03%    : RegionUsageTeb

      10:    78c82000 ( 1978888) : 94.36%    95.34%    : RegionUsageHeap

      11:           0 (       0) : 00.00%    00.00%    : RegionUsagePageHeap

      12:        1000 (       4) : 00.00%    00.00%    : RegionUsagePeb

      13:        1000 (       4) : 00.00%    00.00%    : RegionUsageProcessParametrs

      14:        2000 (       8) : 00.00%    00.00%    : RegionUsageEnvironmentBlock

      15:        Tot: 7fff0000 (2097088 KB) Busy: 7eafd000 (2075636 KB)

      16:  

      17: Largest free region: Base 57818000 - Size 00068000 (416 KB)

So from this output we can see that 94.36% of the entire virtual address space is in RegionUsageHeap, which means heap memory. We can also see the size – 1,978,888 KB or 1.88 GB! Remember I indicated a few moments back that our dump file itself is just 500 MB in size. So what this most likely means is that this value is the reserved memory vs. committed bytes plus other information that the dump file contains. We can also see that the largest contiguous free region is just 416 KB, which explains why this process ran into out of memory errors. There is just no large contiguous free block to satisfy allocation requests.

A process will have at least one heap, the default process heap which is created by the operating system for you when the process starts. This heap is used for allocating memory if no other heaps are created and used. Components loaded within the process can create their own heaps. For eg the C Runtime heap. Many of you will remember it as MSVCRT.dll – that’s our C Runtime library.

OK, so how many heaps and which heap has the most number of allocations? The trick I usually use is to look at all the heaps and check how many segments each heap has. I think the maximum number of segments a heap can have is 64. Segments are contiguous blocks of memory which hold smaller memory ranges of various sizes. These ranges are of various sizes are handed out to applications when they request memory. Thus, if a segment does not have enough memory to satisfy an allocation request, a new segment is created. The more number of segments, the more the chances are that it is our problem heap. Recommended reading. To view the segments, you can use the inbuilt Windbg extension command !heap

From this example: !heap 0

115:   02990000 <------------- Heap Handle
    Segment at 02990000 to 029d0000 (00030000 bytes committed)
    Segment at 0bc10000 to 0bd10000 (00037000 bytes committed)
    Segment at 0e350000 to 0e550000 (00007000 bytes committed)
    Segment at 15fe0000 to 163e0000 (00002000 bytes committed)
    Segment at 59530000 to 59d30000 (00001000 bytes committed)
    .
    .
    .
    Segment at 5e980000 to 5e997000 (00001000 bytes committed)
    Segment at 60040000 to 60057000 (00001000 bytes committed)
    Segment at 611e0000 to 611f7000 (00001000 bytes committed)


117:   02a10000 <------------- Heap Handle
    Segment at 02a10000 to 02a50000 (00040000 bytes committed)
    Segment at 0fc90000 to 0fd90000 (000b7000 bytes committed)
    Segment at 17640000 to 17840000 (0000e000 bytes committed)
    Segment at 21ba0000 to 21fa0000 (00001000 bytes committed)
    Segment at 58530000 to 58d30000 (00001000 bytes committed)
    Segment at 5e9c0000 to 5f9c0000 (00001000 bytes committed)
    .
    .
    .
    Segment at 7fe70000 to 7ff23000 (00001000 bytes committed)
    Segment at 23de0000 to 23e3a000 (00001000 bytes committed)
    Segment at 52770000 to 527ca000 (00001000 bytes committed)
    Segment at 52900000 to 5295b000 (00001000 bytes committed)
    Segment at 584c0000 to 5851b000 (00001000 bytes committed)
    Segment at 5a270000 to 5a2cb000 (00001000 bytes committed)

I have truncated the above entry for brevity, but essentially there were many segments. An easier way to see how many segments are there in a heap is to use the !heap command again with the –s switch (for statistics) followed by heap handle. Thus: !heap –s 02a10000

Take a look at lines #12 & # 13 in the following output.

   1: 0:000> !heap -s 02a10000

   2: Walking the heap 02a10000 .......................................................................

   3:  0: Heap 02a10000

   4:    Flags          00001003 - HEAP_NO_SERIALIZE HEAP_GROWABLE

   5:    Reserved memory in segments              184708 (k)

   6:    Commited memory in segments              18014398506966656 (k)

   7:    Virtual bytes (correction for large UCR) 1252 (k)

   8:    Free space                               254 (k) (45 blocks)

   9:    External fragmentation          0% (45 free blocks)

  10:    Virtual address fragmentation   201004% (77 uncommited ranges)

  11:    Virtual blocks  0 - total 0 KBytes

  12:    Lock contention 2989

  13:    Segments        64

  14:    896 hash table for the free list

  15:        Commits 0

  16:        Decommitts 0

17:

  18:                     Default heap   Front heap       Unused bytes

  19:    Range (bytes)     Busy  Free    Busy   Free     Total  Average

  20: ------------------------------------------------------------------

  21:      0 -   1024       64    142      0      0          0      0

  22:   1024 -   2048      279     24      0      0       2280      8

  23:   2048 -   3072       21      3      0      0        176      8

  24:   3072 -   4096        2      3      0      0         16      8

  25:   4096 -   5120       69      6      0      0        560      8

  26:   5120 -   6144        6      0      0      0         48      8

  27:   6144 -   7168       35      3      0      0        280      8

  28:   7168 -   8192        0      2      0      0          0      0

  29:   8192 -   9216        0      1      0      0          0      0

  30:   9216 -  10240        2      1      0      0         16      8

  31:  12288 -  13312        2      0      0      0         16      8

  32:  13312 -  14336        0      1      0      0          0      0

  33:  19456 -  20480        2      0      0      0         16      8

  34:  24576 -  25600        2      0      0      0         16      8

  35:  36864 -  37888        0      1      0      0          0      0

  36: ------------------------------------------------------------------

  37:   Total              484    187      0      0       3424      7

From the above output, you can also see the ranges of memory and their utilization. We can also obtain worst offender byte sizes and worst offender count size by using the –stat parameter of !heap command. Here’s the output.

   1: 0:000> !heap -stat -h 02a10000

   2:  heap @ 02a10000

   3: group-by: TOTSIZE max-display: 20

   4:     size     #blocks     total     ( %) (percent of total busy bytes)

   5:     1008 45 - 45228  (26.78) <----------- Worst offender Bytes (WOB)

   6:     19f8 21 - 358f8  (20.75)

   7:     418 c6 - 32a90  (19.62)

   8:     5e8 21 - c2e8  (4.72)

   9:     6008 2 - c010  (4.65)

  10:     5a0 21 - b9a0  (4.49)

  11:     4c08 2 - 9810  (3.68)

  12:     1440 6 - 7980  (2.94)

  13:     808 d - 6868  (2.53)

  14:     3008 2 - 6010  (2.33)

  15:     2708 2 - 4e10  (1.89)

  16:     1808 2 - 3010  (1.16)

  17:     a18 4 - 2860  (0.98)

  18:     6c0 5 - 21c0  (0.82)

  19:     408 6 - 1830  (0.59)

  20:     c08 2 - 1810  (0.58)

  21:     ac0 2 - 1580  (0.52)

  22:     a90 2 - 1520  (0.51)

  23:     778 2 - ef0  (0.36)

  24:     450 1 - 450  (0.10)

All values are in hex in the above output except the percent column. So from the above output (line # 5) we can say that worst offender bytes [WOB - allocation size that is using the most bytes in the heap] is 0x1008 Bytes [4,104 Bytes or 4K] and it adds up to a total of 0x45228 Bytes [283,176 Bytes or 276 KB]

Similarly, you could group by block size if you want to figure the worst offender count size [WOC - allocation size that has the most duplicates in the heap] and count of worst offender count by using the –grp switch.

   1: 0:000> !heap -stat -h 02a10000 -grp B

   2:  heap @ 02a10000

   3: group-by: BLOCKCOUNT max-display: 20

   4:     size     #blocks    total     ( %) (percent of totalblocks)

   5:     418 c6 - 32a90  (47.26)  <----------- Worst offender count (WOC)

   6:     1008 45 - 45228  (16.47)

   7:     19f8 21 - 358f8  (7.88)

   8:     5e8 21 - c2e8  (7.88)

   9:     5a0 21 - b9a0  (7.88)

  10:     808 d - 6868  (3.10)

  11:     1440 6 - 7980  (1.43)

  12:     408 6 - 1830  (1.43)

  13:     6c0 5 - 21c0  (1.19)

  14:     a18 4 - 2860  (0.95)

  15:     6008 2 - c010  (0.48)

  16:     4c08 2 - 9810  (0.48)

  17:     3008 2 - 6010  (0.48)

  18:     2708 2 - 4e10  (0.48)

  19:     1808 2 - 3010  (0.48)

  20:     c08 2 - 1810  (0.48)

  21:     ac0 2 - 1580  (0.48)

  22:     a90 2 - 1520  (0.48)

  23:     778 2 - ef0  (0.48)

  24:     450 1 - 450  (0.24)

Thus in this case, the most duplicates are of allocation size 0x418 bytes [1048 Bytes] and there are 0xc6 [196] of them. You could also dump the allocations in the 1 K – 4 K range and then dump out the contents using the address value in UserPtr column. To do that execute: dc <address value in UserPtr column>

Warning: This command can generate a huge output as it dumps allocations in the specified range from all heaps.

   1: !heap -flt r 418 1008

   2:     _HEAP @ 2a10000

   3:       HEAP_ENTRY Size Prev Flags    UserPtr UserSize - state

   4:         02a14a58 0084 00f0  [01]   02a14a60    00418 - (busy)

   5:         02a15000 0084 0084  [01]   02a15008    00418 - (busy)

   6:         02a20330 0144 0102  [01]   02a20338    00a18 - (busy)

   7:         02a25a40 0102 0144  [01]   02a25a48    00808 - (busy)

   8:         02a265f0 00be 0102  [01]   02a265f8    005e8 - (busy)

   9:         02a2d1b0 00b5 00be  [01]   02a2d1b8    005a0 - (busy)

  10:         5e9c0040 0084 00b6  [00]   5e9c0048    00418 - (free)

  11:         7c1d0040 0084 0084  [00]   7c1d0048    00418 - (free)

  12:     .

  13:     .

14:

  15: dc 02a14a60

So our story so far…

There are a couple of heaps that have lots of segments.
A lot of memory is reserved – in MBs and there are few committed blocks – in KBs.
The allocations appear to be in the range of 1 KB to 4 KB.

Next questions: So what are these heaps and who is allocating here?

If you want to see the stack back trace for the allocation, you can dump out the page heap information for a given address [UserPtr], but stack back trace is only displayed when available. If I remember correctly, it is available when page heap is enabled for the process.

   1: 0:000> !heap -p -a 7c1d0048

   2:     address 7c1d0048 found in

   3:     _HEAP @ 2a10000

   4:       HEAP_ENTRY Size Prev Flags    UserPtr UserSize - state

   5:         7c1d0040 0084 0000  [00]   7c1d0048    00418 - (free)

   6:         Trace: 0025

   7:         7c96d6dc ntdll!RtlDebugAllocateHeap+0x000000e1

   8:         7c949d18 ntdll!RtlAllocateHeapSlowly+0x00000044

   9:         7c91b298 ntdll!RtlAllocateHeap+0x00000e64

  10:         102c103e MSVCR90D!_heap_alloc_base+0x0000005e

  11:         102cfd76 MSVCR90D!_heap_alloc_dbg_impl+0x000001f6

  12:         102cfb2f MSVCR90D!_nh_malloc_dbg_impl+0x0000001f

  13:         102cfadc MSVCR90D!_nh_malloc_dbg+0x0000002c

  14:         102db25b MSVCR90D!malloc+0x0000001b

  15:         102bd691 MSVCR90D!operator new+0x00000011

  16:         102bd71f MSVCR90D!operator new[]+0x0000000f

  17:         4113d8 MyModule1!AllocateMemory+0x00000028

  18:         41145c MyModule1!wmain+0x0000002c

  19:         411a08 MyModule1!__tmainCRTStartup+0x000001a8

  20:         41184f MyModule1!wmainCRTStartup+0x0000000f

  21:         7c816fd7 kernel32!BaseProcessStart+0x00000023

The above output is just an example but you get the idea of how you can use this technique to help track the source of leaks in your application.

When memory at a given address is de-allocated, the heap manager checks how many contiguous bytes are free around that address. After that check is complete, the heap manager can do one of two things:

Keep the contiguous memory block committed.
De-commit the contiguous memory block and mark it as reserved only.

There is a registry key that controls the de-commit behavior. That key is:

Path: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager
Key: HeapDeCommitFreeBlockThreshold
Value Type: Reg_DWORD

For sake of completing this blog post, adjusting the value for this registry key was the resolution in my example. It could be something else in your case depending on the circumstances under which this occurs. Once a software developer has enough information about the pattern and source of the memory consumption, he will be able to recommend changes or make a suitable fix to resolve the issue.

In this blog post, I made an attempt to show how you can track down native memory leaks manually vs. using DebugDiag scripts as discussed in this blog post. Again, this doesn’t apply to every situation as there are umpteen possibilities to the cause of a leak. Hopefully this blog post is a good starter and a future reference.

Using debugdiag for memory pressure analysis

Sudeep Gopalakrishnan — Thu, 21 May 2009 09:00:39 GMT

In my last post, I discussed a generic approach to collecting memory dumps using Debug Diagnostics tool. In this post, I discuss how to use DebugDiag’s memory pressure scripts. Please note that the current version of DebugDiag does not have the ability to look up .NET heaps and draw conclusions. For .NET debugging, the best resources are the following blogs:

Tess’s Blog

Doug's Blog

Step 1: Capture a high memory dump as discussed in this post.

Step 2: Start Debug Diagnostics Tool. If prompted to select a rule, click Cancel.

Step 3: Select the Analysis Tab and select the memory pressure analysis scripts

Step 4: Add the dump files for analysis

Step 5: Start analysis

Wait for DebugDiag to finish. DebugDiag will automatically connect to the Microsoft Public symbol server, download and cache symbols on your local drive for analysis. You can also add your custom symbol stores and the location where you want to cache the symbols using the Tools, Options & Settings dialog box.

Have fun!

debugging native memory leaks

Sudeep Gopalakrishnan — Tue, 19 May 2009 00:05:16 GMT

Debugging native memory leaks is one of the most difficult things to do - (at least for me). There are a few Escalation Engineers at Microsoft Product Support Services who are extremely good at debugging all kinds of issues. I learn a lot from these guys whenever I get an opportunity.

In this blog post, I am not going to talk about a specific issue, but rather a general approach to debugging native memory leaks. I work in the IIS/ASP support group and therefore some things I discuss may be more IIS/ASP specific at times.

To solve the problems of common debugging issues, Escalation Engineers in the IIS support group created a fantastic tool called Debug Diagnostics Tool. This link points you to to the 32 bit (x86 version). To obtain the 64 bit (x64) version, you need to call Microsoft Product Support at this time. What this tool allows you to do is inject a module called Leaktrack.dll into the target process so that it starts collecting allocation/de-allocation information. The concept is simple - create a heap where you track allocations from various memory managers. It works by hooking into the known Windows memory managers NTDLL, MSVCRT etc.

How it works

When a module makes an allocation request, it increments the count and also gets the size of allocation and also maintains a total size of allocation. When a de-allocation request is made by the same component, it reduces the count and updates the totals. For this to work effectively, you must inject leaktrack soon after you start the process. When the process has consumed memory in the upwards of 700 MB, you can dump out the process and then run Debug Diag’s inbuilt memory pressure analysis scripts against that dump file. Debug Diag is so cool that it will connect to the public Microsoft symbol server, download the symbols, analyze and create a nice report about the memory allocations and components responsible for those allocations. It is very easy to determine issues related to memory leaks & fragmentation with DebugDiag script. DebugDiag is very effective against issues in web applications hosted in IIS worker processes because it uses heuristics and is accurate many times. Below are the screen shots on how to setup a leak rule in Debug Diag.

NOTE: If you are debugging a web application hosted in IIS that is leaking memory, before you setup a memory leak rule, restart IIS and then send the first request to the application. This is to start tracking from the beginning of the life of the process and also to start the IIS worker process.

Step 1: Open Debug Diagnostics Tool

Step 2: If prompted to select a rule, select Memory & Handle Leak OR click Add Rule button to get to this screen

Step 3: Click Next to get to the Select Target Screen. Then select w3wp.exe if debugging IIS process or the process that you wish to debug. If you see multiple worker processes & is not sure which w3wp.exe instance to select, run the following command from a command prompt running as Administrator

CScript %windir%\system32\iisapp.vbs

The above script will output the IIS web application pool name and its corresponding PID value that you can use below.

Step 4: Click Next, then click On the Configure button

Step 5: Setup the rules as follows

Generate a userdump when private bytes reach - (Enter value)
And each additional 100 MB thereafter.
Auto-create a crash rule to get userdump on unexpected process exit

Step 6: Click Save & Close and then the Next Button from the previous screen.

Step 7: Type in any name that you like for the rule and also type in the path where you want the dumps to be generated. This drive must have lots of disk space as each dump file will be equal to the size of the process when the dump is captured. So since we are capturing it at 800 MB upwards here as in this example, this will create 10 dumps (by default) of 800 MB or higher each.

Step 8: Finish up the rule and activate it. Then make sure you see the information screen like below

You are done! You can see the rules that you just configured in the rules window. When a dump is captured, the userdump count column will have a value of 1 or more.

Next Post: Using Analysis Scripts.

How to get the http headers for asp.net requests from a dump file

Sudeep Gopalakrishnan — Tue, 10 Mar 2009 21:06:45 GMT

So, you have a managed dump and you want to find out the request headers. Here’s one of the methods I use to find this information. I use it especially when I want to view the session ID or cookies.

Load SOS
Run !aspxpages and note down the HttpContext address you are interested in. Eg:

0x120bea74    54000 Sec       yes                       XXX        200   GET /default.aspx
0x14244228    54000 Sec        no       284 Sec     XXX        200   GET /default.aspx
0x1426f4c0     54000 Sec        no       101 Sec     XXX        200   GET /default.aspx
0x163a3038    54000 Sec        no       193 Sec     XXX        200   GET /default.aspx
0x1e0c360c    54000 Sec        no       376 Sec      39          200   GET /default.aspx

Dump the HttpContext noted in Step 2: !do 0x1426f4c0
Dump the _wr field which is HttpWorkerRequest object.
Dump the _basicServerVars field: !dumparray <address of _basicServerVars>
Dump the last entry: !do <address of last entry in the list>
You should get the output like:

String: Connection: Keep-Alive
Cookie: ASP.NET_SessionId=5kwvjlzd3ksgii45ephn00aq; HTTP_REFERER::6841=
Host: Skyraider
User-Agent: DebugDiag Service HTTP Pinger

Hopefully this is what you wanted.

setting managed breakpoints in windbg

Sudeep Gopalakrishnan — Fri, 27 Feb 2009 11:28:52 GMT

Windbg is a native debugger and you can use it to set a breakpoint on a virtual address. Any managed code running within the process wouldn’t have a virtual address associated with it until it is JIT compiled. Thus setting a breakpoint on a managed function is a bit tricky in Windbg. You can set a breakpoint on managed methods using windbg only:

When you are performing a live debug & not on a post mortem dump file.
You have a .RUN file from a Time Travel Debug trace.

When I started learning how to set managed breakpoints, one of the first questions I had is: How to set a breakpoint on a specific line of code in a managed method – because that is what we usually do in other IDE environments like Visual Studio. This is somewhat very difficult to do because, though you can get the virtual address where your method starts using the SOS commands, you will need to know the exact offset from the method’s starting virtual address [The actual address which corresponds to your line of code] and it isn’t easy at all to co-relate that to your source code. You will need to have an extremely good understanding of IL code, un-assemble the function using !u command and then set the breakpoint on that address. I do not have that skill yet, but will surely put out a post once I figure that out. So over here, I will describe how to set a breakpoint on a managed method for .NET Framework 2.0.

STEP 1: So assuming you are doing a live debug, the first step is to attach to the process that you want to debug. You can use the attach option in Windbg user interface [File menu]. Then load the SOS debugger extension - !loadby SOS mscorwks

STEP 2: You need to know which method you want to set a breakpoint on. The SOS command you need is !dumpmt with the –md parameter. This lists out the method table. For example, Dump the method table of System.Timespan

        1: !dumpmt -md 0x7911228c 

       2: EEClass: 791121e4 

       3: Module: 790c2000 

       4: Name: System.TimeSpan 

       5: mdToken: 02000114  (C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll) 

       6: BaseSize: 0x10 

       7: ComponentSize: 0x0 

       8: Number of IFaces in IFaceMap: 3 

       9: Slots in VTable: 56 

      10: -------------------------------------- 

      11: MethodDesc Table 

      12:    Entry MethodDesc      JIT Name 

      13: 796d2710   7914fb28     NONE System.TimeSpan.ToString() 

      14: 793624d0   7914b950   PreJIT System.Object.Finalize() 

      15: 796c07f8   7914fb08     NONE System.TimeSpan.CompareTo(System.TimeSpan) 

      16: 796d2708   7914fb18     NONE System.TimeSpan.Equals(System.TimeSpan) 

      17: 79381054   79266eb8   PreJIT System.TimeSpan..ctor(Int64) 

      18: 7939f058   79266ec0   PreJIT System.TimeSpan..ctor(Int32, Int32, Int32) 

      19: 7939f07c   79266ed8   PreJIT System.TimeSpan.get_Ticks() 

      20: 794002c8   79266ee0   PreJIT System.TimeSpan.get_Days() 

      21: 794002e8   79266ee8   PreJIT System.TimeSpan.get_Hours() 

      22: 79400328   79266ef0   PreJIT System.TimeSpan.get_Milliseconds() 

      23: 7940036c   79266ef8   PreJIT System.TimeSpan.get_Minutes() 

      24: 794003ac   79266f00   PreJIT System.TimeSpan.get_Seconds() 

      25: 794003ec   79266f08   PreJIT System.TimeSpan.get_TotalDays() 

      26: 7940040c   79266f10   PreJIT System.TimeSpan.get_TotalHours() 

      27: 79380c10   79266f18   PreJIT System.TimeSpan.get_TotalMilliseconds()

      28:

STEP 3: [Optional] Using the method descriptor command, !dumpmd you can view if the code is JITted. See line #7 below. You can skip this and go to STEP 4 directly using the corresponding MethodDesc value from the previous output.

       1: !dumpmd 79266f18 

       2: Method Name: System.TimeSpan.get_TotalMilliseconds() 

       3: Class: 791121e4 

       4: MethodTable: 7911228c 

       5: mdToken: 0600101e 

       6: Module: 790c2000 

       7: IsJitted: yes 

       8: m_CodeOrIL: 79380c10

STEP 4: Add the breakpoint using !bpmd –md command.

       1: !bpmd –md 79380c10
  

Another way…

Syntax: !bpmd <ModuleName> <FunctionName>

Example: !bpmd mscorlib.dll System.TimeSpan.get_TotalMilliseconds

Notes

The method names are case sensitive.
In many cases, the breakpoints you set may be indicated as “Pending breakpoints”, which is normal, because your method may not yet be JITted.

Once your breakpoints are set, you can execute the g command to let the process execute till it hits the breakpoint. Once it hits the breakpoint you can do other tasks like examine callstacks, stack objects, local variables etc.

Have fun!

catching a memory dump on system.outofmemoryexception

Sudeep Gopalakrishnan — Thu, 26 Feb 2009 22:28:19 GMT

Issues related to high memory utilization on an IIS application server are common. With .NET there is a little misconception that the Garbage Collector (GC) will clean up objects and therefore the process can never run out of memory. This isn’t true. GC will never clean up an object which is in use. If that was the case, you can imagine the kind of problems it would create.

While debugging memory problems, it is a good idea to capture memory dump when the process memory consumption is at its peak maximum usage. For .NET applications, a System.OutOfMemoryException is thrown when GC fails on a VirtualAlloc().

So how do we capture a memory dump when this Exception is thrown? Here’s how.

For .NET Framework version 1.1

Open the registry path: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework

Key: GCFailFastOnOOM

Type: DWORD

Value: 2

For .NET Framework version 2.0 and above

Open the registry path: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework

Key: GCBreakOnOOM

Type: DWORD

Value: 2

Setting the above key causes a DebugBreak within the process when a System.OutOfMemoryException is encountered. You can then use a tool like DebugDiag or a Debugger like WinDBG/CDB/NTSD to capture a dump on this DebugBreak Exception. Windbg/CDB/NTSD debuggers are for advanced users and DebugDiag is generally preferred due to ease of use and is designed to be used in production environments.

Configuring Debug Diagnostic Tool

Download and install DebugDiag to a drive with at least 4-5 GB of disk space.
Open DebugDiag. If prompted to select a rule, select Crash. Else click on Add Rule button and select Crash.
Click Next & select “A specific process”
Select the process name. For IIS 6.0, this will be w3wp.exe. Click Next
Under Advanced Settings, click on Exceptions, then click on Add Exception
From the list of exceptions, select 80000003 Breakpoint Exception
Set Action Type to Full userdump & Action limit to 1. Click OK
Click Save & Close button
Click Next and provide a name for the rule and location where the dump files must be saved.
Click Next and then Finish button.

NOTE: When the dump is captured, the Userdump count column will be incremented by 1. You can then do post mortem debugging using Windbg and SOS.

debugging memory problems – a beginner’s view

Sudeep Gopalakrishnan — Sun, 22 Feb 2009 12:55:51 GMT

Unlike native debugging, you don’t need symbols for debugging managed code. SOS.dll can also be used with WinDBG and Visual Studio debugger. Simply use the intermediate window in Visual Studio to load SOS.dll and then you can use the commands provided by the extension.

My favorite blog for managed debugging (and almost everyone else’s) is by Tess Ferrandez-Norlander. Tess is an Escalation Engineer with Microsoft. I don’t think anyone else explained managed debugging better than Tess. It is a good place to start with the basics & I highly recommend reading her blog. Her blog not only teaches the techniques of debugging, but also contains code examples that illustrate various problems. The comments section of each post also has very valuable information.

So here, I explain how I go about looking at high memory dumps:

Big Question: I always have a question in my mind: Where is most of the memory? Then start looking for it. Once I identify what all components are majorly contributing to memory usage, I check to see if the problem can be resolved with a hotfix. While this usually requires identifying patterns and some experience knowledge, for beginners, it is still worth checking the Microsoft support site to see if the problem you are running into is described in a KB article and if a hotfix is available. The easiest way is to stay updated because Microsoft continuously investigates such issues and provides hotfixes. If the problem is traced down to application code, the developers should work to reduce memory usage.

First step - Open the dump in WinDBG. For instructions on setting up WinDBG, please see this post.

First, we need to find out where “most” of the memory is. A debugger command is available that will help us here – !address –summary. To effectively understand and interpret what the output means we need to first understand what the output values represent. Take a look at this MSDN article. In the output of !address –summary command, the column named “KB” indicates in KiloBytes, the amount of memory in each of these “areas” and the Pct(Tots) column indicates a percentage of the value in KB column against Total address space (4 GB on 32 bit).

Thus, if RegionUsageIsVAD is high, it indicates that most memory is in virtual allocations. .NET heaps are made with calls to VirtualAlloc. The GC in .NET uses the Microsoft Win32® VirtualAlloc() application programming interface (API) to reserve a block of memory for its heap. Depending on the flavor of GC in use, it may be 32 MB (Workstation GC) allocations or 64 MB (Server GC). For ASP.net applications running in IIS, by default, the Server GC is automatically selected, if the computer has 2 or more processors.

Similarly, RegionUsageFree indicates most of memory is “Free”. If applications start throwing out of memory exceptions and a captured dump shows most of memory is Free, then it indicates fragmentation within the process address space. Typically if 30% or more of memory is Free, then I’d suggest investigating why memory is fragmented.

If RegionUsageImage is high, then in means you have lots of DLLs being loaded in the process. Ideally 100 MB or less is good. ASP.NET applications should not have debug attribute set to TRUE in web.config. Enabling debugging emits debug information which can be an overhead and cause memory problems. Similarly, there should not be any modules that are built in debug mode.

If RegionUsageHeap is high, it means most of the memory is on heaps. Eg the C Runtime heap, MDAC (Microsoft Data Access Components) heap, heaps used for compression etc. When you use the MSVCRT.dll library, it internally ends up allocating on the C runtime heap by calls to the CAlloc API.

Note that there are many tools out there (not just winDBG) that allows you to identify the allocation profiles of applications. Eg: For .NET applications you can use CLRProfiler (Microsoft) and Ant’s Profiler (RedGate). I do not know about profilers for native C, C++ applications. However Debug Diagnostics Tool v1.1 from Microsoft includes a capability for injecting a DLL – Leaktrack into the process that hooks into the calls like CAlloc and can keep track of who made the allocations. DebugDiag also includes a Memory pressure analysis script that can read a dump with leak track injected and give you a very nice report on where the allocations are – By size, by count and many other parameters. It’s 99% helpful in debugging web applications hosted on IIS. I will discuss this in another blog post.

WinDBG is usually used in post mortem debugging. Profilers should be used only in development and test environments, not production. They are very invasive and will reduce performance by about 10 times.

Continuing a bit more about RegionUsageIsVAD… If you determine that most memory is from Virtual Allocations AND you have ASP.net web applications running in IIS, then almost always it is likely that large amounts of memory is allocated in managed heaps. You will then need to investigate into .NET heaps to determine the top memory consumers. That’s where the !dumpheap –stat command comes into the picture.

installing & configuring debugging tools for windows

Sudeep Gopalakrishnan — Mon, 09 Feb 2009 13:32:50 GMT

There are umpteen blogs out there that discuss various debugging techniques. So what is so different about this one? Nothing really. It’s just something that I am adding as a reference for myself and for any soul out there that is starting out with debugging. As part of my job, I need to start learning debugging and that’s the whole reason why I am writing this out. Writing this out also helps me remember things better.

Everyone knows what Debugging Tools For Windows is. Also known as Windbg, it is a very powerful debugger that you can use to analyze memory dumps (.DMP files) for user more and kernel mode processes, from any Windows environment. Debugging skills are essential for any serious developer & any serious support professional. There is native debugging and .NET (Managed) debugging, which I will discuss later. So let’s get started.

So what type of machine do you need? Any Windows XP or later PC with at least 2 GB of RAM is what I recommend. The more, the better.

Installing Debugging tools for Windows (x86). Download from this Microsoft site

I recommend installing using the typical options. The install isn’t very big, but the symbols (aka .PDB files) will take up space on your hard disk. For those who do not know what symbols are & why they are important, please jump to this URL: PDB Files

You need symbols only if you are performing native debugging – Typical C, C++, Windows SDK programs. .NET debugging doesn’t require symbols.

Create a folder on a hard drive with at least 2-4 Giga bytes of disk space for your symbol store. The symbol store can be in any location. You only need to tell the debugger where the symbols are. I usually create a folder named Symbols with 2 sub folders. Private & Public. Private symbols are PDB files that are proprietary to your company. Public symbols contain lesser information so that others cannot re-engineer products. Also, you shouldn’t mix private & public symbols as it may confuse the debugger. However, you can put x86 and x64 symbols in the same folder.

Fire up Windbg from the Start menu. (Under Debugging tools for Windows folder).
Select File menu and then Symbol File path. Alternatively, press CTRL+S.
Enter the path to your symbol store as follows: SRV*D:\symbols\Public*http://msdl.microsoft.com/download/symbols
Click on File menu and then select Save Workspace.

NOTE: SRV is a keyword here. Each parameter is separated with a *. The internet address provided above is the address of Microsoft’s public symbol server. Yes! Microsoft provides public symbols for most of its products so that others can debug problems with their products.

NOTE: If you want to further customize the appearance of Windbg, click on View menu and then Option… Do not forget to click on File menu and select Save Workspace after making the changes. The reason for this is, it saves all your settings for the next debug session, or else you will need to enter the symbol path again. You can also use the _NT_SYMBOL_PATH to configure where the debugger should look for symbols. For more information, see this KB Article: Using Microsoft Symbol Server

You are all set to debug now.