Troubleshooting Memory Leaks With Just a Dump

Troubleshooting Memory Leaks With Just a Dump

Rate This
  • Comments 3

Hello debuggers, the debug ninja is back again.  Sometimes we have a scenario where a process is using a lot of memory, and the only data we are able to get at the moment is a user dump.  Ordinarily data from tools such as umdh or xperf would be preferable because they provide memory usage data over a period of time and can include call stack information. However, umdh requires restarting the process (which loses the state of high memory usage), and xperf requires the installation of the Windows Performance Toolkit which may not always be an immediate option.

 

When we have such a dump we may not be able to specifically identify what piece of code is generating the high memory usage, but we may be able to narrow the scope of troubleshooting to a specific dll.

 

The first thing we need to do is identify what type of memory is using most of the address space.  The debugger command !address –summary allows us to do this: 

0:000> !address -summary

 

--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal

Free                                    489      7fe`6ff5a000 (   7.994 Tb)           99.92%

Heap                                   9094        1`75ed1000 (   5.843 Gb)  93.47%    0.07%

<unknown>                               275        0`12e41000 ( 302.254 Mb)  4.72%    0.00%

Image                                   937        0`05a6a000 (  90.414 Mb)  1.41%    0.00%

Stack                                   138        0`01700000 (  23.000 Mb)  0.36%    0.00%

Other                                    14        0`001bd000 (   1.738 Mb)  0.03%    0.00%

TEB                                      46        0`0005c000 ( 368.000 kb)   0.01%   0.00%

PEB                                       1        0`00001000 (   4.000 kb)  0.00%    0.00%

 

From this example we can see that most of the memory is used by heap.  A process will usually have multiple heaps, each created by a call to HeapCreate.  We can examine the size of each of these heaps with !heap –s: 

0:000> !heap -s

LFH Key                   : 0x0000006c1104d280

Termination on corruption : ENABLED

          Heap     Flags  Reserv  Commit  Virt  Free  List   UCR Virt  Lock  Fast

                            (k)     (k)   (k)     (k) length      blocks cont. heap

-------------------------------------------------------------------------------------

0000000000100000 00000002  16384  12824  16384  1180   254     5   0      3   LFH

0000000000010000 00008000     64      4     64     1     1     1   0      0     

00000000003d0000 00001002   1088    708   1088   121    20     2   0      0   LFH

0000000003080000 00001002   1536    700   1536     4     4     2   0      0   LFH

00000000033a0000 00001002 5229696 1377584 5229696 414244  4039  3059   0     2c   LFH

    External fragmentation  30 % (4039 free blocks)

    Virtual address fragmentation  73 % (3059 uncommited ranges)

0000000003380000 00001002     64      8     64     3     1     1   0      0     

0000000003600000 00001002    512     56    512     3     1     1   0      0     

0000000003c20000 00001002    512      8    512     3     1     1   0      0     

0000000003220000 00001002    512      8    512     3     1     1   0      0     

0000000003e50000 00001002    512      8    512     3     1     1   0      0     

0000000003d00000 00001002    512    148    512     5     3     1   0      0   LFH

 

From the above output we can see that most of the memory is being used by heap 00000000033a0000.

 

At this point we need to try to identify what this heap is used for.  A brute force method to do this is to search memory with the ‘s’ command. 

0:000> s -q 0 l?7fffffffffffffff 00000000033a0000

<snip>

000007fe`f21810a0  00000000`033a0000 00000000`00000001

 

The output of the ‘s’ command may be verbose.  You will need to manually examine the addresses where ‘s’ finds hits.  Most of these addresses will probably be in heap memory, we are looking for an address that matches a module.  I snipped the above output to just show the relevant hit, an address inside of a loaded module.

 

The search of memory revealed that the heap 00000000033a0000 is used by the module useheap.dll, specifically it is part of the global ‘Blob’ class. 

0:000> ln 000007fe`f21810a0

(000007fe`f21810a0)  useheap!Blob::m_Heap

 

At this point we don’t know specifically what code in useheap.dll has allocated a lot of heap, however we have significantly narrowed the scope of the problem.  We can now determine if there is a known issue with heap usage in useheap.dll that is addressed in a later version.  We may also know from experience that this module uses a lot of memory under specific circumstances, such as a high volume of work sent to this service.

 

I hope this example helps the next time you have high memory usage and only have a user dump to troubleshoot with.  Good luck!

Leave a Comment
  • Please add 3 and 7 and type the answer here:
  • Post
  • A more efficient way, if you have private symbols, is

     x *!*

    And search that output for 33a0000

    Which would give you

    Ushheap!blob::m_heap = 0x33a0000

  • Very often heap handles are stored in a global variables and got "heap" at the end of the variable name (like MS CRT's "_crtheap"). I usually try to locate such global variables by running "x *!_crtheap" or "x *!*_heap" or even "x *!*heap" or "x *!*heap*". There can be a lot of matches including function names but gloabl heap variables are usually visible. Of course this will not work if there are no symbols available.

  • kd> !address summary

    *** ERROR: Module load completed but symbols could not be loaded for vmci.sys

    *** ERROR: Module load completed but symbols could not be loaded for pvscsi.sys

    *** ERROR: Module load completed but symbols could not be loaded for intelide.sys

    *** ERROR: Module load completed but symbols could not be loaded for spldr.sys

    *** ERROR: Module load completed but symbols could not be loaded for vmxnet3n61x64.sys

    *** ERROR: Module load completed but symbols could not be loaded for intelppm.sys

    *** ERROR: Module load completed but symbols could not be loaded for vm3dmp.sys

    *** ERROR: Module load completed but symbols could not be loaded for tmtdi.sys

    *** ERROR: Module load completed but symbols could not be loaded for vmmouse.sys

    *** ERROR: Module load completed but symbols could not be loaded for dump_pvscsi.sys

    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for VSApiNt.sys -

    Let me know why i am getting above errors.

     

    [Hi Manoj.  The command to use is !address -summary .  If you leave off the '-' the debugger will attempt to resolve the word "summary" as a symbol, leading to the symbol load failures you are seeing.]

Page 1 of 1 (3 items)