So let's continue our digging into Memory problems and how to track down what is happening.  So our last post, High Memory continued, went into how to look into the managed heap and if the problem is a System.Data.DataTable, we can look at it and see what the columns are.

So what if the memory isn't all in the Virtual Memory.  What if it is in the native NT Heap?  In that case, !address -summary may look like:

 

0:000> !address -summary

-------------------- Usage SUMMARY --------------------------
    TotSize (      KB)   Pct(Tots) Pct(Busy)   Usage
    41c2000 (   67336) : 03.21%    04.95%    : RegionUsageIsVAD
   2cf36000 (  736472) : 35.12%    00.00%    : RegionUsageFree
    3d0b000 (   62508) : 02.98%    04.59%    : RegionUsageImage
     d80000 (   13824) : 00.66%    01.02%    : RegionUsageStack
      36000 (     216) : 00.01%    00.02%    : RegionUsageTeb
   4a434000 ( 1216720) : 58.02%    89.42%    : RegionUsageHeap
          0 (       0) : 00.00%    00.00%    : RegionUsagePageHeap
       1000 (       4) : 00.00%    00.00%    : RegionUsagePeb
       1000 (       4) : 00.00%    00.00%    : RegionUsageProcessParametrs
       1000 (       4) : 00.00%    00.00%    : RegionUsageEnvironmentBlock
       Tot: 7fff0000 (2097088 KB) Busy: 530ba000 (1360616 KB)

-------------------- Type SUMMARY --------------------------
    TotSize (      KB)   Pct(Tots)  Usage
   2cf36000 (  736472) : 35.12%   : 
    3d0b000 (   62508) : 02.98%   : MEM_IMAGE
     c10000 (   12352) : 00.59%   : MEM_MAPPED
   4e79f000 ( 1285756) : 61.31%   : MEM_PRIVATE

-------------------- State SUMMARY --------------------------
    TotSize (      KB)   Pct(Tots)  Usage
   3ffd2000 ( 1048392) : 49.99%   : MEM_COMMIT
   2cf36000 (  736472) : 35.12%   : MEM_FREE
   130e8000 (  312224) : 14.89%   : MEM_RESERVE

Largest free region: Base 685ba000 - Size 04b36000 (77016 KB)

 

From here we can see that most of the memory now is native.  The best way to find out what is causing all of these memory to be allocated is to use a tool from the IIS team called DebugDiag.  When you run this, you can use the Memory and Handle Leak rule to have DebugDiag track native memory and try to help you find out what is causing the high memory.

After you have run the test under DebugDiag, you load the dump file into DebugDiag and run the analysis report Memory Pressure Analyzers.  When you look at the report it creates, you will see some summary information about memory, such as:

 

mscorwks.dll (a known Windows memory manager) is responsible for 761.87
MBytes worth of outstanding allocations. These allocations appear
to have originated from the following module(s) and function(s):

System.Threading._TimerCallback.TimerCallback_Context(System.Object)

 

From this we can click on the name of the DLL and jump to the report for the file.  The main part to look at are the Top 5 functions by allocation count and Top 5 functions by allocation size

 

Top 5 functions by allocation size
mscorwks!EEHeapAlloc+12d               454.89 MBytes 
mscorwks!EEvirtualAlloc+104            306.94 MBytes 
mscorwks!DebuggerHeap::Alloc+2f         39.16 KBytes 
mscorwks!CExecutionEngine::CheckThreadState+14d   288 Bytes  
mscorwks!NLSTable::OpenOrCreateMemoryMapping+120    0 Bytes 

 

Each function, if we click on it, will give us a Leak Probability and also is then further broken down to show the Top 10 allocation sizes by allocation count and Top 10 allocation sizes by total size for each of these functions.  So we will get something like:

 

Top 10 allocation sizes by allocation count
24 Bytes         44,766 allocation(s) 
28 Bytes          8,973 allocation(s) 
40 Bytes          8,404 allocation(s) 
20 Bytes          7,388 allocation(s) 
16 Bytes          6,761 allocation(s) 
64 Bytes          5,584 allocation(s) 
5.26 KBytes       5,580 allocation(s) 
19.54 KBytes      5,580 allocation(s) 
2.35 KBytes       5,580 allocation(s) 
36 Bytes          4,895 allocation(s) 

 

So now we can see for the function, we should focus on allocations of a particular size.  So if we want to focus on the 24 Byte allocations, we then scroll a little farther down in the report and we will see callstacks.  Just scroll until you find one of the size you are looking for.  For example:

 

Call stack sample 3

Address   0x1c4704d0 
Allocation Time   00:05:02 since tracking started 
Allocation Size   24 Bytes 

Function
mscorwks!EEHeapAlloc+12d
mscorwks!EEHeapAllocInProcessHeap+51
System.Threading._TimerCallback.TimerCallback_Context(System.Object) System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)       
webengine!HashtableIUnknown::AddCallback+a
webengine!HttpCompletion::ProcessRequestInManagedCode+1a3
webengine!HttpCompletion::ProcessRequestInManagedCode+1a3
webengine!HttpCompletion::ProcessCompletion+3e
webengine!CorThreadPoolWorkitemCallback+18
mscorwks!ThreadpoolMgr::intermediateThreadProc+49
kernel32!BaseThreadStart+34

 

So now we have callstacks that allocated the memory that we need to focus on.  There is some more information about DebugDiag on it's blog: http://blogs.msdn.com/debugdiag/