In order to debug any high memory issue we rely heavily on the output of !address –summary command
[You’ll find !address –summary as the part of Ext.dll extension].

We can interpret quite a few things from it which can help us in further debugging. Here’s how

For example (for  32 bit app)

0:027> !address –summary
-------------------- Usage SUMMARY --------------------------
    TotSize (      KB)   Pct(Tots) Pct(Busy)   Usage
   29b32000 (  683208) : 32.58%    41.98%    :
RegionUsageIsVAD
   1cab1000 (  469700) : 22.40%    00.00%    :
RegionUsageFree
    d3b4000 (  216784) : 10.34%    13.32%    :
RegionUsageImage
    3bfc000 (   61424) : 02.93%    03.77%    : RegionUsageStack
      f0000 (     960) : 00.05%    00.06%    : RegionUsageTeb
   2896a000 (  665000) : 31.71%    40.86%    :
RegionUsageHeap
          0 (       0) : 00.00%    00.00%    : RegionUsagePageHeap
       1000 (       4) : 00.00%    00.00%    : RegionUsagePeb
       1000 (       4) : 00.00%    00.00%    : RegionUsageProcessParametrs
       1000 (       4) : 00.00%    00.00%    : RegionUsageEnvironmentBlock
       Tot: 7fff0000 (2097088 KB) Busy: 6353f000 (1627388 KB)

-------------------- Type SUMMARY --------------------------
    TotSize (      KB)   Pct(Tots)  Usage
   1cab1000 (  469700) : 22.40%   : <free>
   119a8000 (  288416) : 13.75%   : MEM_IMAGE
    10b5000 (   17108) : 00.82%   : MEM_MAPPED
   50ae2000 ( 1321864) : 63.03%   : MEM_PRIVATE

------------------- State SUMMARY --------------------------
    TotSize (      KB)   Pct(Tots)  Usage
   3152f000 (  808124) : 38.54%   : MEM_COMMIT
   1cab1000 (  469700) : 22.40%   : MEM_FREE
   32010000 (  819264) : 39.07%   : MEM_RESERVE Largest free region: Base 6b0b2000 - Size 0203e000 (33016 KB) *

From above output

RegionUsageIsVAD

.Net make allocations in this region   [682 MB]

We have use !dumpheap –stat or !eeheap –gc to get to know actual GC heap size.
And to know which objects are taking most memory

!dumpheap –stat::
0x79330a00     61,005    315,568,148 System.String
0x793041d0      8,654    25,555,704 System.Object[]
0x000e43e8        244    143,488,500      Free è
  hmm Large Free.
Check LOH for big allocations

Total 384,371 objects, Total size: 700,942,864

!eeheap –gc::
GC Heap Size 0x7932d0(700,942,864)

You can also use .Net memory analyzer script written by Tess
Above commands[!dumpheap,!eeheap] are part of psscor2 extension. [superset of sos.dl]

RegionUsageFree

Actual free space out of 2 GB usermode  [269 MB]

If you see large memory in this region but small value of Largest contiguous block that indicates memory is fragmented.

RegionUsageImage

Region where Dll’s/modules get loaded  [216 Mb]

Either heavy dll’s loaded / Debug is set true / Dynamic Assemblies due to XML serialization

RegionUsageHeap

Native/Umanaged allocations here [665 MB]

Use the DebugDiag memory analysis script on the existing dump file to get more information on the heaps.  A DebugDiag leak rule can be used while the process is running to collect data on the allocations.  Please see the following:

 

  • Within RegionUsageFree what matter is the Largest contiguous memory block(33016 KB). .Net needs minimum of 64 MB on contiguous memory block. Less than this we are prone to OutofMemoryException

 

Looking at State Summary we can get to know where exactly 2Gb has ended up

MEM_COMMIT [Actual process size]  +   MEM_FREE[same as RegionUsageFree]      MEM_RESERVE  == 2 GB usermode space == Tot: 7fff0000 (2097088 KB)

MEM_COMMIT [Actual process size]   + MEM_RESERVE  == Busy bytes == Busy: 6353f000 (1627388 KB)

 

Following is the output of !address -summary  under different conditions i.e when  application is running on different platforms[32 bit/64 bit]:

1) When app is in 32 bit mode [windows 2003]

0:027> !address -summary

-------------------- Usage SUMMARY --------------------------
    TotSize (      KB)   Pct(Tots) Pct(Busy)   Usage
   29b32000 (  683208) : 32.58%    41.98%    : RegionUsageIsVAD
   1cab1000 (  469700) : 22.40%    00.00%    : RegionUsageFree
    d3b4000 (  216784) : 10.34%    13.32%    : RegionUsageImage
    3bfc000 (   61424) : 02.93%    03.77%    : RegionUsageStack
      f0000 (     960) : 00.05%    00.06%    : RegionUsageTeb
   2896a000 (  665000) : 31.71%    40.86%    : RegionUsageHeap
          0 (       0) : 00.00%    00.00%    : RegionUsagePageHeap
       1000 (       4) : 00.00%    00.00%    : RegionUsagePeb
       1000 (       4) : 00.00%    00.00%    : RegionUsageProcessParametrs
       1000 (       4) : 00.00%    00.00%    : RegionUsageEnvironmentBlock
       Tot: 7fff0000 (2097088 KB) Busy: 6353f000 (1627388 KB)

-------------------- Type SUMMARY --------------------------
    TotSize (      KB)   Pct(Tots)  Usage
   1cab1000 (  469700) : 22.40%   : <free>
   119a8000 (  288416) : 13.75%   : MEM_IMAGE
    10b5000 (   17108) : 00.82%   : MEM_MAPPED
   50ae2000 ( 1321864) : 63.03%   : MEM_PRIVATE

-------------------- State SUMMARY --------------------------
    TotSize (      KB)   Pct(Tots)  Usage
   3152f000 (  808124) : 38.54%   : MEM_COMMIT
   1cab1000 (  469700) : 22.40%   : MEM_FREE
   32010000 (  819264) : 39.07%   : MEM_RESERVE

Largest free region: Base 6b0b2000 - Size 0203e000 (33016 KB)

Max available memory in usermode would be 2 GB[2097088 KB]

2) When app is on 32 bit mode with /3gb switch [windows 2003]

0:000> !address -summary

-------------------- Usage SUMMARY --------------------------
    TotSize (      KB)   Pct(Tots) Pct(Busy)   Usage
   3447b000 (  856556) : 27.23%    49.95%    : RegionUsageIsVAD
   5753e000 ( 1430776) : 45.48%    00.00%    : RegionUsageFree
   109cb000 (  272172) : 08.65%    15.87%    : RegionUsageImage
     e3d000 (   14580) : 00.46%    00.85%    : RegionUsageStack
      39000 (     228) : 00.01%    00.01%    : RegionUsageTeb
   22df4000 (  571344) : 18.16%    33.32%    : RegionUsageHeap
          0 (       0) : 00.00%    00.00%    : RegionUsagePageHeap
          0 (       0) : 00.00%    00.00%    : RegionUsagePeb
       1000 (       4) : 00.00%    00.00%    : RegionUsageProcessParametrs
       1000 (       4) : 00.00%    00.00%    : RegionUsageEnvironmentBlock
       Tot: bfff0000 (3145664 KB) Busy: 68ab2000 (1714888 KB)

-------------------- Type SUMMARY --------------------------
    TotSize (      KB)   Pct(Tots)  Usage
   5753e000 ( 1430776) : 45.48%   : <free>
   1267d000 (  301556) : 09.59%   : MEM_IMAGE
     808000 (    8224) : 00.26%   : MEM_MAPPED
   55c2d000 ( 1405108) : 44.67%   : MEM_PRIVATE

-------------------- State SUMMARY --------------------------
    TotSize (      KB)   Pct(Tots)  Usage
   41ac2000 ( 1075976) : 34.21%   : MEM_COMMIT
   5753e000 ( 1430776) : 45.48%   : MEM_FREE
   26ff0000 (  638912) : 20.31%   : MEM_RESERVE

Largest free region: Base 88010000 - Size 37ef9000 (916452 KB)

Total available would be 3 GB(3145664 KB)

3) When app is in 32 bit mode on x64 box[windows 2003]

0:000> !address -summary
-------------------- Usage SUMMARY --------------------------
    TotSize (      KB)   Pct(Tots) Pct(Busy)   Usage
   21290000 (  543296) : 12.95%    24.52%    : RegionUsageIsVAD
   78bc5000 ( 1978132) : 47.16%    00.00%    : RegionUsageFree
    ad14000 (  177232) : 04.23%    08.00%    : RegionUsageImage
    25fc000 (   38896) : 00.93%    01.76%    : RegionUsageStack
          0 (       0) : 00.00%    00.00%    : RegionUsageTeb
   58e89000 ( 1456676) : 34.73%    65.73%    : RegionUsageHeap
          0 (       0) : 00.00%    00.00%    : RegionUsagePageHeap
          0 (       0) : 00.00%    00.00%    : RegionUsagePeb
       1000 (       4) : 00.00%    00.00%    : RegionUsageProcessParametrs
       1000 (       4) : 00.00%    00.00%    : RegionUsageEnvironmentBlock
      Tot: ffff0000 (4194240 KB) Busy: 8742b000 (2216108 KB)

-------------------- Type SUMMARY --------------------------
    TotSize (      KB)   Pct(Tots)  Usage
   78bc5000 ( 1978132) : 47.16%   : <free>
    c7a8000 (  204448) : 04.87%   : MEM_IMAGE
     b74000 (   11728) : 00.28%   : MEM_MAPPED
   7a10f000 ( 1999932) : 47.68%   : MEM_PRIVATE

-------------------- State SUMMARY --------------------------
    TotSize (      KB)   Pct(Tots)  Usage
   46049000 ( 1147172) : 27.35%   : MEM_COMMIT
   78bc5000 ( 1978132) : 47.16%   : MEM_FREE
   413e2000 ( 1068936) : 25.49%   : MEM_RESERVE

Largest free region: Base b0010000 - Size 4fd80000 (1308160 KB)

Max we would have under this condition is 4 GB of usermode space

4) When app is on 64 bit mode on 64 bit box [windows 2003]

0:014> !address -summary

-------------------- Usage SUMMARY --------------------------
    TotSize (      KB)   Pct(Tots) Pct(Busy)   Usage
   145d58000 ( 5338464) : 00.06%    89.85%    : RegionUsageIsVAD
   7fe95589000 (8583992868) : 99.93%    00.00%    : RegionUsageFree
   10e53000 (  276812) : 00.00%    04.66%    : RegionUsageImage
    2100000 (   33792) : 00.00%    00.57%    : RegionUsageStack
      84000 (     528) : 00.00%    00.01%    : RegionUsageTeb
   11d35000 (  292052) : 00.00%    04.92%    : RegionUsageHeap
          0 (       0) : 00.00%    00.00%    : RegionUsagePageHeap
       1000 (       4) : 00.00%    00.00%    : RegionUsagePeb
       1000 (       4) : 00.00%    00.00%    : RegionUsageProcessParametrs
       1000 (       4) : 00.00%    00.00%    : RegionUsageEnvironmentBlock
       Tot: 7ffffff0000 (8589934528 KB) Busy: 000000016aa67000 (5941660 KB)

-------------------- Type SUMMARY --------------------------
    TotSize (      KB)   Pct(Tots)  Usage
   7fe95589000 (8583992868) : 99.93%   : <free>
   12fcc000 (  311088) : 00.00%   : MEM_IMAGE
     8fc000 (    9200) : 00.00%   : MEM_MAPPED
   15719f000 ( 5621372) : 00.07%   : MEM_PRIVATE

-------------------- State SUMMARY --------------------------
    TotSize (      KB)   Pct(Tots)  Usage
   6eb9d000 ( 1814132) : 00.02%   : MEM_COMMIT
   7fe95589000 (8583992868) : 99.93%   : MEM_FREE
   fbeca000 ( 4127528) : 00.05%   : MEM_RESERVE

Largest free region: Base 0000000516fc5000 - Size 0000063d2589b000 (6698918508 KB)

Max available would be 8 terabyte [usermode] + 8 terabyte[kernelmode] = 16 TB

All the of above scenarios explaining memory availability goes in line with Tom’s blog

32-bit OS

64-bit OS

32-bit process

2 GB

4 GB

32-bit process with /3GB

3 GB

N/A

64-bit process

N/A

16 TB

These process numbers are contingent on how much RAM and disk space you have, so if you have 4 GB of RAM and 4 GB Pagefile,
the total memory of all running processes can’t be greater then 8 GB.

Happy Debugging!