So in a previous post, High memory, CPU, or other performance problems with .NET, we began to give some things to look for when dealing with High Memory.  So what if debug isn't set and we aren't using debug modules?

So the best thing to do first is to try to figure out where all the memory is.  For example, it could be in managed (.NET memory) or it could be native, or we could have a problem like Dynamic Assemblies and what to do about them that makes it look like high memory but it really isn't the cause of our issue.

The best command to use to look at memory as a whole is !address -summary.  It will give you output that looks like:


0:000> !address -summary

-------------------- Usage SUMMARY --------------------------
    TotSize (      KB)   Pct(Tots) Pct(Busy)   Usage
   4a434000 ( 1216720) : 58.02%    89.42%    : RegionUsageIsVAD
   2cf36000 (  736472) : 35.12%    00.00%    : RegionUsageFree
    3d0b000 (   62508) : 02.98%    04.59%    : RegionUsageImage
     d80000 (   13824) : 00.66%    01.02%    : RegionUsageStack
      36000 (     216) : 00.01%    00.02%    : RegionUsageTeb
    41c2000 (   67336) : 03.21%    04.95%    : RegionUsageHeap
          0 (       0) : 00.00%    00.00%    : RegionUsagePageHeap
       1000 (       4) : 00.00%    00.00%    : RegionUsagePeb
       1000 (       4) : 00.00%    00.00%    : RegionUsageProcessParametrs
       1000 (       4) : 00.00%    00.00%    : RegionUsageEnvironmentBlock
       Tot: 7fff0000 (2097088 KB) Busy: 530ba000 (1360616 KB)

-------------------- Type SUMMARY --------------------------
    TotSize (      KB)   Pct(Tots)  Usage
   2cf36000 (  736472) : 35.12%   : 
    3d0b000 (   62508) : 02.98%   : MEM_IMAGE
     c10000 (   12352) : 00.59%   : MEM_MAPPED
   4e79f000 ( 1285756) : 61.31%   : MEM_PRIVATE

-------------------- State SUMMARY --------------------------
    TotSize (      KB)   Pct(Tots)  Usage
   3ffd2000 ( 1048392) : 49.99%   : MEM_COMMIT
   2cf36000 (  736472) : 35.12%   : MEM_FREE
   130e8000 (  312224) : 14.89%   : MEM_RESERVE

Largest free region: Base 685ba000 - Size 04b36000 (77016 KB)


So from this, we can see if the problem is in NT Heaps, Virtual Memory, Images (DLL's) or if none of them are that high and the issue is memory fragmentation.  Another important thing to look at is at the bottom.  If the largest free region is low, that is always a good sign that we are out of memory.  In this case, the dump is mostly RegionUsageIsVAD which is Virtual Memory, where .NET allocates things.  Had this been in RegionUsageHeap mostly, it would be a native NT Heap leak and we would troubleshoot that differently, we will address that in a future post with using DebugDiag.

Now that we know the problem is with Virtual Memory, we will see if the issue is with .NET itself (as other programs can also allocate memory in the Virtual Address Space).  There are a few ways to do this, such as running !clr10\sos.eeheap -gc and look at the amount of space that the heaps are taking up.  You can also just run the command which prints out all the objects in the .NET heap as it totals them all up at the end.


0:000> !clr10\sos.dumpheap -stat
Loaded Son of Strike data table version 5 from "C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\mscorsvr.dll"
Loading the heap objects into our cache.
Heap 0
total 3,234,104 objects
Heap 1
total 1,753,696 objects
Heap 2
total 2,309,441 objects
Heap 3
total 2,128,943 objects
Heap 4
total 2,554,432 objects
Heap 5
total 3,801,613 objects
Heap 6
total 3,083,516 objects
Heap 7
total 2,645,112 objects
total 21,510,857 objects
        MT      Count     TotalSize Class Name
0x79be7078          1            12 System.Runtime.Remoting.Proxies.ProxyAttribute
0x79bc08e0          1            12 System.Empty
0x79b9f448          1            12 System.Byte
0x03179c0c          1            12 System.InvariantComparer
0x79bab93c      1,934       100,568 System.Collections.Hashtable
0x025ea844      2,400       115,200 System.Web.Caching.ExpiresBucket
0x027958c4      4,992       119,808 System.Xml.NameTable/Entry
0x030653d4      2,491       139,496 System.Data.SimpleType
0x79ba2ee4      6,868       164,832 System.Collections.ArrayList
0x032a5b5c     14,795       236,720 System.Data.DataRowChangeEventArgs
0x025e1294        123       253,380 System.Decimal[]
0x0306309c     13,033       312,792 System.Data.ParentForeignKeyConstraintEnumerator
0x0317f934      5,348       406,448 System.Data.SqlClient._SqlMetaData
0x03060024      4,113       526,464 System.Data.DataColumn
0x01ea2970      2,275       610,128 System.Collections.Hashtable/bucket[]
0x79baaf78     75,231       902,772 System.Int32
0x03064b44     22,838     1,461,632 System.Data.DataColumnPropertyDescriptor
0x01ea375c        244     1,603,180 System.Boolean[]
0x01ea236c      1,464     1,950,192 System.Char[]
0x79ba3adc     95,948     2,302,752 System.Collections.ArrayList/ArrayListEnumeratorSimple
0x032a57b4    256,739     5,134,780 System.Data.DataColumnChangeEventArgs
0x01ea2c3c      1,425     8,196,224 System.Byte[]
0x025e1da4        153    12,738,360 System.DateTime[]
0x01ea26b0      2,090    20,273,700 System.Int32[]
0x030bb59c  1,319,665    58,065,260 System.Data.DataRow
0x000e1de8  1,004,003    81,543,256      Free
0x01ea209c     21,974   127,692,108 System.Object[]
0x79b94638 18,579,050   554,497,748 System.String
Total 21,510,857 objects, Total size: 881,488,504

Fragmented blocks larger than 0.5MB:
      Addr     Size Followed by
0x2ff9f2c0 1.0MB large free object followed by 0x3009cfd8 System.String
0x303ec4cc 0.6MB large free object followed by 0x30484af0 System.Object[]
0x0b608a8c 0.6MB large free object followed by 0x0b6a3d4c Travelers.CL.LPE.BoilerWorkstation.Datasets.PolicyInfo/PolicyLocRow
0x3ad2b8a4 0.7MB large free object followed by 0x3ade5ad8 Travelers.CL.LPE.BoilerWorkstation.Datasets.PolicyInfo/PolicyLocRow
0x3d6598f4 0.8MB large free object followed by 0x3d732f40 System.String
0x3fe78f04 0.6MB large free object followed by 0x3ff1e72c System.String
0x401d3a94 0.7MB large free object followed by 0x40283c94 System.Web.HttpInputStream


From this output we can see that the .NET heap is taking up 881,488,504 bytes or 840.7MB.  We will look at some other possible causes later, but here we see that strings are taking up 554,497,748 bytes.  So that is where the bulk of our problems are.  Now strings can be caused by a number of things.  Like not using the StringBuilder class (KB306822).  But in this case, we can notice the high number of DataRow objects and DataColumn objects and run another command to look at the System.Data.DataTable objects


0:000> !clr10\sos.dumpdatatables
 DataTable       Rows    Columns    DataSet nextRowID ColumnCount
0x55807f2c 0x558081e4 0x55808408 0x55807e24         1        11 
0x407454e0 0x40745774 0x407459a8 0x40745358       218         2 
0x4074347c 0x40743710 0x40743944 0x407432f4       218         2 
0x408969a8 0x40896c24 0x40896e58 0x40895ee8       854        12 
0x168731bc 0x168734a4 0x16873678 0x1686ffb0   402,521        23 
0x123596fc 0x123599e4 0x12359bb8 0x123564f0   446,028        23 
0x1439ba30 0x1439bd18 0x1439beec 0x14398824   471,117        23 
Total 189 DataTable objects


Most of the time, all cells of a DataTable aren't strings, but each of those bottom three tables have over 10 million cells.  So a lot of the Strings are coming from these tables.  Now we can't get a lot of info on these tables, but we can get the names of the columns.  Below are the commands, along with a few new commands that make the process much shorter. Note: !do is short for !dumpobj


0:000> !clr10\ 0x1439beec 
Name: System.Data.DataColumnCollection
MethodTable 0x03062254
EEClass 0x02d8d868
Size 52(0x34) bytes
GC Generation: 2
mdToken: 0x0200001a  (c:\windows\assembly\gac\\1.0.5000.0__b77a5c561934e089\
FieldDesc*: 0x03061e94
        MT      Field     Offset                 Type       Attr      Value Name
0x0302b7f8 0x4000331        0            CLASS     shared   static RefreshEventArgs
    >> Domain:Value 0x000dcbb0:NotInit  0x00104fa0:0x1e2fb450 0x026c12a0:0x1e34a790 0x02f6b6a0:0x1a314a24 0x02fed358:0x12309044 0x03f78600:0x103cbad0 <<
0x03062254 0x4000376      0x4            CLASS   instance 0x1439ba30 table
0x03062254 0x4000377      0x8            CLASS   instance 0x1439bf20 list
0x03062254 0x4000378     0x28     System.Int32   instance 1 defaultNameIndex
0x03062254 0x4000379      0xc            CLASS   instance 0x00000000 delayedAddRangeColumns
0x03062254 0x400037a     0x10            CLASS   instance 0x1439d490 columnQueue
0x03062254 0x400037b     0x14            CLASS   instance 0x1439bf38 columnFromName
0x03062254 0x400037c     0x18            CLASS   instance 0x1439bf6c hashCodeProvider
0x03062254 0x400037d     0x1c            CLASS   instance 0x00000000 onCollectionChangedDelegate
0x03062254 0x400037e     0x20            CLASS   instance 0x00000000 onCollectionChangingDelegate
0x03062254 0x400037f     0x24            CLASS   instance 0x00000000 onColumnPropertyChangedDelegate
0x03062254 0x4000380     0x2c   System.Boolean   instance 0 fInClear
0:000> !clr10\ 0x1439bf20 
Name: System.Collections.ArrayList
MethodTable 0x79ba2ee4
EEClass 0x79ba3020
Size 24(0x18) bytes
GC Generation: 2
mdToken: 0x02000100  (c:\windows\\framework\v1.1.4322\mscorlib.dll)
FieldDesc*: 0x79ba3084
        MT      Field     Offset                 Type       Attr      Value Name
0x79ba2ee4 0x4000362      0x4            CLASS   instance 0x1439cab4 _items
0x79ba2ee4 0x4000363      0xc     System.Int32   instance 23 _size
0x79ba2ee4 0x4000364     0x10     System.Int32   instance 24 _version
0x79ba2ee4 0x4000365      0x8            CLASS   instance 0x00000000 _syncRoot
0:000> .foreach ( myobj { !clr10\ -v 0x1439cab4 -short } ) {
    !clr10\sos.dumpfield -field _columnName myobj }
String: Loc1

String: Loc2

String: SLoc

String: Num

String: Address

String: Address2

String: City

String: County

String: State

String: Zip

String: Country

String: ContactNm

String: ContactTitle

String: ContactPhone

String: ContactEmail

String: DataSource

String: Status

String: Occupancy

String: TotalAmt

String: NewInd

String: UniqueId

String: EspLocation

String: PrevInsp


We will continue to look at high memory issues in future posts.  So stay tuned and feel free to ask questions or give any feedback.