So in a previous post, High memory, CPU, or other performance problems with .NET, we began to give some things to look for when dealing with High Memory. So what if debug isn't set and we aren't using debug modules?
So the best thing to do first is to try to figure out where all the memory is. For example, it could be in managed (.NET memory) or it could be native, or we could have a problem like Dynamic Assemblies and what to do about them that makes it look like high memory but it really isn't the cause of our issue.
The best command to use to look at memory as a whole is !address -summary. It will give you output that looks like:
0:000> !address -summary -------------------- Usage SUMMARY -------------------------- TotSize ( KB) Pct(Tots) Pct(Busy) Usage 4a434000 ( 1216720) : 58.02% 89.42% : RegionUsageIsVAD 2cf36000 ( 736472) : 35.12% 00.00% : RegionUsageFree 3d0b000 ( 62508) : 02.98% 04.59% : RegionUsageImage d80000 ( 13824) : 00.66% 01.02% : RegionUsageStack 36000 ( 216) : 00.01% 00.02% : RegionUsageTeb 41c2000 ( 67336) : 03.21% 04.95% : RegionUsageHeap 0 ( 0) : 00.00% 00.00% : RegionUsagePageHeap 1000 ( 4) : 00.00% 00.00% : RegionUsagePeb 1000 ( 4) : 00.00% 00.00% : RegionUsageProcessParametrs 1000 ( 4) : 00.00% 00.00% : RegionUsageEnvironmentBlock Tot: 7fff0000 (2097088 KB) Busy: 530ba000 (1360616 KB) -------------------- Type SUMMARY -------------------------- TotSize ( KB) Pct(Tots) Usage 2cf36000 ( 736472) : 35.12% : 3d0b000 ( 62508) : 02.98% : MEM_IMAGE c10000 ( 12352) : 00.59% : MEM_MAPPED 4e79f000 ( 1285756) : 61.31% : MEM_PRIVATE -------------------- State SUMMARY -------------------------- TotSize ( KB) Pct(Tots) Usage 3ffd2000 ( 1048392) : 49.99% : MEM_COMMIT 2cf36000 ( 736472) : 35.12% : MEM_FREE 130e8000 ( 312224) : 14.89% : MEM_RESERVE Largest free region: Base 685ba000 - Size 04b36000 (77016 KB)
So from this, we can see if the problem is in NT Heaps, Virtual Memory, Images (DLL's) or if none of them are that high and the issue is memory fragmentation. Another important thing to look at is at the bottom. If the largest free region is low, that is always a good sign that we are out of memory. In this case, the dump is mostly RegionUsageIsVAD which is Virtual Memory, where .NET allocates things. Had this been in RegionUsageHeap mostly, it would be a native NT Heap leak and we would troubleshoot that differently, we will address that in a future post with using DebugDiag.
Now that we know the problem is with Virtual Memory, we will see if the issue is with .NET itself (as other programs can also allocate memory in the Virtual Address Space). There are a few ways to do this, such as running !clr10\sos.eeheap -gc and look at the amount of space that the heaps are taking up. You can also just run the command which prints out all the objects in the .NET heap as it totals them all up at the end.
0:000> !clr10\sos.dumpheap -stat Loaded Son of Strike data table version 5 from "C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\mscorsvr.dll" Loading the heap objects into our cache. ------------------------------ Heap 0 total 3,234,104 objects ------------------------------ Heap 1 total 1,753,696 objects ------------------------------ Heap 2 total 2,309,441 objects ------------------------------ Heap 3 total 2,128,943 objects ------------------------------ Heap 4 total 2,554,432 objects ------------------------------ Heap 5 total 3,801,613 objects ------------------------------ Heap 6 total 3,083,516 objects ------------------------------ Heap 7 total 2,645,112 objects ------------------------------ total 21,510,857 objects Statistics: MT Count TotalSize Class Name 0x79be7078 1 12 System.Runtime.Remoting.Proxies.ProxyAttribute 0x79bc08e0 1 12 System.Empty 0x79b9f448 1 12 System.Byte 0x03179c0c 1 12 System.InvariantComparer ... 0x79bab93c 1,934 100,568 System.Collections.Hashtable 0x025ea844 2,400 115,200 System.Web.Caching.ExpiresBucket 0x027958c4 4,992 119,808 System.Xml.NameTable/Entry 0x030653d4 2,491 139,496 System.Data.SimpleType 0x79ba2ee4 6,868 164,832 System.Collections.ArrayList 0x032a5b5c 14,795 236,720 System.Data.DataRowChangeEventArgs 0x025e1294 123 253,380 System.Decimal[] 0x0306309c 13,033 312,792 System.Data.ParentForeignKeyConstraintEnumerator 0x0317f934 5,348 406,448 System.Data.SqlClient._SqlMetaData 0x03060024 4,113 526,464 System.Data.DataColumn 0x01ea2970 2,275 610,128 System.Collections.Hashtable/bucket[] 0x79baaf78 75,231 902,772 System.Int32 0x03064b44 22,838 1,461,632 System.Data.DataColumnPropertyDescriptor 0x01ea375c 244 1,603,180 System.Boolean[] 0x01ea236c 1,464 1,950,192 System.Char[] 0x79ba3adc 95,948 2,302,752 System.Collections.ArrayList/ArrayListEnumeratorSimple 0x032a57b4 256,739 5,134,780 System.Data.DataColumnChangeEventArgs 0x01ea2c3c 1,425 8,196,224 System.Byte[] 0x025e1da4 153 12,738,360 System.DateTime[] 0x01ea26b0 2,090 20,273,700 System.Int32[] 0x030bb59c 1,319,665 58,065,260 System.Data.DataRow 0x000e1de8 1,004,003 81,543,256 Free 0x01ea209c 21,974 127,692,108 System.Object[] 0x79b94638 18,579,050 554,497,748 System.String Total 21,510,857 objects, Total size: 881,488,504 Fragmented blocks larger than 0.5MB: Addr Size Followed by 0x2ff9f2c0 1.0MB large free object followed by 0x3009cfd8 System.String 0x303ec4cc 0.6MB large free object followed by 0x30484af0 System.Object[] 0x0b608a8c 0.6MB large free object followed by 0x0b6a3d4c Travelers.CL.LPE.BoilerWorkstation.Datasets.PolicyInfo/PolicyLocRow 0x3ad2b8a4 0.7MB large free object followed by 0x3ade5ad8 Travelers.CL.LPE.BoilerWorkstation.Datasets.PolicyInfo/PolicyLocRow 0x3d6598f4 0.8MB large free object followed by 0x3d732f40 System.String 0x3fe78f04 0.6MB large free object followed by 0x3ff1e72c System.String 0x401d3a94 0.7MB large free object followed by 0x40283c94 System.Web.HttpInputStream
From this output we can see that the .NET heap is taking up 881,488,504 bytes or 840.7MB. We will look at some other possible causes later, but here we see that strings are taking up 554,497,748 bytes. So that is where the bulk of our problems are. Now strings can be caused by a number of things. Like not using the StringBuilder class (KB306822). But in this case, we can notice the high number of DataRow objects and DataColumn objects and run another command to look at the System.Data.DataTable objects
0:000> !clr10\sos.dumpdatatables DataTable Rows Columns DataSet nextRowID ColumnCount ----------------------------------------------------------------------------------------------- 0x55807f2c 0x558081e4 0x55808408 0x55807e24 1 11 ... 0x407454e0 0x40745774 0x407459a8 0x40745358 218 2 0x4074347c 0x40743710 0x40743944 0x407432f4 218 2 0x408969a8 0x40896c24 0x40896e58 0x40895ee8 854 12 0x168731bc 0x168734a4 0x16873678 0x1686ffb0 402,521 23 0x123596fc 0x123599e4 0x12359bb8 0x123564f0 446,028 23 0x1439ba30 0x1439bd18 0x1439beec 0x14398824 471,117 23 Total 189 DataTable objects
Most of the time, all cells of a DataTable aren't strings, but each of those bottom three tables have over 10 million cells. So a lot of the Strings are coming from these tables. Now we can't get a lot of info on these tables, but we can get the names of the columns. Below are the commands, along with a few new commands that make the process much shorter. Note: !do is short for !dumpobj
0:000> !clr10\sos.do 0x1439beec Name: System.Data.DataColumnCollection MethodTable 0x03062254 EEClass 0x02d8d868 Size 52(0x34) bytes GC Generation: 2 mdToken: 0x0200001a (c:\windows\assembly\gac\system.data\1.0.5000.0__b77a5c561934e089\system.data.dll) FieldDesc*: 0x03061e94 MT Field Offset Type Attr Value Name 0x0302b7f8 0x4000331 0 CLASS shared static RefreshEventArgs >> Domain:Value 0x000dcbb0:NotInit 0x00104fa0:0x1e2fb450 0x026c12a0:0x1e34a790 0x02f6b6a0:0x1a314a24 0x02fed358:0x12309044 0x03f78600:0x103cbad0 << 0x03062254 0x4000376 0x4 CLASS instance 0x1439ba30 table 0x03062254 0x4000377 0x8 CLASS instance 0x1439bf20 list 0x03062254 0x4000378 0x28 System.Int32 instance 1 defaultNameIndex 0x03062254 0x4000379 0xc CLASS instance 0x00000000 delayedAddRangeColumns 0x03062254 0x400037a 0x10 CLASS instance 0x1439d490 columnQueue 0x03062254 0x400037b 0x14 CLASS instance 0x1439bf38 columnFromName 0x03062254 0x400037c 0x18 CLASS instance 0x1439bf6c hashCodeProvider 0x03062254 0x400037d 0x1c CLASS instance 0x00000000 onCollectionChangedDelegate 0x03062254 0x400037e 0x20 CLASS instance 0x00000000 onCollectionChangingDelegate 0x03062254 0x400037f 0x24 CLASS instance 0x00000000 onColumnPropertyChangedDelegate 0x03062254 0x4000380 0x2c System.Boolean instance 0 fInClear 0:000> !clr10\sos.do 0x1439bf20 Name: System.Collections.ArrayList MethodTable 0x79ba2ee4 EEClass 0x79ba3020 Size 24(0x18) bytes GC Generation: 2 mdToken: 0x02000100 (c:\windows\microsoft.net\framework\v1.1.4322\mscorlib.dll) FieldDesc*: 0x79ba3084 MT Field Offset Type Attr Value Name 0x79ba2ee4 0x4000362 0x4 CLASS instance 0x1439cab4 _items 0x79ba2ee4 0x4000363 0xc System.Int32 instance 23 _size 0x79ba2ee4 0x4000364 0x10 System.Int32 instance 24 _version 0x79ba2ee4 0x4000365 0x8 CLASS instance 0x00000000 _syncRoot 0:000> .foreach ( myobj { !clr10\sos.do -v 0x1439cab4 -short } ) { !clr10\sos.dumpfield -field _columnName myobj } String: Loc1 String: Loc2 String: SLoc String: Num String: Address String: Address2 String: City String: County String: State String: Zip String: Country String: ContactNm String: ContactTitle String: ContactPhone String: ContactEmail String: DataSource String: Status String: Occupancy String: TotalAmt String: NewInd String: UniqueId String: EspLocation String: PrevInsp
We will continue to look at high memory issues in future posts. So stay tuned and feel free to ask questions or give any feedback.