@TessFerrandez
I was writing a post on debugging a high memory usage problem caused by storing too much in session scope, but I realized I got in to tangent discussions all the time so I decided to create a separate post on caching first. Caching is by far the most common cause of high memory problems. It’s an understatement to say that the cache is often overused. As with everything else nothing comes for free, so the performance you gain by storing items in cache has to be compared to the performance you loose by having to scavenge this memory. Be careful with the design and set expiration times etc. appropriately, and make sure that what you cache will actually be accessed enough times to make it worth while. Sos.dll has some nice features for examining your cache through the !dumpaspnetcache command; however before we get in to that, let’s look at some commands to determine if you have a cache problem. Where is the cache? Each HttpRuntime (running application) has a reference to a System.Web.Caching.CacheMultiple object (the _cache field), where the cached objects for that application is stored. In the particular version I am running (1.1.4322.2300), the _cache field is located at offset 0xc from the address of the HttpRuntime object. 0:023> !do 0x061b47d4 Name: System.Web.HttpRuntime MethodTable 0x01b29dfc EEClass 0x01ee8314 Size 116(0x74) bytes GC Generation: 2 mdToken: 0x02000079 (c:\windows\assembly\gac\system.web\1.0.5000.0__b03f5f7f11d50a3a\system.web.dll) FieldDesc*: 0x01b29554 MT Field Offset Type Attr Value Name 0x01b29dfc 0x4000683 0x4 CLASS instance 0x00000000 _namedPermissionSet 0x01b29dfc 0x4000684 0x8 CLASS instance 0x061b4950 _fcm 0x01b29dfc 0x4000685 0xc CLASS instance 0x061b4ca0 _cache 0x01b29dfc 0x4000686 0x54 System.Boolean instance 0 _isOnUNCShare 0x01b29dfc 0x4000687 0x10 CLASS instance 0x061b6e18 _profiler 0x01b29dfc 0x4000688 0x14 CLASS instance 0x061b6e34 _timeoutManager 0x01b29dfc 0x4000689 0x18 CLASS instance 0x021b368c _requestQueue 0x01b29dfc 0x400068a 0x55 System.Boolean instance 0 _apartmentThreading 0x01b29dfc 0x400068b 0x56 System.Boolean instance 0 _beforeFirstRequest 0x01b29dfc 0x400068c 0x60 VALUETYPE instance start at 0x061b4834 _firstRequestStartTime 0x01b29dfc 0x400068d 0x57 System.Boolean instance 1 _firstRequestCompleted 0x01b29dfc 0x400068e 0x58 System.Boolean instance 0 _userForcedShutdown 0x01b29dfc 0x400068f 0x59 System.Boolean instance 1 _configInited 0x01b29dfc 0x4000690 0x50 System.Int32 instance 1 _activeRequestCount 0x01b29dfc 0x4000691 0x5a System.Boolean instance 0 _someBatchCompilationStarted 0x01b29dfc 0x4000692 0x5b System.Boolean instance 0 _shutdownInProgress 0x01b29dfc 0x4000693 0x1c CLASS instance 0x00000000 _shutDownStack 0x01b29dfc 0x4000694 0x20 CLASS instance 0x00000000 _shutDownMessage 0x01b29dfc 0x4000695 0x68 VALUETYPE instance start at 0x061b483c _lastShutdownAttemptTime 0x01b29dfc 0x4000696 0x5c System.Boolean instance 1 _enableHeaderChecking 0x01b29dfc 0x4000697 0x24 CLASS instance 0x061b7014 _handlerCompletionCallback 0x01b29dfc 0x4000698 0x28 CLASS instance 0x061b7030 _asyncEndOfSendCallback 0x01b29dfc 0x4000699 0x2c CLASS instance 0x061b704c _appDomainUnloadallback 0x01b29dfc 0x400069a 0x30 CLASS instance 0x00000000 _initializationError 0x01b29dfc 0x400069b 0x34 CLASS instance 0x00000000 _appDomainShutdownTimer 0x01b29dfc 0x400069c 0x38 CLASS instance 0x061d4b24 _codegenDir 0x01b29dfc 0x400069d 0x3c CLASS instance 0x02188f1c _appDomainAppId 0x01b29dfc 0x400069e 0x40 CLASS instance 0x02188f68 _appDomainAppPath 0x01b29dfc 0x400069f 0x44 CLASS instance 0x0218a434 _appDomainAppVPath … I’m not going to go in too much in the nitty-gritty but basically this CacheMultiple has a _caches Object[] member variable, which contains some CacheSingle’s and they each have an _entries membervariable which contains the actual CacheEntries, if you want to delve in to them manually. Luckily for us, !dumpaspnetcache automates this as you will see later. Getting the size of the cache The reason that I still mention where the cache is stored, is because with this knowledge we can now compose a command that goes through all our HttpRuntimes and dumps out the name of the app and how much it stores in cache. The pseudo-code for doing this would be: For each (HttpRuntime in (all the runtimes on the heap)) Print “*******************” //to divide the entries Dump out the AppDomainID Print “Cache Size: “ Dump out the size of the cache End for each Translated to windbg terms: To find all the runtimes on the heap, first find the methodtable for System.Web.HttpRuntime and then run !dumpheap –mt . First run !dumpheap –type HttpRuntime which gives us all objects whose name contains HttpRuntime and their method tables (MT). 0:023> !dumpheap -type HttpRuntime Using our cache to search the heap. Address MT Size Gen 0x061d4dbc 0x020c0010 12 -1 System.Web.Configuration.HttpRuntimeConfigurationHandler 0x021b321c 0x0209ff40 52 -1 System.Web.Configuration.HttpRuntimeConfig 0x061b47d4 0x01b29dfc 116 -1 System.Web.HttpRuntime Statistics: MT Count TotalSize Class Name 0x020c0010 1 12 System.Web.Configuration.HttpRuntimeConfigurationHandler 0x0209ff40 1 52 System.Web.Configuration.HttpRuntimeConfig 0x01b29dfc 1 116 System.Web.HttpRuntime Total 3 objects, Total size: 180 And then do !dumpheap –mt 0x01b29dfc –short 0:023> !dumpheap -mt 0x01b29dfc -short 0x61b47d4 The short switch gives us only the address of the objects so we can use it more easily with .foreach. In this case I had only one HttpRuntime because I have only one vdir that I have accessed since the start of the process. The AppDomainId (IIS name for the app) is stored in the _appDomainAppId member variable for HttpRuntime at offset 0x3c... 0x01b29dfc 0x400069d 0x3c CLASS instance 0x02188f1c _appDomainAppId ...so to dump this out given a HttpRuntime address you would run 0:023> !dumpobj poi(0x61b47d4+0x3c) String: /LM/w3svc/1/root/MemoryIssues The poi operator gives us a dword stored at the address we are giving it, i.e. in this case it will give us what is stored at 0x61b47d4+0x3c, where 0x61b47d4 is the address of the HttpRuntime, so it will give us 0x02188f1c, which is the address of _appDomainAppId from above. The reason we use this technique is so we can automate it to run on all the HttpRuntime objects on the heap, since with .foreach where we will only know the address of each HttpRuntime object. Much the same way we get the size of the cache by running !objsize on the _cache member variable 0x01b29dfc 0x4000685 0xc CLASS instance 0x061b4ca0 _cache0:023> !objsize poi(0x61b47d4+0xc) sizeof(0x61b4ca0) = 1,336,709,120 (0x4fac9000) bytes (System.Web.Caching.CacheMultiple) Now we have all the components to run a command that will give us the name and the cache size for all HttpRuntime objects. A short comment on .foreach before we continue... In its simplest form the .foreach command looks like this: .foreach ( Variable { InCommands } ) { OutCommands } Where InCommands is the commands that will give you the array/collection to loop through, and the OutCommands is a semicolon separated list of what commands you want to run each iteration. Our command will look like this: .foreach (runtime {!dumpheap -mt 0x01b29dfc -short}){.echo ******************;!dumpobj poi(${runtime}+0x3c);.echo cache size:;!objsize poi(${runtime}+0xc) } We have to enclose runtime (our foreach variable) in ${} so that it doesn’t get interpreted as a symbol. And the output looks like this: ****************** String: /LM/w3svc/1/root/MemoryIssues cache size: sizeof(0x61b4ca0) = 1,336,709,120 (0x4fac9000) bytes (System.Web.Caching.CacheMultiple) If I would have had more than one application it would have listed all of them one after another, separating them with a line of *** So here we can see that my one application MemoryIssues, has a cache that contains 1,336,709,120 bytes. WOW!!! that's a lot!!! We definitely seem to have some sort of cache problem. What is in my cache? Using !dumpaspnetcache… !dumpaspnetcache –stat gives you a nice overview over what types of objects you are storing in cache. (check the post on !dumpheap –stat to understand TotalSize). 0:023> !dumpaspnetcache -stat Going to dump the ASP.NET Cache. MT Count TotalSize Class Name 0x0211cc9c 1 20 System.Web.Security.FileSecurityDescriptorWrapper 0x020c242c 2 56 System.Web.UI.ParserCacheItem 0x0206c66c 5 260 System.Web.Configuration.HttpConfigurationRecord 0x0c2e7014 1 316 System.Web.Mobile.MobileCapabilities 0x79b94638 4 376 System.String 0x0c2eaeb4 151 7,248 System.Web.SessionState.InProcSessionState Total 164 objects, Total size: 8,276 !dumpaspnetcache –s gives you a summary of your objects with priority (removable or NotRemovable etc.) along with create time, expires and last updated which can be useful to see if you are caching something for too long. The ASP.NET 1.1 cache uses an LRU (Least Recently Used) model for removing objects from cache if memory is needed, for the objects that are not marked NotRemovable. In this example we can see for example some InProcSessionState items (sessions) that started around 15:31:36 and expire on 15:51:36 (after 20 min) and during that time they can not be removed (NotRemovable) by someone trying to clean up the cache to lower memory usage. 0:023> !dumpaspnetcache -s Going to dump the ASP.NET Cache. Address MT Priority Create Time Expires Last Updated Key Class Name 0x061d2658 0x0206c66c Normal 01/25/2006 15:31:35 12/31/9999 23:59:59 01/25/2006 15:31:35 System.Web.HttpConfig: System.Web.Configuration.HttpConfigurationRecord 0x021b1acc 0x0206c66c Normal 01/25/2006 15:31:35 12/31/9999 23:59:59 01/25/2006 15:31:35 System.Web.HttpConfig:/MEMORYISSUES System.Web.Configuration.HttpConfigurationRecord 0x061b3f7c 0x79b94638 Normal 01/25/2006 15:31:35 01/25/2006 15:41:35 01/25/2006 15:31:35 ISAPIWorkerRequest.MapPath:/MemoryIssues/StoringStuffInSessions.aspx System.String 0x021b503c 0x0206c66c Normal 01/25/2006 15:31:35 12/31/9999 23:59:59 01/25/2006 15:31:36 System.Web.HttpConfig:/MEMORYISSUES/STORINGSTUFFINSESSIONS.ASPX System.Web.Configuration.HttpConfigurationRecord 0x021b80b4 0x0211cc9c Normal 01/25/2006 15:31:36 12/31/9999 23:59:59 01/25/2006 15:31:36 System.Web.Security.FileSecurityDescriptorWrapper:c:\inetpub\wwwroot\MemoryIssues\StoringStuffInSessions.aspx System.Web.Security.FileSecurityDescriptorWrapper 0x021bc74c 0x020c242c NotRemovable 01/25/2006 15:31:36 12/31/9999 23:59:59 01/25/2006 15:31:36 System.Web.UI.TemplateParser:/MemoryIssues/StoringStuffInSessions.aspx System.Web.UI.ParserCacheItem 0x022028fc 0x0c2eaeb4 NotRemovable 01/25/2006 15:31:36 01/25/2006 15:51:36 01/25/2006 15:31:37 System.Web.SessionState.SessionStateItem:x4ud0tb53l2wfo554aw0yny4 System.Web.SessionState.InProcSessionState 0x02202a10 0x0c2eaeb4 NotRemovable 01/25/2006 15:31:36 01/25/2006 15:51:36 01/25/2006 15:31:37 System.Web.SessionState.SessionStateItem:iw0thd55js0d44qjm2cwao45 System.Web.SessionState.InProcSessionState 0x02202cfc 0x0c2eaeb4 NotRemovable 01/25/2006 15:31:36 01/25/2006 15:51:36 01/25/2006 15:31:37 System.Web.SessionState.SessionStateItem:2r1dtbzh00tanc451pejqqrl System.Web.SessionState.InProcSessionState … And finally, if you want to drill down to what cache objects were largest you can use the .foreach again 0:023> .foreach (cachedObject {!dumpaspnetcache -short}) {!objsize ${cachedObject}} sizeof(0x626b1b8) = 12,000,772 (0xb71e04) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x6268284) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x6265318) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x62623ac) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x625f440) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x625c4d4) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x6259568) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x62565fc) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x6253690) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x6250724) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x624d7b8) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x624a84c) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x62485e8) = 12,000,772 (0xb71e04) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x6248334) = 12,000,772 (0xb71e04) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x6248080) = 12,000,772 (0xb71e04) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x6247dcc) = 12,000,772 (0xb71e04) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x62479a8) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x62476bc) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState) … So in this particular case, most of our cache is used up by large sessions (~12 and 8 MB each), given this knowledge we would of course drill down into the sessions and find out what is in them, but I’ll leave that for my next post. Phew!:) At last I have the prerequisites for the post I’m writing on sessions Until then…
I was writing a post on debugging a high memory usage problem caused by storing too much in session scope, but I realized I got in to tangent discussions all the time so I decided to create a separate post on caching first.
Caching is by far the most common cause of high memory problems. It’s an understatement to say that the cache is often overused.
As with everything else nothing comes for free, so the performance you gain by storing items in cache has to be compared to the performance you loose by having to scavenge this memory. Be careful with the design and set expiration times etc. appropriately, and make sure that what you cache will actually be accessed enough times to make it worth while.
Sos.dll has some nice features for examining your cache through the !dumpaspnetcache command; however before we get in to that, let’s look at some commands to determine if you have a cache problem.
Where is the cache?
Each HttpRuntime (running application) has a reference to a System.Web.Caching.CacheMultiple object (the _cache field), where the cached objects for that application is stored. In the particular version I am running (1.1.4322.2300), the _cache field is located at offset 0xc from the address of the HttpRuntime object.
0:023> !do 0x061b47d4 Name: System.Web.HttpRuntime MethodTable 0x01b29dfc EEClass 0x01ee8314 Size 116(0x74) bytes GC Generation: 2 mdToken: 0x02000079 (c:\windows\assembly\gac\system.web\1.0.5000.0__b03f5f7f11d50a3a\system.web.dll) FieldDesc*: 0x01b29554 MT Field Offset Type Attr Value Name 0x01b29dfc 0x4000683 0x4 CLASS instance 0x00000000 _namedPermissionSet 0x01b29dfc 0x4000684 0x8 CLASS instance 0x061b4950 _fcm 0x01b29dfc 0x4000685 0xc CLASS instance 0x061b4ca0 _cache 0x01b29dfc 0x4000686 0x54 System.Boolean instance 0 _isOnUNCShare 0x01b29dfc 0x4000687 0x10 CLASS instance 0x061b6e18 _profiler 0x01b29dfc 0x4000688 0x14 CLASS instance 0x061b6e34 _timeoutManager 0x01b29dfc 0x4000689 0x18 CLASS instance 0x021b368c _requestQueue 0x01b29dfc 0x400068a 0x55 System.Boolean instance 0 _apartmentThreading 0x01b29dfc 0x400068b 0x56 System.Boolean instance 0 _beforeFirstRequest 0x01b29dfc 0x400068c 0x60 VALUETYPE instance start at 0x061b4834 _firstRequestStartTime 0x01b29dfc 0x400068d 0x57 System.Boolean instance 1 _firstRequestCompleted 0x01b29dfc 0x400068e 0x58 System.Boolean instance 0 _userForcedShutdown 0x01b29dfc 0x400068f 0x59 System.Boolean instance 1 _configInited 0x01b29dfc 0x4000690 0x50 System.Int32 instance 1 _activeRequestCount 0x01b29dfc 0x4000691 0x5a System.Boolean instance 0 _someBatchCompilationStarted 0x01b29dfc 0x4000692 0x5b System.Boolean instance 0 _shutdownInProgress 0x01b29dfc 0x4000693 0x1c CLASS instance 0x00000000 _shutDownStack 0x01b29dfc 0x4000694 0x20 CLASS instance 0x00000000 _shutDownMessage 0x01b29dfc 0x4000695 0x68 VALUETYPE instance start at 0x061b483c _lastShutdownAttemptTime 0x01b29dfc 0x4000696 0x5c System.Boolean instance 1 _enableHeaderChecking 0x01b29dfc 0x4000697 0x24 CLASS instance 0x061b7014 _handlerCompletionCallback 0x01b29dfc 0x4000698 0x28 CLASS instance 0x061b7030 _asyncEndOfSendCallback 0x01b29dfc 0x4000699 0x2c CLASS instance 0x061b704c _appDomainUnloadallback 0x01b29dfc 0x400069a 0x30 CLASS instance 0x00000000 _initializationError 0x01b29dfc 0x400069b 0x34 CLASS instance 0x00000000 _appDomainShutdownTimer 0x01b29dfc 0x400069c 0x38 CLASS instance 0x061d4b24 _codegenDir 0x01b29dfc 0x400069d 0x3c CLASS instance 0x02188f1c _appDomainAppId 0x01b29dfc 0x400069e 0x40 CLASS instance 0x02188f68 _appDomainAppPath 0x01b29dfc 0x400069f 0x44 CLASS instance 0x0218a434 _appDomainAppVPath …
I’m not going to go in too much in the nitty-gritty but basically this CacheMultiple has a _caches Object[] member variable, which contains some CacheSingle’s and they each have an _entries membervariable which contains the actual CacheEntries, if you want to delve in to them manually. Luckily for us, !dumpaspnetcache automates this as you will see later.
Getting the size of the cache
The reason that I still mention where the cache is stored, is because with this knowledge we can now compose a command that goes through all our HttpRuntimes and dumps out the name of the app and how much it stores in cache.
The pseudo-code for doing this would be:
For each (HttpRuntime in (all the runtimes on the heap)) Print “*******************” //to divide the entries Dump out the AppDomainID Print “Cache Size: “ Dump out the size of the cache End for each
Translated to windbg terms:
To find all the runtimes on the heap, first find the methodtable for System.Web.HttpRuntime and then run !dumpheap –mt . First run !dumpheap –type HttpRuntime which gives us all objects whose name contains HttpRuntime and their method tables (MT). 0:023> !dumpheap -type HttpRuntime Using our cache to search the heap. Address MT Size Gen 0x061d4dbc 0x020c0010 12 -1 System.Web.Configuration.HttpRuntimeConfigurationHandler 0x021b321c 0x0209ff40 52 -1 System.Web.Configuration.HttpRuntimeConfig 0x061b47d4 0x01b29dfc 116 -1 System.Web.HttpRuntime Statistics: MT Count TotalSize Class Name 0x020c0010 1 12 System.Web.Configuration.HttpRuntimeConfigurationHandler 0x0209ff40 1 52 System.Web.Configuration.HttpRuntimeConfig 0x01b29dfc 1 116 System.Web.HttpRuntime Total 3 objects, Total size: 180 And then do !dumpheap –mt 0x01b29dfc –short 0:023> !dumpheap -mt 0x01b29dfc -short 0x61b47d4 The short switch gives us only the address of the objects so we can use it more easily with .foreach. In this case I had only one HttpRuntime because I have only one vdir that I have accessed since the start of the process. The AppDomainId (IIS name for the app) is stored in the _appDomainAppId member variable for HttpRuntime at offset 0x3c... 0x01b29dfc 0x400069d 0x3c CLASS instance 0x02188f1c _appDomainAppId ...so to dump this out given a HttpRuntime address you would run 0:023> !dumpobj poi(0x61b47d4+0x3c) String: /LM/w3svc/1/root/MemoryIssues The poi operator gives us a dword stored at the address we are giving it, i.e. in this case it will give us what is stored at 0x61b47d4+0x3c, where 0x61b47d4 is the address of the HttpRuntime, so it will give us 0x02188f1c, which is the address of _appDomainAppId from above. The reason we use this technique is so we can automate it to run on all the HttpRuntime objects on the heap, since with .foreach where we will only know the address of each HttpRuntime object. Much the same way we get the size of the cache by running !objsize on the _cache member variable 0x01b29dfc 0x4000685 0xc CLASS instance 0x061b4ca0 _cache0:023> !objsize poi(0x61b47d4+0xc) sizeof(0x61b4ca0) = 1,336,709,120 (0x4fac9000) bytes (System.Web.Caching.CacheMultiple) Now we have all the components to run a command that will give us the name and the cache size for all HttpRuntime objects. A short comment on .foreach before we continue... In its simplest form the .foreach command looks like this: .foreach ( Variable { InCommands } ) { OutCommands } Where InCommands is the commands that will give you the array/collection to loop through, and the OutCommands is a semicolon separated list of what commands you want to run each iteration. Our command will look like this: .foreach (runtime {!dumpheap -mt 0x01b29dfc -short}){.echo ******************;!dumpobj poi(${runtime}+0x3c);.echo cache size:;!objsize poi(${runtime}+0xc) } We have to enclose runtime (our foreach variable) in ${} so that it doesn’t get interpreted as a symbol. And the output looks like this: ****************** String: /LM/w3svc/1/root/MemoryIssues cache size: sizeof(0x61b4ca0) = 1,336,709,120 (0x4fac9000) bytes (System.Web.Caching.CacheMultiple) If I would have had more than one application it would have listed all of them one after another, separating them with a line of *** So here we can see that my one application MemoryIssues, has a cache that contains 1,336,709,120 bytes. WOW!!! that's a lot!!! We definitely seem to have some sort of cache problem. What is in my cache? Using !dumpaspnetcache… !dumpaspnetcache –stat gives you a nice overview over what types of objects you are storing in cache. (check the post on !dumpheap –stat to understand TotalSize). 0:023> !dumpaspnetcache -stat Going to dump the ASP.NET Cache. MT Count TotalSize Class Name 0x0211cc9c 1 20 System.Web.Security.FileSecurityDescriptorWrapper 0x020c242c 2 56 System.Web.UI.ParserCacheItem 0x0206c66c 5 260 System.Web.Configuration.HttpConfigurationRecord 0x0c2e7014 1 316 System.Web.Mobile.MobileCapabilities 0x79b94638 4 376 System.String 0x0c2eaeb4 151 7,248 System.Web.SessionState.InProcSessionState Total 164 objects, Total size: 8,276 !dumpaspnetcache –s gives you a summary of your objects with priority (removable or NotRemovable etc.) along with create time, expires and last updated which can be useful to see if you are caching something for too long. The ASP.NET 1.1 cache uses an LRU (Least Recently Used) model for removing objects from cache if memory is needed, for the objects that are not marked NotRemovable. In this example we can see for example some InProcSessionState items (sessions) that started around 15:31:36 and expire on 15:51:36 (after 20 min) and during that time they can not be removed (NotRemovable) by someone trying to clean up the cache to lower memory usage. 0:023> !dumpaspnetcache -s Going to dump the ASP.NET Cache. Address MT Priority Create Time Expires Last Updated Key Class Name 0x061d2658 0x0206c66c Normal 01/25/2006 15:31:35 12/31/9999 23:59:59 01/25/2006 15:31:35 System.Web.HttpConfig: System.Web.Configuration.HttpConfigurationRecord 0x021b1acc 0x0206c66c Normal 01/25/2006 15:31:35 12/31/9999 23:59:59 01/25/2006 15:31:35 System.Web.HttpConfig:/MEMORYISSUES System.Web.Configuration.HttpConfigurationRecord 0x061b3f7c 0x79b94638 Normal 01/25/2006 15:31:35 01/25/2006 15:41:35 01/25/2006 15:31:35 ISAPIWorkerRequest.MapPath:/MemoryIssues/StoringStuffInSessions.aspx System.String 0x021b503c 0x0206c66c Normal 01/25/2006 15:31:35 12/31/9999 23:59:59 01/25/2006 15:31:36 System.Web.HttpConfig:/MEMORYISSUES/STORINGSTUFFINSESSIONS.ASPX System.Web.Configuration.HttpConfigurationRecord 0x021b80b4 0x0211cc9c Normal 01/25/2006 15:31:36 12/31/9999 23:59:59 01/25/2006 15:31:36 System.Web.Security.FileSecurityDescriptorWrapper:c:\inetpub\wwwroot\MemoryIssues\StoringStuffInSessions.aspx System.Web.Security.FileSecurityDescriptorWrapper 0x021bc74c 0x020c242c NotRemovable 01/25/2006 15:31:36 12/31/9999 23:59:59 01/25/2006 15:31:36 System.Web.UI.TemplateParser:/MemoryIssues/StoringStuffInSessions.aspx System.Web.UI.ParserCacheItem 0x022028fc 0x0c2eaeb4 NotRemovable 01/25/2006 15:31:36 01/25/2006 15:51:36 01/25/2006 15:31:37 System.Web.SessionState.SessionStateItem:x4ud0tb53l2wfo554aw0yny4 System.Web.SessionState.InProcSessionState 0x02202a10 0x0c2eaeb4 NotRemovable 01/25/2006 15:31:36 01/25/2006 15:51:36 01/25/2006 15:31:37 System.Web.SessionState.SessionStateItem:iw0thd55js0d44qjm2cwao45 System.Web.SessionState.InProcSessionState 0x02202cfc 0x0c2eaeb4 NotRemovable 01/25/2006 15:31:36 01/25/2006 15:51:36 01/25/2006 15:31:37 System.Web.SessionState.SessionStateItem:2r1dtbzh00tanc451pejqqrl System.Web.SessionState.InProcSessionState … And finally, if you want to drill down to what cache objects were largest you can use the .foreach again 0:023> .foreach (cachedObject {!dumpaspnetcache -short}) {!objsize ${cachedObject}} sizeof(0x626b1b8) = 12,000,772 (0xb71e04) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x6268284) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x6265318) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x62623ac) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x625f440) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x625c4d4) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x6259568) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x62565fc) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x6253690) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x6250724) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x624d7b8) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x624a84c) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x62485e8) = 12,000,772 (0xb71e04) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x6248334) = 12,000,772 (0xb71e04) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x6248080) = 12,000,772 (0xb71e04) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x6247dcc) = 12,000,772 (0xb71e04) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x62479a8) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x62476bc) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState) … So in this particular case, most of our cache is used up by large sessions (~12 and 8 MB each), given this knowledge we would of course drill down into the sessions and find out what is in them, but I’ll leave that for my next post. Phew!:) At last I have the prerequisites for the post I’m writing on sessions Until then…
First run !dumpheap –type HttpRuntime which gives us all objects whose name contains HttpRuntime and their method tables (MT).
0:023> !dumpheap -type HttpRuntime Using our cache to search the heap. Address MT Size Gen 0x061d4dbc 0x020c0010 12 -1 System.Web.Configuration.HttpRuntimeConfigurationHandler 0x021b321c 0x0209ff40 52 -1 System.Web.Configuration.HttpRuntimeConfig 0x061b47d4 0x01b29dfc 116 -1 System.Web.HttpRuntime Statistics: MT Count TotalSize Class Name 0x020c0010 1 12 System.Web.Configuration.HttpRuntimeConfigurationHandler 0x0209ff40 1 52 System.Web.Configuration.HttpRuntimeConfig 0x01b29dfc 1 116 System.Web.HttpRuntime Total 3 objects, Total size: 180
And then do !dumpheap –mt 0x01b29dfc –short
0:023> !dumpheap -mt 0x01b29dfc -short 0x61b47d4
The short switch gives us only the address of the objects so we can use it more easily with .foreach. In this case I had only one HttpRuntime because I have only one vdir that I have accessed since the start of the process.
The AppDomainId (IIS name for the app) is stored in the _appDomainAppId member variable for HttpRuntime at offset 0x3c...
0x01b29dfc 0x400069d 0x3c CLASS instance 0x02188f1c _appDomainAppId
...so to dump this out given a HttpRuntime address you would run
0:023> !dumpobj poi(0x61b47d4+0x3c) String: /LM/w3svc/1/root/MemoryIssues
The poi operator gives us a dword stored at the address we are giving it, i.e. in this case it will give us what is stored at 0x61b47d4+0x3c, where 0x61b47d4 is the address of the HttpRuntime, so it will give us 0x02188f1c, which is the address of _appDomainAppId from above.
The reason we use this technique is so we can automate it to run on all the HttpRuntime objects on the heap, since with .foreach where we will only know the address of each HttpRuntime object.
Much the same way we get the size of the cache by running !objsize on the _cache member variable
0x01b29dfc 0x4000685 0xc CLASS instance 0x061b4ca0 _cache
0:023> !objsize poi(0x61b47d4+0xc) sizeof(0x61b4ca0) = 1,336,709,120 (0x4fac9000) bytes (System.Web.Caching.CacheMultiple)
Now we have all the components to run a command that will give us the name and the cache size for all HttpRuntime objects.
A short comment on .foreach before we continue...
In its simplest form the .foreach command looks like this:
.foreach ( Variable { InCommands } ) { OutCommands }
Where InCommands is the commands that will give you the array/collection to loop through, and the OutCommands is a semicolon separated list of what commands you want to run each iteration.
Our command will look like this:
.foreach (runtime {!dumpheap -mt 0x01b29dfc -short}){.echo ******************;!dumpobj poi(${runtime}+0x3c);.echo cache size:;!objsize poi(${runtime}+0xc) }
We have to enclose runtime (our foreach variable) in ${} so that it doesn’t get interpreted as a symbol.
And the output looks like this:
****************** String: /LM/w3svc/1/root/MemoryIssues cache size: sizeof(0x61b4ca0) = 1,336,709,120 (0x4fac9000) bytes (System.Web.Caching.CacheMultiple)
If I would have had more than one application it would have listed all of them one after another, separating them with a line of ***
So here we can see that my one application MemoryIssues, has a cache that contains 1,336,709,120 bytes. WOW!!! that's a lot!!! We definitely seem to have some sort of cache problem.
What is in my cache? Using !dumpaspnetcache…
!dumpaspnetcache –stat gives you a nice overview over what types of objects you are storing in cache. (check the post on !dumpheap –stat to understand TotalSize).
0:023> !dumpaspnetcache -stat Going to dump the ASP.NET Cache. MT Count TotalSize Class Name 0x0211cc9c 1 20 System.Web.Security.FileSecurityDescriptorWrapper 0x020c242c 2 56 System.Web.UI.ParserCacheItem 0x0206c66c 5 260 System.Web.Configuration.HttpConfigurationRecord 0x0c2e7014 1 316 System.Web.Mobile.MobileCapabilities 0x79b94638 4 376 System.String 0x0c2eaeb4 151 7,248 System.Web.SessionState.InProcSessionState Total 164 objects, Total size: 8,276
!dumpaspnetcache –s gives you a summary of your objects with priority (removable or NotRemovable etc.) along with create time, expires and last updated which can be useful to see if you are caching something for too long.
The ASP.NET 1.1 cache uses an LRU (Least Recently Used) model for removing objects from cache if memory is needed, for the objects that are not marked NotRemovable.
In this example we can see for example some InProcSessionState items (sessions) that started around 15:31:36 and expire on 15:51:36 (after 20 min) and during that time they can not be removed (NotRemovable) by someone trying to clean up the cache to lower memory usage.
0:023> !dumpaspnetcache -s Going to dump the ASP.NET Cache. Address MT Priority Create Time Expires Last Updated Key Class Name 0x061d2658 0x0206c66c Normal 01/25/2006 15:31:35 12/31/9999 23:59:59 01/25/2006 15:31:35 System.Web.HttpConfig: System.Web.Configuration.HttpConfigurationRecord 0x021b1acc 0x0206c66c Normal 01/25/2006 15:31:35 12/31/9999 23:59:59 01/25/2006 15:31:35 System.Web.HttpConfig:/MEMORYISSUES System.Web.Configuration.HttpConfigurationRecord 0x061b3f7c 0x79b94638 Normal 01/25/2006 15:31:35 01/25/2006 15:41:35 01/25/2006 15:31:35 ISAPIWorkerRequest.MapPath:/MemoryIssues/StoringStuffInSessions.aspx System.String 0x021b503c 0x0206c66c Normal 01/25/2006 15:31:35 12/31/9999 23:59:59 01/25/2006 15:31:36 System.Web.HttpConfig:/MEMORYISSUES/STORINGSTUFFINSESSIONS.ASPX System.Web.Configuration.HttpConfigurationRecord 0x021b80b4 0x0211cc9c Normal 01/25/2006 15:31:36 12/31/9999 23:59:59 01/25/2006 15:31:36 System.Web.Security.FileSecurityDescriptorWrapper:c:\inetpub\wwwroot\MemoryIssues\StoringStuffInSessions.aspx System.Web.Security.FileSecurityDescriptorWrapper 0x021bc74c 0x020c242c NotRemovable 01/25/2006 15:31:36 12/31/9999 23:59:59 01/25/2006 15:31:36 System.Web.UI.TemplateParser:/MemoryIssues/StoringStuffInSessions.aspx System.Web.UI.ParserCacheItem 0x022028fc 0x0c2eaeb4 NotRemovable 01/25/2006 15:31:36 01/25/2006 15:51:36 01/25/2006 15:31:37 System.Web.SessionState.SessionStateItem:x4ud0tb53l2wfo554aw0yny4 System.Web.SessionState.InProcSessionState 0x02202a10 0x0c2eaeb4 NotRemovable 01/25/2006 15:31:36 01/25/2006 15:51:36 01/25/2006 15:31:37 System.Web.SessionState.SessionStateItem:iw0thd55js0d44qjm2cwao45 System.Web.SessionState.InProcSessionState 0x02202cfc 0x0c2eaeb4 NotRemovable 01/25/2006 15:31:36 01/25/2006 15:51:36 01/25/2006 15:31:37 System.Web.SessionState.SessionStateItem:2r1dtbzh00tanc451pejqqrl System.Web.SessionState.InProcSessionState …
And finally, if you want to drill down to what cache objects were largest you can use the .foreach again
0:023> .foreach (cachedObject {!dumpaspnetcache -short}) {!objsize ${cachedObject}} sizeof(0x626b1b8) = 12,000,772 (0xb71e04) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x6268284) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x6265318) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x62623ac) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x625f440) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x625c4d4) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x6259568) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x62565fc) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x6253690) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x6250724) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x624d7b8) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x624a84c) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x62485e8) = 12,000,772 (0xb71e04) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x6248334) = 12,000,772 (0xb71e04) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x6248080) = 12,000,772 (0xb71e04) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x6247dcc) = 12,000,772 (0xb71e04) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x62479a8) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState) sizeof(0x62476bc) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState) …
So in this particular case, most of our cache is used up by large sessions (~12 and 8 MB each), given this knowledge we would of course drill down into the sessions and find out what is in them, but I’ll leave that for my next post.
Phew!:) At last I have the prerequisites for the post I’m writing on sessions
Until then…
This is part four in the ASP.Net tips series . In particular, this is a follow-up to, " ASP.Net Quick
In the .Net from 2.0 there is no !dumpaspnetcache command. Do you know if its called something different now, or if there is a way to do something similar?
Hi Jason,
There isn't a !dumpaspnetcache command yet in 2.0 so the only way you can do it is through manually dumping out hte contents of System.Web.Caching.Cache but a few common tasks so you dont have to run though the whole ordeal of dumping it out, would be to do the following
1. Find the System.Web.Caching.Cache MT in !dumpheap -stat
2. !dumpheap -mt on that MT
3. !objsize on the objects so that you can see how much you are saving in cache
For session state, find the MT for the InProcSessionState objects and run
.foreach (Session {!dumpheap -mt <MT> -short}){!objsize ${Session}}
Which will give you the size of the individual Sessions and then from there you can drill down into the particular session object that is causing you grief.
Hope this helps,
Tess
This is a bit of a continuation of ASP.NET Memory Issue: High memory usage in a 64bit w3wp.exe process