I was writing a post on debugging a high memory usage problem caused by storing too much in session scope, but I realized I got in to tangent discussions all the time so I decided to create a separate post on caching first.

Caching is by far the most common cause of high memory problems. It’s an understatement to say that the cache is often overused.

As with everything else nothing comes for free, so the performance you gain by storing items in cache has to be compared to the performance you loose by having to scavenge this memory. Be careful with the design and set expiration times etc. appropriately, and make sure that what you cache will actually be accessed enough times to make it worth while.

Sos.dll has some nice features for examining your cache through the !dumpaspnetcache command; however before we get in to that, let’s look at some commands to determine if you have a cache problem.

Where is the cache?

Each HttpRuntime (running application) has a reference to a System.Web.Caching.CacheMultiple object (the _cache field), where the cached objects for that application is stored. In the particular version I am running (1.1.4322.2300), the _cache field is located at offset 0xc from the address of the HttpRuntime object. 

0:023> !do 0x061b47d4 
Name: System.Web.HttpRuntime
MethodTable 0x01b29dfc
EEClass 0x01ee8314
Size 116(0x74) bytes
GC Generation: 2
mdToken: 0x02000079  (c:\windows\assembly\gac\system.web\1.0.5000.0__b03f5f7f11d50a3a\system.web.dll)
FieldDesc*: 0x01b29554
        MT      Field     Offset                 Type       Attr      Value Name
0x01b29dfc 0x4000683      0x4                CLASS   instance 0x00000000 _namedPermissionSet
0x01b29dfc 0x4000684      0x8                CLASS   instance 0x061b4950 _fcm
0x01b29dfc 0x4000685      0xc                CLASS   instance 0x061b4ca0 _cache
0x01b29dfc 0x4000686     0x54       System.Boolean   instance 0 _isOnUNCShare
0x01b29dfc 0x4000687     0x10                CLASS   instance 0x061b6e18 _profiler
0x01b29dfc 0x4000688     0x14                CLASS   instance 0x061b6e34 _timeoutManager
0x01b29dfc 0x4000689     0x18                CLASS   instance 0x021b368c _requestQueue
0x01b29dfc 0x400068a     0x55       System.Boolean   instance 0 _apartmentThreading
0x01b29dfc 0x400068b     0x56       System.Boolean   instance 0 _beforeFirstRequest
0x01b29dfc 0x400068c     0x60            VALUETYPE   instance start at 0x061b4834 _firstRequestStartTime
0x01b29dfc 0x400068d     0x57       System.Boolean   instance 1 _firstRequestCompleted
0x01b29dfc 0x400068e     0x58       System.Boolean   instance 0 _userForcedShutdown
0x01b29dfc 0x400068f     0x59       System.Boolean   instance 1 _configInited
0x01b29dfc 0x4000690     0x50         System.Int32   instance 1 _activeRequestCount
0x01b29dfc 0x4000691     0x5a       System.Boolean   instance 0 _someBatchCompilationStarted
0x01b29dfc 0x4000692     0x5b       System.Boolean   instance 0 _shutdownInProgress
0x01b29dfc 0x4000693     0x1c                CLASS   instance 0x00000000 _shutDownStack
0x01b29dfc 0x4000694     0x20                CLASS   instance 0x00000000 _shutDownMessage
0x01b29dfc 0x4000695     0x68            VALUETYPE   instance start at 0x061b483c _lastShutdownAttemptTime
0x01b29dfc 0x4000696     0x5c       System.Boolean   instance 1 _enableHeaderChecking
0x01b29dfc 0x4000697     0x24                CLASS   instance 0x061b7014 _handlerCompletionCallback
0x01b29dfc 0x4000698     0x28                CLASS   instance 0x061b7030 _asyncEndOfSendCallback
0x01b29dfc 0x4000699     0x2c                CLASS   instance 0x061b704c _appDomainUnloadallback
0x01b29dfc 0x400069a     0x30                CLASS   instance 0x00000000 _initializationError
0x01b29dfc 0x400069b     0x34                CLASS   instance 0x00000000 _appDomainShutdownTimer
0x01b29dfc 0x400069c     0x38                CLASS   instance 0x061d4b24 _codegenDir
0x01b29dfc 0x400069d     0x3c                CLASS   instance 0x02188f1c _appDomainAppId
0x01b29dfc 0x400069e     0x40                CLASS   instance 0x02188f68 _appDomainAppPath
0x01b29dfc 0x400069f     0x44                CLASS   instance 0x0218a434 _appDomainAppVPath
…

I’m not going to go in too much in the nitty-gritty but basically this CacheMultiple has a _caches Object[] member variable, which contains some CacheSingle’s and they each have an _entries membervariable which contains the actual CacheEntries, if you want to delve in to them manually. Luckily for us, !dumpaspnetcache automates this as you will see later.

Getting the size of the cache

The reason that I still mention where the cache is stored, is because with this knowledge we can now compose a command that goes through all our HttpRuntimes and dumps out the name of the app and how much it stores in cache.

The pseudo-code for doing this would be: 

For each (HttpRuntime in (all the runtimes on the heap))
           Print “*******************” //to divide the entries
           Dump out the AppDomainID
           Print “Cache Size: “   
           Dump out the size of the cache 
End for each

Translated to windbg terms:

To find all the runtimes on the heap, first find the methodtable for System.Web.HttpRuntime and then run !dumpheap –mt .

First run !dumpheap –type HttpRuntime which gives us all objects whose name contains HttpRuntime and their method tables (MT). 

0:023> !dumpheap -type HttpRuntime
Using our cache to search the heap.
   Address         MT     Size  Gen
0x061d4dbc 0x020c0010       12   -1 System.Web.Configuration.HttpRuntimeConfigurationHandler 
0x021b321c 0x0209ff40       52   -1 System.Web.Configuration.HttpRuntimeConfig 
0x061b47d4 0x01b29dfc      116   -1 System.Web.HttpRuntime 
Statistics:
        MT      Count     TotalSize Class Name
0x020c0010          1            12 System.Web.Configuration.HttpRuntimeConfigurationHandler
0x0209ff40          1            52 System.Web.Configuration.HttpRuntimeConfig
0x01b29dfc          1           116 System.Web.HttpRuntime
Total 3 objects, Total size: 180

And then do !dumpheap –mt 0x01b29dfc –short 

0:023> !dumpheap -mt 0x01b29dfc -short
0x61b47d4

The short switch gives us only the address of the objects so we can use it more easily with .foreach. In this case I had only one HttpRuntime because I have only one vdir that I have accessed since the start of the process.

The AppDomainId (IIS name for the app) is stored in the _appDomainAppId member variable for HttpRuntime at offset 0x3c... 

0x01b29dfc 0x400069d     0x3c                CLASS   instance 0x02188f1c _appDomainAppId

...so to dump this out given a HttpRuntime address you would run 

0:023> !dumpobj poi(0x61b47d4+0x3c)
String: /LM/w3svc/1/root/MemoryIssues

The poi operator gives us a dword stored at the address we are giving it, i.e. in this case it will give us what is stored at 0x61b47d4+0x3c, where 0x61b47d4 is the address of the HttpRuntime, so it will give us 0x02188f1c, which is the address of _appDomainAppId from above.

The reason we use this technique is so we can automate it to run on all the HttpRuntime objects on the heap, since with .foreach where we will only know the address of each HttpRuntime object.

Much the same way we get the size of the cache by running !objsize on the _cache member variable 

0x01b29dfc 0x4000685      0xc                CLASS   instance 0x061b4ca0 _cache
0:023> !objsize poi(0x61b47d4+0xc)
sizeof(0x61b4ca0) = 1,336,709,120 (0x4fac9000) bytes (System.Web.Caching.CacheMultiple)

Now we have all the components to run a command that will give us the name and the cache size for all HttpRuntime objects.

A short comment on .foreach before we continue...

In its simplest form the .foreach command looks like this: 

.foreach ( Variable  { InCommands } ) { 
OutCommands 
}

Where InCommands is the commands that will give you the array/collection to loop through, and the OutCommands is a semicolon separated list of what commands you want to run each iteration.

Our command will look like this: 

.foreach (runtime {!dumpheap -mt 0x01b29dfc -short}){.echo ******************;!dumpobj poi(${runtime}+0x3c);.echo cache size:;!objsize poi(${runtime}+0xc) }

We have to enclose runtime (our foreach variable) in ${} so that it doesn’t get interpreted as a symbol.

And the output looks like this: 

******************
String: /LM/w3svc/1/root/MemoryIssues

cache size:
sizeof(0x61b4ca0) = 1,336,709,120 (0x4fac9000) bytes (System.Web.Caching.CacheMultiple)

If I would have had more than one application it would have listed all of them one after another, separating them with a line of ***

So here we can see that my one application MemoryIssues, has a cache that contains 1,336,709,120 bytes. WOW!!! that's a lot!!! We definitely seem to have some sort of cache problem.

What is in my cache? Using !dumpaspnetcache…

!dumpaspnetcache –stat gives you a nice overview over what types of objects you are storing in cache. (check the post on !dumpheap –stat to understand TotalSize). 

0:023> !dumpaspnetcache -stat
Going to dump the ASP.NET Cache.
        MT      Count     TotalSize Class Name
0x0211cc9c          1            20 System.Web.Security.FileSecurityDescriptorWrapper
0x020c242c          2            56 System.Web.UI.ParserCacheItem
0x0206c66c          5           260 System.Web.Configuration.HttpConfigurationRecord
0x0c2e7014          1           316 System.Web.Mobile.MobileCapabilities
0x79b94638          4           376 System.String
0x0c2eaeb4        151         7,248 System.Web.SessionState.InProcSessionState
Total 164 objects, Total size: 8,276

!dumpaspnetcache –s gives you a summary of your objects with priority (removable or NotRemovable etc.) along with create time, expires and last updated which can be useful to see if you are caching something for too long.

The ASP.NET 1.1 cache uses an LRU (Least Recently Used) model for removing objects from cache if memory is needed, for the objects that are not marked NotRemovable.

In this example we can see for example some InProcSessionState items (sessions) that started around 15:31:36 and expire on 15:51:36 (after 20 min) and during that time they can not be removed (NotRemovable) by someone trying to clean up the cache to lower memory usage. 

0:023> !dumpaspnetcache -s
Going to dump the ASP.NET Cache.
   Address	        MT	    Priority	        Create Time	            Expires	       Last Updated	Key	Class Name
0x061d2658	0x0206c66c	      Normal	01/25/2006 15:31:35	12/31/9999 23:59:59	01/25/2006 15:31:35	System.Web.HttpConfig:	System.Web.Configuration.HttpConfigurationRecord
0x021b1acc	0x0206c66c	      Normal	01/25/2006 15:31:35	12/31/9999 23:59:59	01/25/2006 15:31:35	System.Web.HttpConfig:/MEMORYISSUES	System.Web.Configuration.HttpConfigurationRecord
0x061b3f7c	0x79b94638	      Normal	01/25/2006 15:31:35	01/25/2006 15:41:35	01/25/2006 15:31:35	ISAPIWorkerRequest.MapPath:/MemoryIssues/StoringStuffInSessions.aspx	System.String
0x021b503c	0x0206c66c	      Normal	01/25/2006 15:31:35	12/31/9999 23:59:59	01/25/2006 15:31:36	System.Web.HttpConfig:/MEMORYISSUES/STORINGSTUFFINSESSIONS.ASPX	System.Web.Configuration.HttpConfigurationRecord
0x021b80b4	0x0211cc9c	      Normal	01/25/2006 15:31:36	12/31/9999 23:59:59	01/25/2006 15:31:36	System.Web.Security.FileSecurityDescriptorWrapper:c:\inetpub\wwwroot\MemoryIssues\StoringStuffInSessions.aspx	System.Web.Security.FileSecurityDescriptorWrapper
0x021bc74c	0x020c242c	NotRemovable	01/25/2006 15:31:36	12/31/9999 23:59:59	01/25/2006 15:31:36	System.Web.UI.TemplateParser:/MemoryIssues/StoringStuffInSessions.aspx	System.Web.UI.ParserCacheItem
0x022028fc	0x0c2eaeb4	NotRemovable	01/25/2006 15:31:36	01/25/2006 15:51:36	01/25/2006 15:31:37	System.Web.SessionState.SessionStateItem:x4ud0tb53l2wfo554aw0yny4	System.Web.SessionState.InProcSessionState
0x02202a10	0x0c2eaeb4	NotRemovable	01/25/2006 15:31:36	01/25/2006 15:51:36	01/25/2006 15:31:37	System.Web.SessionState.SessionStateItem:iw0thd55js0d44qjm2cwao45	System.Web.SessionState.InProcSessionState
0x02202cfc	0x0c2eaeb4	NotRemovable	01/25/2006 15:31:36	01/25/2006 15:51:36	01/25/2006 15:31:37	System.Web.SessionState.SessionStateItem:2r1dtbzh00tanc451pejqqrl	System.Web.SessionState.InProcSessionState
…

And finally, if you want to drill down to what cache objects were largest you can use the .foreach again 

0:023> .foreach (cachedObject {!dumpaspnetcache -short}) {!objsize ${cachedObject}}
sizeof(0x626b1b8) = 12,000,772 (0xb71e04) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x6268284) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x6265318) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x62623ac) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x625f440) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x625c4d4) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x6259568) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x62565fc) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x6253690) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x6250724) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x624d7b8) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x624a84c) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x62485e8) = 12,000,772 (0xb71e04) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x6248334) = 12,000,772 (0xb71e04) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x6248080) = 12,000,772 (0xb71e04) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x6247dcc) = 12,000,772 (0xb71e04) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x62479a8) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x62476bc) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
…

So in this particular case, most of our cache is used up by large sessions (~12 and 8 MB each), given this knowledge we would of course drill down into the sessions and find out what is in them, but I’ll leave that for my next post.

Phew!:) At last I have the prerequisites for the post I’m writing on sessions

Until then…