If broken it is, fix it you should

Using the powers of the debugger to solve the problems of the world - and a bag of chips    by Tess Ferrandez, ASP.NET Escalation Engineer (Microsoft)

ASP.NET Memory - How much are you caching? + an example of .foreach

ASP.NET Memory - How much are you caching? + an example of .foreach

Rate This
  • Comments 19

I was writing a post on debugging a high memory usage problem caused by storing too much in session scope, but I realized I got in to tangent discussions all the time so I decided to create a separate post on caching first.

Caching is by far the most common cause of high memory problems. It’s an understatement to say that the cache is often overused.

As with everything else nothing comes for free, so the performance you gain by storing items in cache has to be compared to the performance you loose by having to scavenge this memory. Be careful with the design and set expiration times etc. appropriately, and make sure that what you cache will actually be accessed enough times to make it worth while.

Sos.dll has some nice features for examining your cache through the !dumpaspnetcache command; however before we get in to that, let’s look at some commands to determine if you have a cache problem.

Where is the cache?

Each HttpRuntime (running application) has a reference to a System.Web.Caching.CacheMultiple object (the _cache field), where the cached objects for that application is stored. In the particular version I am running (1.1.4322.2300), the _cache field is located at offset 0xc from the address of the HttpRuntime object.

0:023> !do 0x061b47d4 
Name: System.Web.HttpRuntime
MethodTable 0x01b29dfc
EEClass 0x01ee8314
Size 116(0x74) bytes
GC Generation: 2
mdToken: 0x02000079  (c:\windows\assembly\gac\system.web\1.0.5000.0__b03f5f7f11d50a3a\system.web.dll)
FieldDesc*: 0x01b29554
        MT      Field     Offset                 Type       Attr      Value Name
0x01b29dfc 0x4000683      0x4                CLASS   instance 0x00000000 _namedPermissionSet
0x01b29dfc 0x4000684      0x8                CLASS   instance 0x061b4950 _fcm
0x01b29dfc 0x4000685      0xc                CLASS   instance 0x061b4ca0 _cache
0x01b29dfc 0x4000686     0x54       System.Boolean   instance 0 _isOnUNCShare
0x01b29dfc 0x4000687     0x10                CLASS   instance 0x061b6e18 _profiler
0x01b29dfc 0x4000688     0x14                CLASS   instance 0x061b6e34 _timeoutManager
0x01b29dfc 0x4000689     0x18                CLASS   instance 0x021b368c _requestQueue
0x01b29dfc 0x400068a     0x55       System.Boolean   instance 0 _apartmentThreading
0x01b29dfc 0x400068b     0x56       System.Boolean   instance 0 _beforeFirstRequest
0x01b29dfc 0x400068c     0x60            VALUETYPE   instance start at 0x061b4834 _firstRequestStartTime
0x01b29dfc 0x400068d     0x57       System.Boolean   instance 1 _firstRequestCompleted
0x01b29dfc 0x400068e     0x58       System.Boolean   instance 0 _userForcedShutdown
0x01b29dfc 0x400068f     0x59       System.Boolean   instance 1 _configInited
0x01b29dfc 0x4000690     0x50         System.Int32   instance 1 _activeRequestCount
0x01b29dfc 0x4000691     0x5a       System.Boolean   instance 0 _someBatchCompilationStarted
0x01b29dfc 0x4000692     0x5b       System.Boolean   instance 0 _shutdownInProgress
0x01b29dfc 0x4000693     0x1c                CLASS   instance 0x00000000 _shutDownStack
0x01b29dfc 0x4000694     0x20                CLASS   instance 0x00000000 _shutDownMessage
0x01b29dfc 0x4000695     0x68            VALUETYPE   instance start at 0x061b483c _lastShutdownAttemptTime
0x01b29dfc 0x4000696     0x5c       System.Boolean   instance 1 _enableHeaderChecking
0x01b29dfc 0x4000697     0x24                CLASS   instance 0x061b7014 _handlerCompletionCallback
0x01b29dfc 0x4000698     0x28                CLASS   instance 0x061b7030 _asyncEndOfSendCallback
0x01b29dfc 0x4000699     0x2c                CLASS   instance 0x061b704c _appDomainUnloadallback
0x01b29dfc 0x400069a     0x30                CLASS   instance 0x00000000 _initializationError
0x01b29dfc 0x400069b     0x34                CLASS   instance 0x00000000 _appDomainShutdownTimer
0x01b29dfc 0x400069c     0x38                CLASS   instance 0x061d4b24 _codegenDir
0x01b29dfc 0x400069d     0x3c                CLASS   instance 0x02188f1c _appDomainAppId
0x01b29dfc 0x400069e     0x40                CLASS   instance 0x02188f68 _appDomainAppPath
0x01b29dfc 0x400069f     0x44                CLASS   instance 0x0218a434 _appDomainAppVPath
…

I’m not going to go in too much in the nitty-gritty but basically this CacheMultiple has a _caches Object[] member variable, which contains some CacheSingle’s and they each have an _entries membervariable which contains the actual CacheEntries, if you want to delve in to them manually. Luckily for us, !dumpaspnetcache automates this as you will see later.

Getting the size of the cache

The reason that I still mention where the cache is stored, is because with this knowledge we can now compose a command that goes through all our HttpRuntimes and dumps out the name of the app and how much it stores in cache.

The pseudo-code for doing this would be:

For each (HttpRuntime in (all the runtimes on the heap))
           Print “*******************” //to divide the entries
           Dump out the AppDomainID
           Print “Cache Size: “   
Dump out the size of the cache End for each

Translated to windbg terms:

To find all the runtimes on the heap, first find the methodtable for System.Web.HttpRuntime and then run !dumpheap –mt .

First run !dumpheap –type HttpRuntime which gives us all objects whose name contains HttpRuntime and their method tables (MT).

0:023> !dumpheap -type HttpRuntime
Using our cache to search the heap.
   Address         MT     Size  Gen
0x061d4dbc 0x020c0010       12   -1 System.Web.Configuration.HttpRuntimeConfigurationHandler 
0x021b321c 0x0209ff40       52   -1 System.Web.Configuration.HttpRuntimeConfig 
0x061b47d4 0x01b29dfc      116   -1 System.Web.HttpRuntime 
Statistics:
        MT      Count     TotalSize Class Name
0x020c0010          1            12 System.Web.Configuration.HttpRuntimeConfigurationHandler
0x0209ff40          1            52 System.Web.Configuration.HttpRuntimeConfig
0x01b29dfc          1           116 System.Web.HttpRuntime
Total 3 objects, Total size: 180

And then do !dumpheap –mt 0x01b29dfc –short

0:023> !dumpheap -mt 0x01b29dfc -short
0x61b47d4

The short switch gives us only the address of the objects so we can use it more easily with .foreach. In this case I had only one HttpRuntime because I have only one vdir that I have accessed since the start of the process.

The AppDomainId (IIS name for the app) is stored in the _appDomainAppId member variable for HttpRuntime at offset 0x3c...

0x01b29dfc 0x400069d     0x3c                CLASS   instance 0x02188f1c _appDomainAppId

...so to dump this out given a HttpRuntime address you would run

0:023> !dumpobj poi(0x61b47d4+0x3c)
String: /LM/w3svc/1/root/MemoryIssues

The poi operator gives us a dword stored at the address we are giving it, i.e. in this case it will give us what is stored at 0x61b47d4+0x3c, where 0x61b47d4 is the address of the HttpRuntime, so it will give us 0x02188f1c, which is the address of _appDomainAppId from above.

The reason we use this technique is so we can automate it to run on all the HttpRuntime objects on the heap, since with .foreach where we will only know the address of each HttpRuntime object.

Much the same way we get the size of the cache by running !objsize on the _cache member variable

0x01b29dfc 0x4000685      0xc                CLASS   instance 0x061b4ca0 _cache
0:023> !objsize poi(0x61b47d4+0xc)
sizeof(0x61b4ca0) = 1,336,709,120 (0x4fac9000) bytes (System.Web.Caching.CacheMultiple)

Now we have all the components to run a command that will give us the name and the cache size for all HttpRuntime objects.

A short comment on .foreach before we continue...

In its simplest form the .foreach command looks like this:

.foreach ( Variable  { InCommands } ) { 
OutCommands 
} 

Where InCommands is the commands that will give you the array/collection to loop through, and the OutCommands is a semicolon separated list of what commands you want to run each iteration.

Our command will look like this:

.foreach (runtime {!dumpheap -mt 0x01b29dfc -short}){.echo ******************;!dumpobj poi(${runtime}+0x3c);.echo cache size:;!objsize poi(${runtime}+0xc) }

We have to enclose runtime (our foreach variable) in ${} so that it doesn’t get interpreted as a symbol.

And the output looks like this:

******************
String: /LM/w3svc/1/root/MemoryIssues

cache size:
sizeof(0x61b4ca0) = 1,336,709,120 (0x4fac9000) bytes (System.Web.Caching.CacheMultiple)

If I would have had more than one application it would have listed all of them one after another, separating them with a line of ***

So here we can see that my one application MemoryIssues, has a cache that contains 1,336,709,120 bytes. WOW!!! that's a lot!!! We definitely seem to have some sort of cache problem.

What is in my cache? Using !dumpaspnetcache…

!dumpaspnetcache –stat gives you a nice overview over what types of objects you are storing in cache. (check the post on !dumpheap –stat to understand TotalSize).

0:023> !dumpaspnetcache -stat
Going to dump the ASP.NET Cache.
        MT      Count     TotalSize Class Name
0x0211cc9c          1            20 System.Web.Security.FileSecurityDescriptorWrapper
0x020c242c          2            56 System.Web.UI.ParserCacheItem
0x0206c66c          5           260 System.Web.Configuration.HttpConfigurationRecord
0x0c2e7014          1           316 System.Web.Mobile.MobileCapabilities
0x79b94638          4           376 System.String
0x0c2eaeb4        151         7,248 System.Web.SessionState.InProcSessionState
Total 164 objects, Total size: 8,276

!dumpaspnetcache –s gives you a summary of your objects with priority (removable or NotRemovable etc.) along with create time, expires and last updated which can be useful to see if you are caching something for too long.

The ASP.NET 1.1 cache uses an LRU (Least Recently Used) model for removing objects from cache if memory is needed, for the objects that are not marked NotRemovable.

In this example we can see for example some InProcSessionState items (sessions) that started around 15:31:36 and expire on 15:51:36 (after 20 min) and during that time they can not be removed (NotRemovable) by someone trying to clean up the cache to lower memory usage.

0:023> !dumpaspnetcache -s
Going to dump the ASP.NET Cache.
   Address	        MT	    Priority	        Create Time	            Expires	       Last Updated	Key	Class Name
0x061d2658	0x0206c66c	      Normal	01/25/2006 15:31:35	12/31/9999 23:59:59	01/25/2006 15:31:35	System.Web.HttpConfig:	System.Web.Configuration.HttpConfigurationRecord
0x021b1acc	0x0206c66c	      Normal	01/25/2006 15:31:35	12/31/9999 23:59:59	01/25/2006 15:31:35	System.Web.HttpConfig:/MEMORYISSUES	System.Web.Configuration.HttpConfigurationRecord
0x061b3f7c	0x79b94638	      Normal	01/25/2006 15:31:35	01/25/2006 15:41:35	01/25/2006 15:31:35	ISAPIWorkerRequest.MapPath:/MemoryIssues/StoringStuffInSessions.aspx	System.String
0x021b503c	0x0206c66c	      Normal	01/25/2006 15:31:35	12/31/9999 23:59:59	01/25/2006 15:31:36	System.Web.HttpConfig:/MEMORYISSUES/STORINGSTUFFINSESSIONS.ASPX	System.Web.Configuration.HttpConfigurationRecord
0x021b80b4	0x0211cc9c	      Normal	01/25/2006 15:31:36	12/31/9999 23:59:59	01/25/2006 15:31:36	System.Web.Security.FileSecurityDescriptorWrapper:c:\inetpub\wwwroot\MemoryIssues\StoringStuffInSessions.aspx	System.Web.Security.FileSecurityDescriptorWrapper
0x021bc74c	0x020c242c	NotRemovable	01/25/2006 15:31:36	12/31/9999 23:59:59	01/25/2006 15:31:36	System.Web.UI.TemplateParser:/MemoryIssues/StoringStuffInSessions.aspx	System.Web.UI.ParserCacheItem
0x022028fc	0x0c2eaeb4	NotRemovable	01/25/2006 15:31:36	01/25/2006 15:51:36	01/25/2006 15:31:37	System.Web.SessionState.SessionStateItem:x4ud0tb53l2wfo554aw0yny4	System.Web.SessionState.InProcSessionState
0x02202a10	0x0c2eaeb4	NotRemovable	01/25/2006 15:31:36	01/25/2006 15:51:36	01/25/2006 15:31:37	System.Web.SessionState.SessionStateItem:iw0thd55js0d44qjm2cwao45	System.Web.SessionState.InProcSessionState
0x02202cfc	0x0c2eaeb4	NotRemovable	01/25/2006 15:31:36	01/25/2006 15:51:36	01/25/2006 15:31:37	System.Web.SessionState.SessionStateItem:2r1dtbzh00tanc451pejqqrl	System.Web.SessionState.InProcSessionState
…

And finally, if you want to drill down to what cache objects were largest you can use the .foreach again

0:023> .foreach (cachedObject {!dumpaspnetcache -short}) {!objsize ${cachedObject}}
sizeof(0x626b1b8) = 12,000,772 (0xb71e04) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x6268284) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x6265318) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x62623ac) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x625f440) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x625c4d4) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x6259568) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x62565fc) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x6253690) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x6250724) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x624d7b8) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x624a84c) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x62485e8) = 12,000,772 (0xb71e04) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x6248334) = 12,000,772 (0xb71e04) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x6248080) = 12,000,772 (0xb71e04) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x6247dcc) = 12,000,772 (0xb71e04) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x62479a8) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x62476bc) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
…

So in this particular case, most of our cache is used up by large sessions (~12 and 8 MB each), given this knowledge we would of course drill down into the sessions and find out what is in them, but I’ll leave that for my next post.

Phew!:) At last I have the prerequisites for the post I’m writing on sessions

Until then…  

 





  • Wouldn't it be nice if we could hook into a static readonly Cache.TotalSize, Application.TotalSize and Session.TotalSize ??

    Karl
  • This post doesn't display well at 1024 width because of the fixed width no wrap fonts in your blue boxes, perhaps you should lower the font size 1 or 2 notches to be more 1024 width friendly.

    PS: I'm a page layout coder, 1024x1280.
  • Hi Travis,

    I'd be happy to change it, because i find it a bit annoying too that some boxes stretch way out on the right hand side... but unfortunately I know practically nothing about CSS:(

    my css override for the blue sections look like this

    pre.debug
    {
    font-family: Courier;
    font-size: 6pt;
    background-color: #99CCFF;
    text-align:left;
    }

    but the 6pt seems to have no effect. If you have any hints on what i'm doing wrong, im all ears:) always interested in learning new stuff:)

  • I see you've gone for font size="2" within the pre tag... If you want to use the font tag you could also try this in your CSS
    pre.debug, pre.debug font
    {
    font-family: Courier;
    font-size: 6pt;
    background-color: #99CCFF;
    text-align:left;
    }
    or was the font tag and size just a workaround anyway?
    All that selector does is say you want to apply the style to elements matching either pre.debug or pre.debug font.
  • Hi Jonno,

    The only difference i see between your tag and mine above is the additional ,pre.debug font

    It doesnt seem like it makes a difference when i change it:(

    Thanks
    Tess
  • Hi Tess

    One thing thats been bothering me for while is the difference between the output of !dumpheap on your posts to the one from the dumps I've got.  Here's an example.

    0:017> !dumpheap -type HttpRuntime
    ------------------------------
    Heap 0
      Address         MT     Size  Gen
    0x021d6160 0x01b89df4      116    2 System.Web.HttpRuntime
    0x021dd8c0 0x021400c8       12    2 System.Web.Configuration.HttpRuntimeConfigurationHandler
    0x021dd8cc 0x02140010       52    2 System.Web.Configuration.HttpRuntimeConfig
    total 3 objects
    ------------------------------
    Heap 1
      Address         MT     Size  Gen
    total 0 objects
    ------------------------------
    total 3 objects

    Why do I have 2 heaps and you've only got 1? I'm aware of the large object heap, but Heap 1 isn't a large object heap because in another dump I've got there's objects of only 20K on it and I thought an object had to >1MB to make onto the large object heap.

    BTW - I've found all your blogs to be of immense value while I'm trying to get my head round how to debug ASP.NET apps. Thanks.
  • Hi Howard,

    In this particular post, the difference between our outputs with !dumpheap -type is that i am running this on a single processor machine while you are on a multiproc machine.

    On single processor machines we run with the workstation version of the GC and only use one heap.  On a multi-proc machine you would run the server version and have one heap per processor or if you run a hyperthreaded multiproc you would have 2 heaps per processor.

    !dumpheap -type shows specifics for each heap. !dumpheap (without -type) doesnt specify per heap.  

    If you notice in some of my other posts where i am running !eeheap -gc (showing multiple heaps), i am running those samples on a multiproc machine.

    !eeversion will show you which runtime version you are running and how many heaps it is using.

    Hope this clarifies things...

    Tess
  • Hokay,
    How about this for a suggestion then...
    You've specified Courier as the font-family. Unfortunately my system doesn't have Courier on it so my browser substitutes Courier New every time that Courier should be applied.
    And then it doesn’t apply your specified font sizing unless the calculated font size is greater than 1em. I don’t know why.
    Anyway, try this as a fix. Replace this line in pre.debug
    font-family: Courier;
    with
    font-family: 'Courier New', Courier, Fixed, monospace;

    Then you should see the font-size rule make a difference.

    And given that you have ditched the font tag then you can also ditch the ', pre.debug font' selector too.

    cheers,
    Jonno

    PS Thanks for the posts - mostly over my head but still awesome!
  • Thank you very much Jonno, really appreciate the help...

    That seemed to do the trick:) I've now been able to change it to 8pt which is about as low as i can go and still have it readable.

    You learn something new every day...  
  • I'm a little confused by the demo in your post. You show us with the .foreach command that there is a cache object for each runtime but then when you demo the !dumpaspnetcache -stat it only list one group of objects. Are those all objects in all cache objects or only the objects in the first cache?
  • Hi Steve,

    In this particular sample there was only one application so only one cache, if there would have been more than one cache all objects from all caches would have been listed with the !dumpaspnetcache -stat command
  • This is part four in the ASP.Net tips series . In particular, this is a follow-up to, " ASP.Net Quick

  • In the .Net from 2.0 there is no !dumpaspnetcache command. Do you know if its called something different now, or if there is a way to do something similar?

  • Hi Jason,

    There isn't a !dumpaspnetcache command yet in 2.0 so the only way you can do it is through manually dumping out hte contents of System.Web.Caching.Cache  but a few common tasks so you dont have to run though the whole ordeal of dumping it out, would be to do the following

    1. Find the System.Web.Caching.Cache MT in !dumpheap -stat

    2. !dumpheap -mt on that MT

    3. !objsize on the objects so that you can see how much you are saving in cache

    For session state, find the MT for the InProcSessionState objects and run

    .foreach (Session {!dumpheap -mt <MT> -short}){!objsize ${Session}}

    Which will give you the size of the individual Sessions and then from there you can drill down into the particular session object that is causing you grief.

    Hope this helps,

    Tess

  • This is a bit of a continuation of ASP.NET Memory Issue: High memory usage in a 64bit w3wp.exe process

Page 1 of 2 (19 items) 12
Leave a Comment
  • Please add 7 and 4 and type the answer here:
  • Post