If broken it is, fix it you should

Using the powers of the debugger to solve the problems of the world - and a bag of chips    by Tess Ferrandez, ASP.NET Escalation Engineer (Microsoft)

ASP.NET Memory Leak Case Study: Sessions Sessions Sessions…

ASP.NET Memory Leak Case Study: Sessions Sessions Sessions…

Rate This
  • Comments 28

In ASP.NET 1.1 as you probably know, there are 3 different locations to store session objects.

In proc which stores session variables in the cache, State server which stores it in the state service and finally SQL Server.

There are of course pros and cons of each but no matter which one you use, you should be careful with how much you store in session state if you want your application to be scalable.

For in-proc session state, storing too much in session means high memory usage. For state server and SQL server you have the performance cost of serializing and de-serializing the session data.

One thing that may not be readily apparent is that if you use either of the out of process storage locations you will be reading in and de-serializing all the session data for a particular user on each web request.

Lot’s of data in out of proc session state may not only cause your performance to suffer, but depending on what you store in session scope you may be suffering a memory hit there too. Take for example a dataset, this will normally be serialized with the binary formatter (unless you don’t create your own dataset surrogate) into an XML format. The problem here is that when using a generic tool to do a specific task a lot of generalizations have to be made.

Say for example that you have a dataset like so:

ID CustomerFirstName CustomerLastName
1 John Doe
2 Jane Doe

Instead of just storing the data “1, Jane, Doe, 2, John, Doe” you store xml markup for the dataset, tables, records etc. and each individual data item looks something like <CustomerFirstName>Jane</CustomerFirstName>.

Add to this, that the binary formatter calculates approximately how many bytes it think its going to need to store this and then adds a little (sometimes a lot) to make sure that it wont have to reallocate the buffer.

Long story short… I have seen situations where the buffer used for serialization was around 10 to 15 times as big as the original dataset because of the length of column names vs. actual data etc.

See this article http://support.microsoft.com/default.aspx?scid=kb;en-us;829740 for a more in-depth discussion on the issue.

So if we take a pretty common scenario… You develop an on-line shop application for a customer with relatively few concurrent users. The initial setup is one standalone web server, using in-proc session state, and your application is storing some datasets per user with info about their past orders etc.

Suddenly the application gets more popular and a larger company wants to use it. It needs to work on a multi-server web farm, so you move to SQL Server session state.

What is wrong with this picture???

The original idea of storing the datasets in session scope was probably based on that it would be faster to get the datasets from cache rather than by doing database requests. Now you are stuck in a situation where not only do you still have to retrieve the datasets from a database, but you do it for every single request whether you are going to use it or not on that page… and every time it has to be de-serialized and/or serialized.

Makes you think twice about what you are storing in session state doesn’t it:)

Let’s take a look at a case study…

Problem description

The memory usage for the ASP.NET process is very high (800MB – 1GB), sometimes resulting in out of memory exceptions or unwanted recycles of the process.

Gathering data

As in the case of the event handlers post I would suggest multiple dumps as the memory increases, but even if that is not possible, a dump when the memory is very high will give us a good clue about what is going on.


The dump size is 1.473.913 bytes so we are using close to 1.4 GB private bytes which is pretty excessive.

Usually one of the first things I do is to run !eeheap –gc to get a feel for if most of the memory is on the managed heap, in which case I can for the most part rule out native leaks or problems with the loader heap.

0:023> !eeheap -gc
Number of GC Heaps: 2
Heap 0 (0x000b7198)
generation 0 starts at 0x022104d4
generation 1 starts at 0x022037c0
generation 2 starts at 0x02170030
ephemeral segment allocation context: none
 segment    begin       allocated     size
0x2170000 0x2170030  0x224a4e0 0xda4b0(894,128)
Large object heap starts at 0x0a170030
 segment    begin       allocated     size
0x0a170000 0x0a170030  0x0acf0b20 0x00b80af0(12,061,424)
0x0d490000 0x0d490030  0x0e3d2450 0x00f42420(16,000,032)
0x12010000 0x12010030  0x12f52460 0x00f42430(16,000,048)
0x13010000 0x13010030  0x13f52460 0x00f42430(16,000,048)
0x15010000 0x15010030  0x15f52460 0x00f42430(16,000,048)
0x1a010000 0x1a010030  0x1af52460 0x00f42430(16,000,048)
0x71ca0000 0x71ca0030  0x72be2470 0x00f42440(16,000,064)
0x748b0000 0x748b0030  0x757f2470 0x00f42440(16,000,064)
0x7d0e0000 0x7d0e0030  0x7d881250 0x007a1220(8,000,032)
Heap Size  0x2d5b4e10(760,958,480)
Heap 1 (0x000ede88)
generation 0 starts at 0x06249b58
generation 1 starts at 0x0623e190
generation 2 starts at 0x06170030
ephemeral segment allocation context: none
 segment    begin       allocated     size
0x6170000 0x6170030  0x6283b64 0x113b34(1,129,268)
Large object heap starts at 0x0b170030
 segment    begin       allocated     size
0x0b170000 0x0b170030  0x0b9f1c90 0x00881c60(8,920,160)
0x0c3e0000 0x0c3e0030  0x0d322460 0x00f42430(16,000,048)
0x0e490000 0x0e490030  0x0f3d2460 0x00f42430(16,000,048)
0x11010000 0x11010030  0x11f52460 0x00f42430(16,000,048)
0x14010000 0x14010030  0x14f52450 0x00f42420(16,000,032)
0x16010000 0x16010030  0x16f52480 0x00f42450(16,000,080)
0x17010000 0x17010030  0x17b81b60 0x00b71b30(12,000,048)
0x18010000 0x18010030  0x18b81b60 0x00b71b30(12,000,048)
0x19010000 0x19010030  0x19f52460 0x00f42430(16,000,048)
0x1b010000 0x1b010030  0x1bf52460 0x00f42430(16,000,048)
0x61010000 0x61010030  0x61f52470 0x00f42440(16,000,064)
0x62db0000 0x62db0030  0x63cf2470 0x00f42440(16,000,064)
0x657e0000 0x657e0030  0x66722470 0x00f42440(16,000,064)
0x685c0000 0x685c0030  0x69502470 0x00f42440(16,000,064)
0x6e110000 0x6e110030  0x6ec81b70 0x00b71b40(12,000,064)
0x72ca0000 0x72ca0030  0x73be2470 0x00f42440(16,000,064)
0x77ff0000 0x77ff0030  0x78f32470 0x00f42440(16,000,064)
0x7e0e0000 0x7e0e0030  0x7f022470 0x00f42440(16,000,064)
Heap Size  0x286a4124(678,052,132)
GC Heap Size  0x55c58f34(1,439,010,612)

So !eeheap –gc tells us that a) the GC Heap Size is around 1.4 GB which is very close to the total amount of memory, meaning that most of our memory is on the managed heap, and b) that most of our memory is in the large object segments, i.e. objects over 85000 bytes.

I have shortened the output here to save space. In the original output we had around a hundred large object segments. In fact this example is a bit exaggerated, it would be very rare to see this situation (hardly anything in the small-object segments, and tonnes in the large object segments), but the case study should still give you an idea of how to troubleshoot these types of issues in the more general case.

Naturally from here the next step is to find out what is on the large object heaps.

We can start off with a summary of the objects stored on the large object heap, to get a feel for what objects we are looking for.

Since the objects stored on the large object heap are all 85000 bytes or above we can dump out the large object heaps by using the –min 85000 switch for !dumpheap, and then –stat to show just the statistics.

0:023> !dumpheap -min 85000 -stat
Using our cache to search the heap.
        MT      Count     TotalSize Class Name
0x000eda20          1       920,144      Free
0x01b2209c         33   132,000,528 System.Object[]
0x01b226b0        163 1,304,001,956 System.Int32[]
Total 197 objects, Total size: 1,436,922,628

Most of the memory is in Int32 arrays but a good chunk (around 132 MB) is used for byte arrays.

We should note here that this is just for the structures (i.e. the arrays themselves). The 132 MB does not include the size of the objects stored in the Object arrays, same for the Int32 arrays. To find out the actual size of each object array or int32 array including the objects stored in them we would have to run !objsize.

The next step is to take a closer look at some of the individual arrays more closely by dumping all objects on the large object heap (without the –stat switch).

0:023> !dumpheap -min 85000
Using our cache to search the heap.
   Address         MT     Size  Gen
0x0b170030 0x000eda20  920,144   -1      Free
0x0a920210 0x01b2209c 4,000,016   -1 System.Object[] 
0x247b1250 0x01b2209c 4,000,016   -1 System.Object[] 
0x277b1250 0x01b2209c 4,000,016   -1 System.Object[] 
0x287b1250 0x01b2209c 4,000,016   -1 System.Object[] 
0x51af0030 0x01b226b0 8,000,012   -1 System.Int32[] 
0x52291250 0x01b226b0 8,000,012   -1 System.Int32[] 
0x53b40950 0x01b226b0 8,000,012   -1 System.Int32[] 
0x56620030 0x01b226b0 8,000,012   -1 System.Int32[] 
0x57620030 0x01b226b0 8,000,012   -1 System.Int32[] 
0x5a440030 0x01b226b0 8,000,012   -1 System.Int32[] 
0x5abe1250 0x01b226b0 8,000,012   -1 System.Int32[] 
0x5c660030 0x01b226b0 8,000,012   -1 System.Int32[] 
0x5ce01250 0x01b226b0 8,000,012   -1 System.Int32[] 
0x61010030 0x01b226b0 8,000,012   -1 System.Int32[] 
0x617b1250 0x01b226b0 8,000,012   -1 System.Int32[] 
0x62db0030 0x01b226b0 8,000,012   -1 System.Int32[] 
0x63551250 0x01b226b0 8,000,012   -1 System.Int32[] 
0x657e0030 0x01b226b0 8,000,012   -1 System.Int32[] 
0x65f81250 0x01b226b0 8,000,012   -1 System.Int32[]

Picking one of the Int32 arrays at random, we can do !gcroot on it to find out where it is rooted to find out why it won’t get garbage collected.

0:023> !gcroot 0x72ca0030 
Scan Thread 16 (0xbd8)
Scan Thread 20 (0x89c)
Scan Thread 22 (0xa5c)
Scan HandleTable 0xdc9d8
Scan HandleTable 0xea6e8
Scan HandleTable 0x1531e8

The root chain tells us that this particular Int32 array is rooted in cache, more specifically in the InProcSessionState object, indicating that it is stored in in-proc session scope.

Once we get here a search through the code for session assignment, focusing on Int32 arrays may be in order. However if the code base is large or there is another reason we can’t just search the code, we can dig further to get a bit more information.

Debugging Tip: From the HttpContext object above you can information about the application that the particular session object belongs to (path etc) if you have many different applications.

Taking a look at the cache (where the sessions are stored) using !dumpaspnetcache –stat we get this

0:023> !dumpaspnetcache -stat
Going to dump the ASP.NET Cache.
        MT      Count     TotalSize Class Name
0x0211cc9c          1            20 System.Web.Security.FileSecurityDescriptorWrapper
0x020c242c          2            56 System.Web.UI.ParserCacheItem
0x0206c66c          5           260 System.Web.Configuration.HttpConfigurationRecord
0x0c2e7014          1           316 System.Web.Mobile.MobileCapabilities
0x79b94638          4           376 System.String
0x0c2eaeb4        151         7,248 System.Web.SessionState.InProcSessionState
Total 164 objects, Total size: 8,276

So we have 151 concurrent sessions (each InProcSessionState object holds the objects for one session)

To find out how much is stored in them we can use a for each loop using the method table for InProcSessionState from above and run !objsize on each InProcSessionState object.

0:023> .foreach (obj {!dumpheap -mt 0x0c2eaeb4 -short}){!objsize ${obj}}
sizeof(0x22028fc) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x2202a10) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x2202cfc) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x2202fe8) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x22032d4) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x22035c0) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x2203a38) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x2203d24) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x2204010) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x6248080) = 12,000,772 (0xb71e04) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x6248334) = 12,000,772 (0xb71e04) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x62485e8) = 12,000,772 (0xb71e04) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x624a84c) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x624d7b8) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x6250724) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x6253690) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x62565fc) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x6259568) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x625c4d4) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x625f440) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x62623ac) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x6265318) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x6268284) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x626b1b8) = 12,000,772 (0xb71e04) bytes (System.Web.SessionState.InProcSessionState)

Wow, 151 sessions and each holds about 8-12 MB of data, yeah that could certainly cause a memory issue:)

Let’s pick one of them and see what we actually store in session …

0:023> !do 0x626b1b8
Name: System.Web.SessionState.InProcSessionState
MethodTable 0x0c2eaeb4
EEClass 0x0c1c5660
Size 48(0x30) bytes
GC Generation: 0
mdToken: 0x02000132  (c:\windows\assembly\gac\system.web\1.0.5000.0__b03f5f7f11d50a3a\system.web.dll)
FieldDesc*: 0x0c2eae0c
        MT      Field     Offset                 Type       Attr      Value Name
0x0c2eaeb4 0x40009f2      0x4                CLASS   instance 0x06269f74 dict
0x0c2eaeb4 0x40009f3      0x8                CLASS   instance 0x00000000 staticObjects
0x0c2eaeb4 0x40009f4      0xc         System.Int32   instance 20 timeout
0x0c2eaeb4 0x40009f5     0x18       System.Boolean   instance 0 isCookieless
0x0c2eaeb4 0x40009f6     0x10         System.Int32   instance 0 streamLength
0x0c2eaeb4 0x40009f7     0x19       System.Boolean   instance 0 locked
0x0c2eaeb4 0x40009f8     0x1c            VALUETYPE   instance start at 0x0626b1d4 utcLockDate
0x0c2eaeb4 0x40009f9     0x14         System.Int32   instance 1 lockCookie
0x0c2eaeb4 0x40009fa     0x24            VALUETYPE   instance start at 0x0626b1dc spinLock

Each InProcSessionState object has a dict (dictionary) member variable that holds the actual session objects in its _entriesArray.

0:023> !do 0x06269f74 
Name: System.Web.SessionState.SessionDictionary
MethodTable 0x0c2e0c54
EEClass 0x0c1c1308
Size 44(0x2c) bytes
GC Generation: 0
mdToken: 0x0200013b  (c:\windows\assembly\gac\system.web\1.0.5000.0__b03f5f7f11d50a3a\system.web.dll)
FieldDesc*: 0x0c2e0b30
        MT      Field     Offset                 Type       Attr      Value Name
0x0206b338 0x4000a8b     0x24       System.Boolean   instance 0 _readOnly
0x0206b338 0x4000a8c      0x4                CLASS   instance 0x06269fb8 _entriesArray
0x0206b338 0x4000a8d      0x8                CLASS   instance 0x06269fa0 _hashProvider
0x0206b338 0x4000a8e      0xc                CLASS   instance 0x06269fac _comparer
0x0206b338 0x4000a8f     0x10                CLASS   instance 0x0626a020 _entriesTable
0x0206b338 0x4000a90     0x14                CLASS   instance 0x00000000 _nullKeyEntry
0x0206b338 0x4000a91     0x18                CLASS   instance 0x00000000 _keys
0x0206b338 0x4000a92     0x1c                CLASS   instance 0x00000000 _serializationInfo
0x0206b338 0x4000a93     0x20         System.Int32   instance 4 _version
0x0c2e0c54 0x4000a0f     0x25       System.Boolean   instance 1 _dirty
0x0c2e0c54 0x4000a0e        0                CLASS     shared   static s_immutableTypes
    >> Domain:Value 0x000dad08:NotInit  0x00104f30:0x021be0dc <<

0:023> !do 0x06269fb8 
Name: System.Collections.ArrayList
MethodTable 0x79ba2ee4
EEClass 0x79ba3020
Size 24(0x18) bytes
GC Generation: 0
mdToken: 0x02000100  (c:\windows\microsoft.net\framework\v1.1.4322\mscorlib.dll)
FieldDesc*: 0x79ba3084
        MT      Field     Offset                 Type       Attr      Value Name
0x79ba2ee4 0x4000362      0x4                CLASS   instance 0x06269fd0 _items
0x79ba2ee4 0x4000363      0xc         System.Int32   instance 3 _size
0x79ba2ee4 0x4000364     0x10         System.Int32   instance 3 _version
0x79ba2ee4 0x4000365      0x8                CLASS   instance 0x00000000 _syncRoot 

The entries array is an ArrayList, and to print out all the objects in an arraylist in one shot we can run !do –v on it’s _items member variable

0:023> !do -v 0x06269fd0 
Name: System.Object[]
MethodTable 0x01b2209c
EEClass 0x01b22018
Size 80(0x50) bytes
GC Generation: 0
Array: Rank 1, Type CLASS
Element Type: System.Object
Content: 16 items
------ Will only dump out valid managed objects ----
   Address	        MT	Class Name
0x0626a81c	0x0206b784	System.Collections.Specialized.NameObjectCollectionBase/NameObjectEntry
0x0626a82c	0x0206b784	System.Collections.Specialized.NameObjectCollectionBase/NameObjectEntry
0x0626a83c	0x0206b784	System.Collections.Specialized.NameObjectCollectionBase/NameObjectEntry


We can either dump out these objects individually or we can let the debugger do the work for us.

If we use !do –v –short instead of just !do –v, we just get the addresses of the NameObject entries so we can use .foreach on them.

0:023> !do -v 0x06269fd0 -short

In this specific case the .foreach might seem like overkill but it’s good to know how to do it for larger collections.

For each object, we print out the session variable name at offset 0x4, the object at offset 0x8 and the size of the object.

(To learn more about how this works, read the post on “how much are you caching”)

0:023> .foreach (obj {!do -v 0x06269fd0 -short}){.echo ***;!do poi(${obj}+0x4);!do poi(${obj}+0x8);!objsize ${obj}}
String: somestring

String: this is a string i stored in session scope

sizeof(0x626a81c) =      160 (    0xa0) bytes (System.Collections.Specialized.NameObjectCollectionBase/NameObjectEntry)
String: alargeintarray

Name: System.Int32[]
MethodTable 0x01b226b0
EEClass 0x01b22638
Size 8000012(0x7a120c) bytes
GC Generation: 3
Array: Rank 1, Type System.Int32
Element Type: System.Int32
Content: 2,000,000 items
sizeof(0x626a82c) = 8,000,076 (0x7a124c) bytes (System.Collections.Specialized.NameObjectCollectionBase/NameObjectEntry)
String: sometimesbig

Name: System.Object[]
MethodTable 0x01b2209c
EEClass 0x01b22018
Size 4000016(0x3d0910) bytes
GC Generation: 3
Array: Rank 1, Type CLASS
Element Type: System.Object
Content: 1,000,000 items
sizeof(0x626a83c) = 4,000,076 (0x3d094c) bytes (System.Collections.Specialized.NameObjectCollectionBase/NameObjectEntry)

So for this user we are storing a pretty small string “this is a string i stored in session scope“ in Session[“somestring”], an 8 MB Int32 array in Session[“alargeintarray”] and a 4 MB object array in Session[“sometimesbig”].

At this point we have the issue pretty well defined and the task will be how to reduce the sizes of these objects, but that I will leave to the developers:) My work here is done:)

Based on the number of issues I have seen with high memory caused by storing too much in cache or sessions, I have made it a standard step in my troubleshooting of high mem issues to look at the size and contents of the cache and session state.

Side note

If the number of users on your site is pretty constant but the number of concurrent sessions (in performance monitor) keeps increasing abnormally, you may be running in to a problem with timers not firing http://support.microsoft.com/kb/900822/en-us. You can obtain this hotfix by contacting Microsoft Product Support.

Over and out…  


  • Hi,
    What about fragmented heap? How do you handle them? Especially in .NET framework 1.1?
  • Hi Naveen,

    If you are talking about managed heap fragmentation, there are two kinds, 1) heap fragmentation caused by pinned objects, in which case you can run !objsize (without parameters) to see what your pinned objects are and where they are located. And then try to reduce the amount of time they are pinned (i.e. shorten the time for remoting requests etc.) If you are pinning objects yourself, you can opt to create a buffer of pinned objects so all pinned objects are created in one specific place in memory, thus reducing greatly the fragmentation in other segments.
    2) heap fragmentation in the LOH. Here i would recommend examining the objects on the LOH and see if any can be reduced in size to not end up on the LOH. The problem gets worse if the objects stored on the LOH keep increasing in size since we can then not reuse a previous free object (as it doesnt fit), and the LOH ends up looking like a swiss cheeze.

    I'll try to expand a bit on this in a later post...

    If you are talking about native heap fragmentation, please see my answer to marks comment on the OOM post http://blogs.msdn.com/tess/archive/2005/11/25/496898.aspx.

    Hope this answers the question a little bit, I'll definitely expand on this later on though...
  • I am talking about heap fragmentation in LOH.
  • Very interesting...
  • Weird, why my !dumpheap command doesn't show the address field? it only has MT, Count and so on. I need the Address field so I can do gcroot and do command.

    Any ideas?
  • That sounds very weird, could you post the exact command you use and part of the output?

    Are you using the clr10\sos.dll that comes with windbg?
  • in which console we can know about memory in GC and identify any memory leakeages for the objects which i am using in my application

    ex: in the above article i saw !eeheap -gc command, in which console
  • Hi Vijenra,

    This is in the windbg.exe console either when live debugging an application or when looking at a dump from your application.
  • Great write up Tess.  I will probably spend all of tomorrow analyzing our high memory consumption using your wisdom :)

  • Great article...I need this.  One question, arent session objects supposed to be released after 20 minutes of inactivity instead of just building up?  
  • Yes they are.  That is not so much the problem... Its the amount of data and the amount of concurrent sessions...
  • That was my thought, but the memory builds and during long periods of site inactivity, it sits there...so I must be doing something else.  Thanks again for the great article.
  • A small comment on that:  Since garbage collection only happens on allocation (with a few exceptions such as reaching the memory limit in asp.net or manually calling GC.Collect), it might still be that objects that were previously rooted in Session scope might still lay around during inactivity.

    Also, if the total number of sessions keep creeping up you might need the fix in the article mentioned in the end of the post.
  • Great article, any plans to update this for asp.net v2? In particular the windbg commands you would need to run?

Page 1 of 2 (28 items) 12
Leave a Comment
  • Please add 5 and 2 and type the answer here:
  • Post