Speaking of which... : SessionState

Aspnet_state.exe crashing on Xp

JohanS — Fri, 21 Dec 2007 18:35:07 GMT

I ran across an interesting situation with a customer today.

He had a workstation running Windows Xp Professional. He'd previously been running Framework 1.1 and had been using the ASP.NET State Service. He had now upgraded to framework 2.0. and upon shutdown he now received the following error:

aspnet_state.exe. Application Error. The instruction at "0x6a2a2fec" referenced memory at "0x0000000c". The memory could not be "read".

To make a long story short it turned out this was due to an access violation upon shutdown. We did the following and resolved the situation:

Start Regedit.
Navigate to the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ASP.NET_2.0.50727 key.
Right-click "Names" and select Permissions on the context menu.
Click Add and enter NETWORK SERVICE, click "Check Names" and click Ok.
With NETWORK SERVICE highlighted in the "Groups or User Names" list click "Advanced".
On the "Advanced Security Settings for Names" dialogue highlight NETWORK SERVICE and click "Edit".
In the "Permission entry for Names" dialogue check the "Name" box is showing "NETWORK SERVICE" and put check marks against "Query Value", "Set Value", "Create Subkey", "Enumerate Subkeys", "Notify" and "Read Control".
Click Ok on all the dialogues and close Regedit.

As I assisted him it turned out my customer had searched the web for a solution and had found quite a few people experiencing the same problem. I guess most of you are working with Windows 2003 / Vista, so this information might be a bit obsolete. But I thought I should share the knowledge anyway.

Happy Holidays! / Johan

Getting started with windbg - part II

JohanS — Mon, 26 Nov 2007 16:49:50 GMT

This is a continuation of my previous post with the imaginative name Getting started with windbg - part I. I'll be assuming that you've read it, so if you haven't I suggest you check it out first. We're still working with the same sample dump, so I'll pretty much pick up right where we left off.

More useful commands

Last time we used some nice commands from the sos-extension to look at the running callstacks, the requests, the CPU load of the threads, etc. We also dug deeper into the callstack to see what distinguished name we used for a SearchRequest. We'll keep using these commands, but also learn a few new ones.

!dumpstackobjects (!dso)

Now, imagine that we're looking at a specific thread and would like to see all managed objects referenced to by the current stack. Is there a way for us to do this? - There most certainly is. It's called dumpstackobjects, or dso for short. If we run it on the current thread we'll see a listing of objects that are referenced by the callstack. The whole output would look something like this (trimmed down to size):

0:050> !dso
OS Thread Id: 0x1e8c (50)
ESP/REG Object   Name
17a9e534 0741f860 System.RuntimeType
17a9e6b8 271fdfe0 System.DirectoryServices.Protocols.LdapConnection
17a9e6bc 27246f20 System.DirectoryServices.Protocols.SEC_WINNT_AUTH_IDENTITY_EX
17a9e740 271fdfe0 System.DirectoryServices.Protocols.LdapConnection
17a9e744 27246f20 System.DirectoryServices.Protocols.SEC_WINNT_AUTH_IDENTITY_EX
17a9e764 27246f20 System.DirectoryServices.Protocols.SEC_WINNT_AUTH_IDENTITY_EX
17a9e768 271fdfe0 System.DirectoryServices.Protocols.LdapConnection
17a9e780 271fdfe0 System.DirectoryServices.Protocols.LdapConnection
17a9e784 27246e38 System.DirectoryServices.Protocols.SearchRequest
17a9e794 271fdf14 System.DirectoryServices.Protocols.LdapDirectoryIdentifier
17a9e7a8 27246ef8 System.Collections.ArrayList
17a9e7bc 27246ef8 System.Collections.ArrayList
17a9e7c8 271fdfe0 System.DirectoryServices.Protocols.LdapConnection
17a9e8a4 27246e38 System.DirectoryServices.Protocols.SearchRequest
17a9e8d0 27246ed8 System.Object[]    (System.Object[])
17a9e8e0 073ff6b8 System.String    cn
17a9e8e4 271fdfe0 System.DirectoryServices.Protocols.LdapConnection
17a9e8f4 27246d00 System.String    CN=Dummy,CN=Accounts,CN=useradm,DC=Dummy,DC=net
17a9e8f8 271fdfe0 System.DirectoryServices.Protocols.LdapConnection
17a9e8fc 27246e38 System.DirectoryServices.Protocols.SearchRequest
17a9e910 03380310 System.String
17a9e914 27246e24 System.Object[]    (System.String[])
17a9e918 272399a8 System.String    CN=OID-Dummy-ABC123,CN=Dummy,CN=Accounts,CN=useradm,DC=Dummy,DC=net
17a9e91c 27246ddc System.String    (CN=OID-Dummy-ABC123)
...etc...

This is extremely useful when we want to see all the objects referenced by this thread alone. If you want to analyze one of the objects, simply copy the the address from the Object-column and use !dumpobject:

0:050> !do 271fdfe0
Name: System.DirectoryServices.Protocols.LdapConnection
MethodTable: 14a2040c
EEClass: 149daf08
Size: 56(0x38) bytes
(C:\WINDOWS\assembly\GAC_MSIL\System.DirectoryServices.Protocols\2.0.0.0__b03f5f7f11d50a3a\System.DirectoryServices.Protocols.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
14a2078c 40000c3        4 ...NetworkCredential 0 instance 00000000 directoryCredential
14a2144c 40000c4        8 ...ificateCollection 0 instance 271fe018 certificatesCollection
1202af88 40000c5       10      System.TimeSpan 1 instance 271fdff0 connectionTimeOut
1466fe50 40000c6        c ...rectoryIdentifier 0 instance 271fdf14 directoryIdentifier
14a2034c 4000236       24         System.Int32 0 instance        2 connectionAuthType
14a223a4 4000237       18 ...dapSessionOptions 0 instance 271fe2d8 options
0fb896d8 4000238       28        System.IntPtr 0 instance 564180944 ldapHandle
120261c8 4000239       2c       System.Boolean 0 instance        0 disposed
120261c8 400023a       2d       System.Boolean 0 instance        0 bounded
120261c8 400023b       2e       System.Boolean 0 instance        0 needRebind
14a22084 400023e       1c ...pResponseCallback 0 instance 271fe03c fd
120261c8 4000243       2f       System.Boolean 0 instance        0 setFQDNDone
120261c8 4000244       30       System.Boolean 0 instance        1 automaticBind
120261c8 4000245       31       System.Boolean 0 instance        1 needDispose
120261c8 4000246       32       System.Boolean 0 instance        1 connected
14a2267c 4000247       20 ...s.QUERYCLIENTCERT 0 instance 271fe394 clientCertificateRoutine
0fd314bc 400023c       20 ...ections.Hashtable 0   shared   static handleTable
    >> Domain:Value 0019daf0:NotInit 11b42540:073fe504 <<
02c36ca0 400023d       24        System.Object 0   shared   static objectLock
    >> Domain:Value 0019daf0:NotInit 11b42540:073fe53c <<
0fd314bc 400023f       28 ...ections.Hashtable 0   shared   static asyncResultTable
    >> Domain:Value 0019daf0:NotInit 11b42540:073fe610 <<
14a21864 4000240       2c ...lResultsProcessor 0   shared   static partialResultsProcessor
    >> Domain:Value 0019daf0:NotInit 11b42540:073fe678 <<
12305e94 4000241       30 ....ManualResetEvent 0   shared   static waitHandle
    >> Domain:Value 0019daf0:NotInit 11b42540:073fe64c <<
14a21954 4000242       34 ...lResultsRetriever 0   shared   static retriever
    >> Domain:Value 0019daf0:NotInit 11b42540:073fe6a8 <<

!dumparray (!da)

As you might have noticed we have a couple of object arrays on the stack. Look for the System.Object[]-type in the listing above and you'll find them. If you execute !dumpobject on an array you'll only get information about the array itself, not it's contents. To get information about what's in the array we need to use the !dumparray-command, or !da for short.

0:050> !do 27239b98
Name: System.Object[]
MethodTable: 02c3896c
EEClass: 02c388ec
Size: 24(0x18) bytes
Array: Rank 1, Number of elements 2, Type CLASS
Element Type: System.String
Fields:
None

0:050> !da 27239b98
Name: System.String[]
MethodTable: 02c3896c
EEClass: 02c388ec
Size: 24(0x18) bytes
Array: Rank 1, Number of elements 2, Type CLASS
Element Methodtable: 02c39310
[0] 272399a8
[1] 27239a44

As you can see the !dumparray-command gives us a bit more information about the object. We can see that it contains System.String data and we get the addresses of the two items in the array. Since they're System.String-objects we can simply use !dumpobject to view their contents.

!objsize

If you look at the listing above you'll see that the size of the object is listed as 24 bytes. To paraphrase Obi Wan Kenobi: "This is true, from a certain point of view." The 24 bytes are the size of the System.Object[]-object itself. Not it's contents. As you can see when we execute !dumparray, the array only contains two references to two strings. These strings are individual objects and could be 32 MB for all we know, so the 24 bytes are not the total size of the array, but it would still be correct to claim that the System.Object[] is only 24 bytes large.

To get the total size of the object we use the !objsize command:

0:050> !objsize 27239b98
sizeof(27239b98) = 348 ( 0x15c) bytes (System.Object[])

!objsize will iterate through all the child objects referenced by the object, as well as all the grandchildren and so on. Apparently the total size of the array and it's child objects is 348 bytes.

If there are a lot of child objects it may take some time for !objsize to calculate the total size of the object. You should also be aware that !objsize may not always be as smart as you'd like. If, for example, you have a custom button control that references it's parent aspx-page you'd get the total size of the aspx-page and all it's child controls. In other words: If !objsize claims that the object in question is ridiculously large you might want to manually check what the object references using !dumpobject.

!dumpheap

This is another command that I use frequently. It is, however a command that you will want to use with at least one argument. Dumpheap with no arguments will dump all objects on the heap, so I usually begin by using the -stat argument, which in itself will write a lot of info on the screen, but it will at least be summarized. Here' you'll find the trimmed down output of !dumpheap -stat:

0:050> !dumpheap -stat
------------------------------
Heap 0
total 2754508 objects
------------------------------
Heap 1
total 2761329 objects
------------------------------
total 5515837 objects
Statistics:
      MT    Count    TotalSize Class Name
16e0a6d8        1           12 System.Collections.Generic.ObjectEqualityComparer`1[[System.Data.ProviderBase.DbConnectionInternal, System.Data]]
16d9cd9c        1           12 System.Xml.Serialization.Configuration.DateTimeSerializationSection+DateTimeSerializationMode
16d9bf30        1           12 System.Diagnostics.OrdinalCaseInsensitiveComparer
16d9112c        1           12 System.Xml.Serialization.NameTable
16d7f664        1           12 System.Xml.Serialization.TempAssemblyCache
163ea85c        1           12 System.Data.Res
1501e4c4        1           12 System.Collections.Generic.ObjectEqualityComparer`1[[System.Web.UI.Control, System.Web]]
14efb138        1           12 System.Net.TimeoutValidator
14ef9964        1           12 System.Net.Cache.HttpRequestCacheLevel
14ef77a8        1           12 Microsoft.Win32.WinInetCache
14ef68e4        1           12 System.Net.WebRequest+WebProxyWrapper
14ef658c        1           12 System.Net.Configuration.ProxyElement+BypassOnLocalValues
14ef63d8        1           12 System.Net.Configuration.ProxyElement+AutoDetectValues
14ef5b68        1           12 System.Net.CaseInsensitiveAscii
14ef5610        1           12 System.Net.HeaderInfoTable
14ef4718        1           12 System.Net.HttpRequestCreator
14db6710        1           12 System.Web.Configuration.MachineKeyValidationConverter
14db3140        1           12 System.Collections.Generic.ObjectEqualityComparer`1[[System.Runtime.Serialization.MemberHolder, mscorlib]]
14b3f4d8        1           12 System.Web.UI.SupportsEventValidationAttribute
...etc...
14a276a8    19578       704808 System.DirectoryServices.Interop.AdsValueHelper
14a2ea24     9196       735680 System.Web.UI.WebControls.Label
14a2e51c    16862       741928 System.Web.UI.WebControls.Style
125778ec    48015       768240 System.Collections.Specialized.NameObjectCollectionBase+NameObjectEntry
120261c8    65842       790104 System.Boolean
14a2ee7c     9198       809424 System.Web.UI.WebControls.Table
14b311c4     9647       810348 System.Web.UI.WebControls.Image
13a2b7dc    34913       837912 System.Web.HttpServerVarsCollectionEntry
14b303a4    10605       848400 System.Web.UI.WebControls.HyperLink
14d8e3d4    77748       932976 Microsoft.Web.UI.WebControls.BaseChildNodeCollection+ActionType
14db90ac    81372       976464 System.Web.UI.WebControls.FontInfo
14b30694    28648      1031328 System.Web.UI.WebControls.TableRow+CellControlCollection
14d8fdbc    38912      1089536 Microsoft.Web.UI.WebControls.TreeNodeCollection
14b3d0bc    86592      1385472 System.Web.UI.Pair
1466c5c4    39305      1414980 System.Web.UI.ControlCollection
14d8e48c    77748      1865952 Microsoft.Web.UI.WebControls.BaseChildNodeCollection+Action
1545061c    38874      2176944 Microsoft.Web.UI.WebControls.TreeNode
14b30eec    52668      2317392 System.Web.UI.WebControls.TableItemStyle
14a2f804    28515      2395260 System.Web.UI.WebControls.TableRow
14a2be6c    40894      2453640 System.Web.UI.LiteralControl
0fd3da00   228792      2745504 System.Int32
14b3e3ac   244793      2937516 System.Web.UI.IndexedString
14a2de94   198580      3177280 System.Web.UI.StateBag
1466c454    80512      3542528 System.Web.UI.Control+OccasionalFields
12302c2c   205849      4116980 System.Collections.Specialized.HybridDictionary
14b30024    52934      4446456 System.Web.UI.WebControls.TableCell
12302f2c   178294      4992232 System.Collections.Specialized.ListDictionary
14a2e284   412762      6604192 System.Web.UI.StateItem
14d8ce64   117078      7024680 Microsoft.Web.UI.WebControls.CssCollection
0fd314bc   132065      7395640 System.Collections.Hashtable
1230319c   422580      8451600 System.Collections.Specialized.ListDictionary+DictionaryNode
1202a58c   380438      9130512 System.Collections.ArrayList
0fd32050   133000     22582944 System.Collections.Hashtable+bucket[]
02c3896c   649842     23275900 System.Object[]
0fd3c12c     3471     36385536 System.Byte[]
001fee20      338     65409920      Free
02c39310   683083    161821000 System.String
Total 5515837 objects
Fragmented blocks larger than 0.5 MB:
    Addr     Size      Followed by
2adf31cc    2.0MB         2aff85d8 System.String
2b006a2c   20.3MB         2c4530d8 System.Net.SocketAddress

As you can see we now get a listing sorted by size of all object-types on the heap. You'll usually find strings down at the bottom since that's what is commonly used the most.

Other useful arguments are -type and -mt (which stands for MethodTable). Using them you're able to see all objects of a specific type. For example. If we want to look at all HttpRequestCreators on the heap (there are one) you'll simply copy it's MethodTable which you'll find in the listing above (14ef4718) and use !dumpheap -mt

0:050> !dumpheap -mt 14ef4718
------------------------------
Heap 0
Address       MT     Size
0342ccf8 14ef4718       12
total 1 objects
------------------------------
Heap 1
Address       MT     Size
total 0 objects
------------------------------
total 1 objects
Statistics:
      MT    Count    TotalSize Class Name
14ef4718        1           12 System.Net.HttpRequestCreator

This gives us the address of the object and if we'd like to inspect it closer we simply use !dumpobject on that address.

!dumpheap -type works pretty much the same way, except this time you filter the results by class name. !dumpheap -type performs a substring match, so if you write !dumpheap -type System.Web you'll get every object who's class name contains System.Web, which would be a lot.

Other useful arguments are -min and -max which accept a number representing the minimum/maximum number of bytes the object size should be. This can be really useful when troubleshooting string-abuse, etc. Also !dumpheap -stat -min 85000 would list all objects on the large object heap.

Putting the tools to use

I'd now like to use the commands we've covered in a bit more practical scenario. The dump we've been looking at is from a previous case of mine. The application in question was running on a Web garden with two workerprocesses. Session State was handled by a SQL Server. The customer was experiencing performance issues and the problem description was hazy at best. Anyway I had tons of dumps to work with, so I simply poked around to see what I could find. One thing I did pretty early on was to look at caching. According to the customer they weren't using the cache at all, but I usually find it best to double-check this type of thing.

Determining the size of the cache

To find out how much data was kept in the cache I first needed to find the System.Web.Caching.Cache class. So I ran !dumpheap -stat -type System.Web.Caching.Cache. Note that I also used the -stat argument. Otherwise I would have gotten a very long list of System.Web.Caching.CacheKeys and System.Web.Caching.CacheEntrys as well. Anyway, here's the result:

0:050> !dumpheap -type System.Web.Caching.Cache -stat
------------------------------
Heap 0
total 665 objects
------------------------------
Heap 1
total 1084 objects
------------------------------
total 1749 objects
Statistics:
      MT    Count    TotalSize Class Name
123056f8        1           12 System.Web.Caching.CacheKeyComparer
1230494c        1           12 System.Web.Caching.Cache
1230500c        1           24 System.Web.Caching.CacheMultiple
1230514c        1           32 System.Web.Caching.CacheMemoryStats
123053b4        1           36 System.Web.Caching.CacheMemoryTotalMemoryPressure
123059bc        2           40 System.Web.Caching.CacheUsage
12304bdc        1           48 System.Web.Caching.CacheCommon
123054f4        1           52 System.Web.Caching.CacheMemoryPrivateBytesPressure
12305874        2           64 System.Web.Caching.CacheExpires
12304e64        2          200 System.Web.Caching.CacheSingle
1255b594       85         1360 System.Web.Caching.CacheDependency+DepFileInfo
123046c4       40         1440 System.Web.Caching.CacheDependency
123042ec       47         1504 System.Web.Caching.CacheItemRemovedCallback
123063fc      832        16640 System.Web.Caching.CacheKey
12306820      732        52704 System.Web.Caching.CacheEntry
Total 1749 objects

Okay, so now I had the MethodTable for the System.Web.Caching.Cache object. Therefore I could now get the address to the object itself. I did this by asking !dumpheap to list all the objects on the heap with that MethodTable. I knew that there was just going to be one hit:

0:050> !dumpheap -mt 1230494c
------------------------------
Heap 0
Address       MT     Size
03392d20 1230494c       12
total 1 objects
------------------------------
Heap 1
Address       MT     Size
total 0 objects
------------------------------
total 1 objects
Statistics:
      MT    Count    TotalSize Class Name
1230494c        1           12 System.Web.Caching.Cache
Total 1 objects

So now I ran !objsize on this object to see how big it was. This took a little time to calculate, since the cache is quite complex and there are a lot of children to iterate through.

0:050> !objsize 03392d20
sizeof(03392d20) = 266640828 ( 0xfe49dbc) bytes (System.Web.Caching.Cache)

So the cache is 266 MB in size. That's quite a lot considering the fact that the customer claimed that they weren't using the cache at all!

What is being cached?

To sample what the customer was caching I then took a look at a few of the CacheEntrys. I already had the MethodTable for the System.Web.Caching.CacheEntry from when I ran !dumpheap -type System.Web.Caching.Cache -stat (above), so I could use that to retrieve all CacheEntrys.

0:050> !dumpheap -mt 12306820
------------------------------
Heap 0
Address       MT     Size
033950bc 12306820       72
033a20d8 12306820       72
033ac79c 12306820       72
033da21c 12306820       72
033f04c4 12306820       72
03428ec8 12306820       72
0344dab4 12306820       72
03815d00 12306820       72
038265d8 12306820       72
....etc...
03af7010 12306820       72
03b291bc 12306820       72
03b2c674 12306820       72
03b6dca0 12306820       72
03b797dc 12306820       72
03b85318 12306820       72
03ba9150 12306820       72
03c258cc 12306820       72
03de43c8 12306820       72
03e160f8 12306820       72
total 382 objects
------------------------------
total 732 objects

Another valid command that would have given me the exact same output would off course have been !dumpheap -type System.Web.Caching.CacheEntry.

Okay, so now I had a long list of CacheEntrys. To sample the contents I just picked an address and examined it by using !dumpobject

0:050> !do 03b2c674
Name: System.Web.Caching.CacheEntry
MethodTable: 12306820
EEClass: 122f6470
Size: 72(0x48) bytes
(C:\WINDOWS\assembly\GAC_32\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
02c39310 4001327        4        System.String 0 instance 03b2c600 _key
0fb8f1f8 4001328        c          System.Byte 0 instance        2 _bits
0fd3da00 4001329        8         System.Int32 0 instance -1314181915 _hashCode
02c36ca0 4001330       10        System.Object 0 instance 03b2c644 _value
120219d0 4001331       1c      System.DateTime 1 instance 03b2c690 _utcCreated
120219d0 4001332       24      System.DateTime 1 instance 03b2c698 _utcExpires
1202af88 4001333       2c      System.TimeSpan 1 instance 03b2c6a0 _slidingExpiration
0fb8f1f8 4001334        d          System.Byte 0 instance        7 _expiresBucket
123062d8 4001335       34 ...g.ExpiresEntryRef 1 instance 03b2c6a8 _expiresEntryRef
0fb8f1f8 4001336        e          System.Byte 0 instance 4294967295 _usageBucket
12306738 4001337       38 ...ing.UsageEntryRef 1 instance 03b2c6ac _usageEntryRef
120219d0 4001338       3c      System.DateTime 1 instance 03b2c6b0 _utcLastUpdate
123046c4 4001339       14 ...g.CacheDependency 0 instance 00000000 _dependency
02c36ca0 400133a       18        System.Object 0 instance 033d8344 _onRemovedTargets
120219d0 400132d      1bc      System.DateTime 1   shared   static NoAbsoluteExpiration
    >> Domain:Value 0019daf0:NotInit 11b42540:03395104 <<
1202af88 400132e      1c0      System.TimeSpan 1   shared   static NoSlidingExpiration
    >> Domain:Value 0019daf0:NotInit 11b42540:03395114 <<
1202af88 400132f      1c4      System.TimeSpan 1   shared   static OneYear
    >> Domain:Value 0019daf0:NotInit 11b42540:03395124 <<

This dumped the CacheEntry and it's properties. I figured the most interesting piece of information would be the _value, so I simply copied the address of that property (look in the Value column) and used !dumpobject again.

0:000> !do 03e160c8
Name: System.Web.SessionState.InProcSessionState
MethodTable: 14dbad5c
EEClass: 14e43af8
Size: 48(0x30) bytes
(C:\WINDOWS\assembly\GAC_32\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
1466c9d8 4001d89        4 ...ateItemCollection 0 instance 1a7f5438 _sessionItems
1292672c 4001d8a        8 ...ObjectsCollection 0 instance 00000000 _staticObjects
0fd3da00 4001d8b        c         System.Int32 0 instance       20 _timeout
120261c8 4001d8c       18       System.Boolean 0 instance        0 _locked
120219d0 4001d8d       1c      System.DateTime 1 instance 03e160e4 _utcLockDate
0fd3da00 4001d8e       10         System.Int32 0 instance        1 _lockCookie
1202bf60 4001d8f       24 ...ReadWriteSpinLock 1 instance 03e160ec _spinLock
0fd3da00 4001d90       14         System.Int32 0 instance        0 _flags

Here I found something interesting. The value stored was in fact an InProcSessionState object, which -in case you didn't know this before- is stored in the Cache. This meant that the claim that the application was using SQL Server Session state was incorrect.

As it turned out the customer had temporarily switched to In-process for a brief test, but forgotten to switch back again. Had the application been live they would have spotted this in no time, but since they were stress testing they weren't really paying attention to what the server was returning as long as it returned something. Unknowingly running session state in-process compromised the stress-test in a number of ways. For example:

Memory usage for the worker process was a lot higher than expected
The load on the network was not as high as it would be under normal circumstances
The response times were no longer accountable
The SQL-server was not stressed to the extent it should have been, so any potential bottlenecks there went unnoticed

This was in no way the final solution for their performance issue. There were a lot more things we had to deal with, but I think it's a really nice example of how to use only three commands (!dumpheap, !objsize & !dumpobject) to dig up some really useful information.

Later! / Johan

Clients getting incorrect session data or sessions are inexplicably reset

JohanS — Fri, 02 Nov 2007 17:41:22 GMT

This is something me and my colleagues come across quite often. It is often quite time consuming to troubleshoot, even though there aren't that many possible reasons.

The typical scenario is that a webapplication stores client-specific data in session variables. Intermittently clients will see other clients' data, find that their session has been reset long before the timeout, or receive an IndexOutOfBounds, NullReference-exception or similar, since their session data suddenly appears corrupt or compromised.

Moving to Production

Unfortunately this problem is normally identified first when the application is in production. Normally you don't pay too much attention to the actual data presented during stress tests. Instead you focus on making sure that the response times are within an acceptable range and at best you only throw a quick glance at the web page to ensure that it looks okay. The discovery that you have a glitch in session state is made in production by the actual clients.

Don't blame the Framework

The first thing you hear is usually: - This has to be a bug in the framework. I can almost assure you that it isn't. It's usually cache, code or configuration that's to blame.

The usual suspects

Caching
Static objects
Cookieless sessions
Process recycling
Load balancing
Incorrect data retrieval

Caching

I normally begin troubleshooting these issues by disabling all caching. Remove all <@ OutputCache >-directives from the .aspx-pages and, if you're running Framework 2.0, verify that the outputCache-element is set up correctly in the machine.config and web.configs.

Normally you wouldn't cache client-specific data anyway, so disabling the cache for the affected pages shouldn't have any impact on performance.

You should also investigate what happens on the way from the client to the server. Is the page cached in a proxy in mid-transit? Is the page even requested at all or is it simply retrieved from the browser cache?

Network traces and IIS-logs are a great tool for verifying that the request actually reaches the server, and that the response is, in fact, returned properly.

Static objects

Static objects and ASP.NET can cause headaches sometimes. The common scenario with session state and static objects is that two clients are using a static object at the same time and one client manages to change it while another is reading it. If you have static objects in your code I suggest you investigate thoroughly, possibly eliminate them temporarily for testing purposes and see if the problem still reproduces.

Cookieless sessions

When you use cookieless sessions you store the session ID in the URL. That's not really a safe alternative and not something I'd recommend for applications with sensitive client data. Also, saving bookmarks, etc. can cause all sorts of weird situations when you try to access the application with a non-existing session ID, or even a re-used one that now belongs to a completely different client.

Process recycling

If you're just loosing session data and are using in-process session state, then the cause is most likely to be the w3wp itself. Is the process recycling at the default intervals? Have you changed any of the settings in IIS manager so that it recycles after 300 request? Are you writing to the bin-directory, etc? I wrote a post on common reasons why your application pool may unexpectedly recycle earlier. If you suspect that this is what's happening I suggest you check it out.

Load balancing

Another reason why sessions may reset or possibly even jump between clients is load balancing. Make sure you have configured the state server correctly on all servers and haven't accidentally switched to in-process session state or have one configuration server 1 and another for server 2.

Incorrect data retrieval

If the client is seeing incorrect data you may need to verify that this data is retrieved correctly. By adding trace info to the code, tracking the session ID, and the variables used to retrieve the customer-specific data you may find that there is actually something wrong in the way the data is retrieved, rather than in the session ID.

Conclusion

If this problem happens to you, be prepared to spend a bit of time troubleshooting it. It usually takes a while to figure out where things go wrong. In order to avoid unpleasant surprises you should really take the time to verify that the correct data is displayed before deploying. It is far too easy to stare at the response times and forget about verifying that the data displayed is actually the data you'd expect.

/ Johan

My web application times out when debugging in IIS7

JohanS — Wed, 12 Sep 2007 16:51:00 GMT

Problem:

When using Visual Studio 2005 to debug a web application under IIS7 you will find that after 90-120 seconds the debugger will time out and you will get an appropriate error message in your browser. Visual Studio will return to it's "non debugging"-state and pretend like nothing happened.

Changing the ExecutionTimout property in web.config doesn't change anything, and since you're debugging you'd expect IIS to ignore that setting anyway.

Cause:

With every new software release there are always small changes that will effect performance / behavior in certain situations. I've mentioned this before in my I've upgraded and now my application doesn't work anymore - post. There are always a few scenarios in which a new feature will appear to limit functionality when it is in fact increasing it.

IIS will periodically ping your worker process in order to ensure that it is still responsive. By default it will be pinging the process every 30 seconds, and the process will have 90 seconds to respond. If it doesn't respond in time it is terminated. All in all this means that If your application pool has hung for some reason it will be recycled and everything will return to normal.

This is not a new feature, but it can, however, cause some problems while debugging. When you hit a breakpoint all execution stops, pending requests are not served and the worker process freezes completely. It will not respond to the health monitoring ping and if it doesn't continue execution within this 90-120 window it will be terminated.

Resolution:

There are three settings you can tweak with to resolve this. You'll find them all in the "Process Model"-section of the Advanced Settings for the Application Pool:

Open up the IIS manager
Locate the Application Pool in question
Right-click on it and select "Advanced Settings"
Scroll down to the "Process Model"-section

The settings of interest, and their default values are:

Ping Enabled (True)
Ping Maximum Response Time (90 seconds)
Ping Period (30 seconds)

This means you can either turn off health monitoring altogether by setting "Ping Enabled" to false, or you can tweak with the response time or the interval. Please note that the settings are there for a reason, so once you're getting close to production you really should return them back to their default values. I personally think the settings are excellent. An application pool that has hung will be reset within a maximum of 2 minutes. With a decent out of process State Server setup your clients will probably notice an unusually long response time for one request, but think nothing else of it.

Over and out / Johan

Common reasons why your application pool may unexpectedly recycle

JohanS — Wed, 16 May 2007 11:12:27 GMT

If your application crashes, hangs and deadlocks it will cause/require the application pool to recycle in order to be resolved, but sometimes your application pool inexplicably recycles for no obvious reason. This is usually a configuration issue or due to the fact that you're performing file system operations in the application directory.

For the sake of elimination I thought I'd list the most common reasons.

Application pool settings

If you check the properties for the application pool you'll see a number of settings for recycling the application pool. In IIS6 they are:

Recycle worker processes (in minutes)
Recycle worker process (in requests)
Recycle worker processes at the following times
Maximum virtual memory
Maximum used memory

These settings should be pretty self explanatory, but if you want to read more, please take a look at this MSDN article

The processModel element of machine.config

If you're running IIS5 or the IIS5 isolation mode you'll have to look at the processModel element. The Properties you should pay the closest attention to are:

memoryLimit
requestLimit
timeout

memoryLimit

The default value of memoryLimit is 60. This value is only of interest if you have fairly little memory on a 32 bit machine. 60 stands for 60% of total system memory. So if you have 1 GB of memory the worker process will automatically restart once it reaches a memory usage of 600 MB. If you have 8 GB, on the other hand, the process would theoretically restart when it reaches 4,8 GB, but since it is a 32 bit process it will never grow that big. See my post on 32 bit processes for more information why.

requestLimit

This setting is "infinite" by default, but if it is set to 5000 for example, then ASP.NET will launch a new worker process once it's served 5000 requests.

timeout

The default timeout is "infinite", but here you can set the lifetime of the worker process. Once the timeout is reached ASP.NET will launch a new worker process, so setting this to "00:05:00" would recycle the application every five minutes.

Other properties

There are other properties within the processModel element that will cause your application pool to recycle, like responseDeadlockInterval. But these other settings usually depend on something going wrong or being out of the ordinary to trigger. If you have a deadlock then that's your main concern. Changing the responseDeadlockInterval setting wouldn't do much to resolve the situation. You'd need to deal with the deadlock itself.

Editing and updating

ASP.NET 2.0 depends on File Change Notifications (FCN) to see if the application has been updated. Depending on the change the application pool will recycle. If you or your application is adding and removing directories to the application folder, then you will be restarting your application pool every time, so be careful with those temporary files.

Altering the following files will also trigger an immediate restart of the application pool:

web.config
machine.config
global.asax
Anything in the bin directory or it's sub-directories

Updating the .aspx files, etc. causing a recompile will eventually trigger a restart of the application pool as well. There is a property of the compilation element under system.web that is called numRecompilesBeforeAppRestart. The default value is 20. This means that after 20 recompiles the application pool will recycle.

A workaround to the sub-directory issue

If your application really depends on adding and removing sub-directories you can use linkd to create a directory junction. Here's how:

Create a directory you'd like to exclude from the FCN, E.g. c:\inetpub\wwwroot\WebApp\MyDir
Create a separate folder somewhere outside the wwwroot. E.g. c:\MyExcludedDir
use linkd to link the two: linkd c:\inetpub\wwwroot\WebApp\MyDir c:\MyExcludedDir
Any changes made in the c:\inetpub\wwwroot\WebApp\MyDir will actually occur in c:\MyExcludedDir so they will go unnoticed by the FCN.

Is recycling the application pool really that bad?

You really shouldn't have to recycle the application pool, but if you're dealing with a memory leak in your application and need to buy time to fix it, then by all means recycling the application pool could be a good idea.

What about session state?

Well, if you're running in-process session state, then obviously it's going to be reset each and every time the application pool is recycled. If you need to brush up on your state server options, then I recommend taking a look at this entry.

Later! / Johan

SessionState Performance

JohanS — Mon, 20 Nov 2006 11:49:00 GMT

Problem:

Unable to make the session state request to the session state server.
Please ensure that the ASP.NET State service is started and that the client and server ports are the same.
If the server is on a remote machine, please ensure that it accepts remote requests by checking the value of HKLM\SYSTEM\CurrentControlSet\Services\aspnet_state\Parameters\AllowRemoteConnection.

Cause:

Excessive use of SessionState

Resolution:

Minor tweaks can be made, but be aware. You are pushing SessionState to it's absolute limits.

Scenario:

Judging from the number of calls we get it seems quite common for people to store all sorts of things in SessionState and to rely quite heavily on it. I would personally say that the rule of thumb is to avoid SessionState as much as possible.

The different ways of storing SessionState

You're probably already familiar with the three different ways of storing SessionState, but I'll recap them quickly, just to refresh your memory:

InProc
In process. Works fine if you have only one web server. Great performance, but if the application pool recycles, then your SessionData is lost forever.
State Server
The simple out-of process solution. By using the ASP.NET State Server you can share data between web servers. The ASP.NET State Server can be run either on one of the web servers or on a separate server.
SQL Server
Using SQL server isn't that different from using the State Server. SQL Server may be a tiny bit slower performance-wise, but the main benefit is that you can set up a fail-over cluster.

Common myths

There are some common misunderstandings regarding SessionState and it's uses. The ones I come across most frequently would probably be the following:

Myth:

"SessionState is really efficient. If you store all datasets, etc. in a SessionVariable it's like having the data in RAM, rather than on disk"

Fact:

When storing session data out of process it is serialized, stored and deserialized. This is a CPU- and memory consuming process and if you store large objects in SessionState you will use a lot of resources to handle the data. For small chunks of data I'd recommend cookies, hidden controls (ViewState) or even query strings. For larger objects... Well, do you really need them for every single request?

Myth:

"The Server will only retrieve SessionState when I actually read or write to it."

Fact:

Sorry, but that's incorrect as well. The data will be deserialized and serialized with each request unless you have clearly specified otherwise.

Myth:

"It doesn't matter what type of data I store in SessionState."

Fact:

It does matter, and it matters a lot. For example; a 40KB string and a 40KB multi-dimensional array are not equivalent. The data must be serialized and deserialized, and serializing a string is a lot less CPU-consuming than serializing a complex array.

What is serialization anyway?

Serialization means transforming an object to a stream of bytes. This stream of bytes can then be sent over the web, saved to disk, etc. and be recreated on demand. When serializing the worker process takes your complex object and transform it to a simple byte stream that is easily dealt with. This byte stream in itself is pretty useless until you deserialize it. You can not access its properties until you've "unpacked it". Think of it as a .cab archive or instant coffee. :)

Increasing performance

If you are aware from the start that SessionState isn't the most effective way to store data, then you should be fine. If you're stuck with the ungrateful task of cleaning up someone else's mess, then you have a little more work to do.

The planning phase

Like I said, you should always try to limit your use of SessionState. It's quite common to store data "for later use", but unless you absolutely need it right now, then why store it in Session State? And if you absolutely need it right now, then why not use ViewState, cookies or some other means of temporary storage like a database? Here are some samples:

If you have a huge dataset that was really taxing to get, then save it to a temporary table or take it as a hint to rethink your database design.
Form data should be kept in ViewState or possibly in cookies.
Try really, really hard to keep Objects out of SessionState. Especially STA COM objects. If you really have to save an STA COM object, then remember to set the AspCompat attribute of the @ Page directive.
Disable Session State on all pages that don't require it, and set it to read-only on pages that don't require write-access. This is done with <@ Page EnableSessionState="..."> (Framework 2.0)
Read the article Improving .NET Application Performance and Scalability. Especially Chapter 6 which deals with ASP.NET.

Existing applications

There are a few things worth tryng if you're running an application and suddenly find yourself with a StateServer on it's knees begging for mercy. Please note that most of this is to be considered respiratory only. A few more users and you'll probably be facing the same scenario again. So my personal suggestion would probably be to quickly get the server up and running within acceptable parameters and then immediately begin working on a redesign. Anyway. here are my quick tips for a quick fix:

Disable Session State on all pages that don't use it, or set it to read-only by using the EnableSessionState attribute in the @ Page directive (Framework 2.0)
If you've hit the ceiling and are running the in-process state server, then there might be some extra performance to be gained by setting up an out-of process State Server on a separate machine.
If you're using the out of process StateServer, then you might want to increase the timeout according to the following article:
State server logs Event ID 1072 or Event ID 1076
You might also want to increase the available number of ports on the StateServer by following the guidelines in this article: (I know it doesn't mention StateServer, but the solution is the same.)
PRB: "WSAEADDRESSINUSE" Error Message When You Try to Connect Through an Anonymous Port After You Increase the IMAP Connection Limit
There is also a possibility that this fix will help you out:
FIX: You may receive an error message when you use the ASP.NET State Server service to store ASP.NET session state in the .NET Framework 1.1

More than one StateServer:

If all else fails you can actually have more than one StateServer, provided that you have a webfarm. If you set up your load balancer to use sticky sessions you could actually run each web server with its own StateServer. This would mean that each web server would have its own settings for SessionState. This gives you many options for load balancing. If you, for example, have 8 web servers you could divide the servers into pairs and let each pair share an out of process StateServer. This would give you a solution with 8 web server and 4 state servers. You could even have a separate fail-over cluster for each web server. But as much as this would please your hardware vendor you're probably compensating for poor planning by adding horsepower.

Well I guess that's all for now.

/ Johan