If broken it is, fix it you should

Using the powers of the debugger to solve the problems of the world - and a bag of chips    by Tess Ferrandez, ASP.NET Escalation Engineer (Microsoft)

Debugging Script: Dumping out ASP.NET Session Contents

Debugging Script: Dumping out ASP.NET Session Contents

Rate This
  • Comments 18

In my last post I wrote a script to dump out all the ASP.NET requests on the heap.  Since one of the most common memory issues I encounter is too much cache or session state I figured that showing you how to retreive session data would be good.

A word of caution, since this script uses !dumpheap (to dump out the objects on the heap) and !objsize (to figure out the size of an object including the size of its membervariables) it may take a long time to execute if you have a very large dump, which is typically the case when you are worried about how much you store in session state. 

There is a lot more data that you can get, other than what I am displaying, and the script can pretty easily be modified to display cache objects instead so if you find some cool use or modification of it, please post it in the comments so all of us can benefit from it.

Note: This script only works if you use in-proc session state since it is doing its magic by dumping out all the System.Web.SessionState.InProcSessionState items on the heap.  On the other hand, if you have high memory usage due to session state it is probably the in-proc session state items you will be worried about.  Btw, if this is what you are running into, check out the post index for other posts on ASP.NET Session memory issues.

Output and comments on output

This script is divided into two parts. 

Part #1 (DumpSessions.txt) displays all the System.Web.SessionState.InProcSessionState items with the session address as a DML (Debugger Markup Language) link so that you can click it to get more detailed info.

So in this case we have 3 active sessions, or at least 3 sessions that are still on the heap, the session timeout is set to 20 minutes and the first session holds an amazing ~268 MB (actually more like 256,001 MB, but what's a few MB in the sceme of things).

Part #2 (DumpSessionVar.txt) dumps out more detailed information about each sessions, i.e. the session variables for that particular session and this is what gets executed when you click the DML link.

So in this case the 256 MB session contains 3 session variables.  UserName = Joe Smith (88 Bytes), UserType = User (80 Bytes) and ProductsOrdered = ArrayList (268,435,600 bytes).   The address of the object stored in session is printed so you can dump it out or explore it further...

An idea for a modification of the script would be to print out all the session variables at once, or add some parsing to get the total size of all sessions. Oh well, there is endless possiblities here...

Finding the data

All in-proc sessions are represented on the heap as System.Web.SessionState.InProcSessionState objects.

0:013> !do 03093d38
Name: System.Web.SessionState.InProcSessionState
MethodTable: 663ac580
EEClass: 663ac510
Size: 48(0x30) bytes
GC Generation: 1
 (C:\WINDOWS\assembly\GAC_32\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
663ecfbc  4001edc        4 ...ateItemCollection  0 instance 03093b18 _sessionItems
663a3390  4001edd        8 ...ObjectsCollection  0 instance 00000000 _staticObjects
791018e0  4001ede        c         System.Int32  1 instance       20 _timeout
79107584  4001edf       18       System.Boolean  1 instance        0 _locked
791084f8  4001ee0       1c      System.DateTime  1 instance 03093d54 _utcLockDate
791018e0  4001ee1       10         System.Int32  1 instance        1 _lockCookie
663aad40  4001ee2       24 ...ReadWriteSpinLock  1 instance 03093d5c _spinLock
791018e0  4001ee3       14         System.Int32  1 instance        0 _flags

and given a particular System.Web.SessionState.InProcSessionState object we can get to the information we want like this...

Size of the session  

0:013> !objsize 03093d38
sizeof(03093d38) =  268,437,272 (  0x10000718) bytes (System.Web.SessionState.InProcSessionState)

Individual Session Variables

We get to the individual session variables by dumping out the _sessionItems member variable, following the steps below...

Note, the number of session variables in the session is available from the _size membervariable of the ArrayList 

0:013> !do 03093b18 
Name: System.Web.SessionState.SessionStateItemCollection
MethodTable: 66411bb4
EEClass: 66411b3c
Size: 60(0x3c) bytes
GC Generation: 2
 (C:\WINDOWS\assembly\GAC_32\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
79107584  4001173       24       System.Boolean  1 instance        0 _readOnly
79105cd4  4001174        4 ...ections.ArrayList  0 instance 03093b60 _entriesArray
791138d4  4001175        8 ...IEqualityComparer  0 instance 03049ce0 _keyComparer
79101634  4001176        c ...ections.Hashtable  0 instance 03093b78 _entriesTable
7a761668  4001177       10 ...e+NameObjectEntry  0 instance 00000000 _nullKeyEntry
7a78e3b0  4001178       14 ...se+KeysCollection  0 instance 00000000 _keys
7910ec98  4001179       18 ...SerializationInfo  0 instance 00000000 _serializationInfo
791018e0  400117a       20         System.Int32  1 instance        4 _version
790fc35c  400117b       1c        System.Object  0 instance 00000000 _syncRoot
79115538  400117c      538 ...em.StringComparer  0   shared   static defaultComparer
    >> Domain:Value  0019b498:NotInit  001cc780:NotInit  <<
79107584  4001f2b       25       System.Boolean  1 instance        1 _dirty
664228fc  4001f2c       28 ...n+KeyedCollection  0 instance 00000000 _serializedItems
79100f74  4001f2d       2c     System.IO.Stream  0 instance 00000000 _stream
791018e0  4001f2e       34         System.Int32  1 instance        0 _iLastOffset
790fc35c  4001f2f       30        System.Object  0 instance 03093b54 _serializedItemsLock
79101634  4001f2a      c54 ...ections.Hashtable  0   shared   static s_immutableTypes
    >> Domain:Value  0019b498:NotInit  001cc780:0708be1c <<
0:013> !do 03093b60 Name: System.Collections.ArrayList MethodTable: 79105cd4 EEClass: 79105c28 Size: 24(0x18) bytes GC Generation: 2 (C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll) Fields: MT Field Offset Type VT Attr Value Name 7912ad90 40008df 4 System.Object[] 0 instance 03093c90 _items 791018e0 40008e0 c System.Int32 1 instance 3 _size 791018e0 40008e1 10 System.Int32 1 instance 3 _version 790fc35c 40008e2 8 System.Object 0 instance 00000000 _syncRoot 7912ad90 40008e3 1c0 System.Object[] 0 shared static emptyArray >> Domain:Value 0019b498:03031ff8 001cc780:030371e4 <<
0:013> dc 03093c90 03093c90 7912ad90 00000004 790fc35c 03093c80 ...y....\..y.<.. 03093ca0 03093cb0 03093ce4 00000000 00000000 .<...<.......... 03093cb0 7a761668 030939c8 03093c64 00000000 h.vz.9..d<...... 03093cc0 79105cd4 31180038 00000000 020bb7a8 .\.y8..1........ 03093cd0 020bb7a8 00000000 00194d40 00000000 ........@M...... 03093ce0 00000000 7a761668 03093a14 03093cc0 ....h.vz.:...<.. 03093cf0 00000000 790fcb30 0000001a 00000019 ....0..y........ 03093d00 0061006a 006e0030 00710078 007a006e j.a.0.n.x.q.n.z.
0:013> !do 03093c80 Name: System.Collections.Specialized.NameObjectCollectionBase+NameObjectEntry MethodTable: 7a761668 EEClass: 7a7ce36c Size: 16(0x10) bytes GC Generation: 2 (C:\WINDOWS\assembly\GAC_MSIL\System\2.0.0.0__b77a5c561934e089\System.dll) Fields: MT Field Offset Type VT Attr Value Name 790fcb30 400117d 4 System.String 0 instance 030939a4 Key 790fc35c 400117e 8 System.Object 0 instance 03093c40 Value
0:013> !do 030939a4 Name: System.String MethodTable: 790fcb30 EEClass: 790fca90 Size: 34(0x22) bytes GC Generation: 2 (C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll) String: UserName Fields: MT Field Offset Type VT Attr Value Name 791018e0 4000096 4 System.Int32 1 instance 9 m_arrayLength 791018e0 4000097 8 System.Int32 1 instance 8 m_stringLength 790fe534 4000098 c System.Char 1 instance 55 m_firstChar 790fcb30 4000099 10 System.String 0 shared static Empty >> Domain:Value 0019b498:790d81bc 001cc780:790d81bc << 7912b1d8 400009a 14 System.Char[] 0 shared static WhitespaceChars >> Domain:Value 0019b498:030303f4 001cc780:03034558 <<
0:013> !do 03093c40 Name: System.String MethodTable: 790fcb30 EEClass: 790fca90 Size: 36(0x24) bytes GC Generation: 2 (C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll) String: Joe Smith Fields: MT Field Offset Type VT Attr Value Name 791018e0 4000096 4 System.Int32 1 instance 10 m_arrayLength 791018e0 4000097 8 System.Int32 1 instance 9 m_stringLength 790fe534 4000098 c System.Char 1 instance 4a m_firstChar 790fcb30 4000099 10 System.String 0 shared static Empty >> Domain:Value 0019b498:790d81bc 001cc780:790d81bc << 7912b1d8 400009a 14 System.Char[] 0 shared static WhitespaceChars >> Domain:Value 0019b498:030303f4 001cc780:03034558 <<  

Running the scripts

To run the scripts copy the text below and put it in two text files somewhere on your harddrive.  I've put mine in c:\tools\extensions\DumpSessions.txt and c:\tools\extensions\DumpSessionVar.txt.   If you put them somewhere else, make sure you change the path in the DumpSessions.txt script so that you get to the right script file when you click the DML links.  

To run it, open up your dump in windbg and load sos.dll, and then run $><c:\tools\extensions\DumpSessions.txt

Script code

DumpSessions.txt 

$$
$$ Dumps all sessions on the heap
$$
$$
$$ Written by: Tess 
$$
$$ Run as: $><c:\tools\extensions\DumpSessions.txt
$$

$$ CLEAR ALL ALIASES (VARIABLES)
$$ ---------------------------------------------------------------------------------- 
ad /q *

r @$t0=0;
r @$t1=0;

$$ GET ALL SESSION ITEMS
.foreach (CurrentSession {!dumpheap -type System.Web.SessionState.InProcSessionState -short}){
	$$ Increment # of sessions
	r @$t0 = @$t0+1

	.printf /D "Session Address:\t<?dml?><exec cmd=\"$$>a< c:\\tools\\extensions\\DumpSessionVar.txt ${CurrentSession}\">${CurrentSession}</exec>\n";
	.printf "Session Timeout:\t%d\n", poi(${CurrentSession}+0xc);
	.foreach /pS 2 /ps 99 (token {!objsize ${CurrentSession}}){.printf "Session Size:\t${token} bytes\n"}
	.printf "_______________________________________________\n";
}

.printf "Number of Sessions: %d\n\n\n\n\n\n\n\n\n", @$t0;

 

 DumpSessionVar.txt

$$
$$ Dumps details of a particular session
$$
$$
$$ Written by: Tess 
$$
$$ Run as: $$>a<c:\tools\extensions\DumpSessionVar.txt SessionAddress
$$

.printf "Session:\t\t${$arg1}\n";
.foreach /pS 2 /ps 99 (token {!objsize ${$arg1}}){.printf "Session Size:\t${token} bytes\n"}

$$ Number of session variables
r @$t0 = poi(poi(poi(${$arg1}+0x4)+0x4)+0xc);

.for(r @$t1=0; @$t1 < @$t0; r @$t1=@$t1+1){ .printf "\n================================================\n" .printf "Session Variable\n" .printf "================================================\n" $$ NameObject Entry (actual session variable) r @$t2 = poi(poi(poi(poi(${$arg1}+0x4)+0x4)+0x4)+0x8+0x4*(@$t1+1)); $$ Size .foreach /pS 2 /ps 99 (token {!objsize @$t2}){.printf "Sessionvar Size:\t${token} bytes\n"} $$ Key .foreach /pS 5 (tk {.foreach /pS 9 (token {!do -nofields poi(@$t2+0x4)}){.printf "${token} "}}) { .printf /D "Session Key:\t<?dml?><col fg=\"emphfg\">${tk}</col>\n" }; $$ Value .printf "Session Variable:\t%p\n****\n", poi(@$t2+0x8); !do poi(@$t2+0x8) } .printf "\n\n\n\n\n\n\n\n\n"; ad /q *

 

Final words

If you have any ideas for scripts that you think would be useful please let me know and if I have the time and energy I will probably implement and post them...

Have fun,
Tess 





  • Link Listing - September 18, 2007

  • This is very good info, Tess, thanks !

    I don't have any suggestions for useful scripts right now, but I do have two questions that have been nagging me since your blog got me started using SoS :

    1) How does !objsize exactly work and how reliable is it ? It seems you could have many sessions that each report being 100 MB in size while in fact they are all pointing to the same 100 MB block of data ?

    2) Is there an easy way to filter out unreachable objects from !dumpheap etc. ? I'm asking this because I once had a situation where some code was creating hundreds of thousands of objects very fast, and every dump was full of these - it was very difficult to discern which were temporary and which were actual "leaks". I know of !gcroot but it's slow and not really practical to run it on every object. Ideally there would be a !collect command that would force or simulate a GC on the current heap. But perhaps I am thinking of this in entirely the wrong terms...

    Well that's it, if you feel like addressing either or both I'd be grateful. Otherwise it's no matter, your blog has already been of tremendous help.

    Cheers,

    --Jonathan

  • Hi Jonathan,

    1.  You are completely right, you cant add upp the result of !objsize since it could potentially be pointing to the same data. i.e. if two items (a and b) both have a pointer to the same object (c) !objsize for a would include the size of c and !objsize for b would also include the size of c.

    However if you have a toplevel object d, pointing to a and b, then the size of c and it's membervariables should only be counted once.  

    The same thing is true if you have a double-link, i.e a->c and c->  (eg. a control points to the parent page and the page has a list of controls which point to the control)

    In this case the size of the control will only be counted once...

    So in short, the !objsize of an item is pretty reliable, but the sum of all objsizes will exceed the total amount of memory on the heap.

    2.  No, there isn't an easy way to do this, and yes, gcroot usually takes a long time because it has to traverse a lot of objects...

    The only recommendation I can make there is to run

     GC.Collect

     GC.WaitForPendingFinalizers

     GC.Collect

    before taking the dump, if you are in a testing environment.  That way you will clean out as much as possible, leaving you with almost only reacheable objects.

    Hope this helps,

    Tess

  • Here is the latest in my link-listing series . Also check out my ASP.NET Tips, Tricks and Tutorials page

  • Here is the latest in my link-listing series . Also check out my ASP.NET Tips, Tricks and Tutorials page

  • Great blog.

    Somehow in Java it seem to be much easier...

  • Awww, my eyes!!!

    Never again textual screenshoots in low quality jpeg, please! :D

    Png or Gif heve their best use right in such situations :)

    ... after manadatory amount of complaining ...

    Great info, Thanks!!!

  • :) I completely see what you're saying... I'll update with better pics shortly...

  • Niktu,

    I found the problem with the pics, turns out that the tool i am using for blogging didn't choose to link to the external source but rather stored the pics on the blog but after having turned them into some low res thumbs of the pics.  Now it should look a lot better:)  I didn't realize this was happening so the pic quality bothered me too, so i'm glad i figured it out...

  • I was trying to use the script dumpsessions.txt , i have win2000 Advance server on my machine

    I m getting following error :

    Address expression missing from ':\Aspnetdumps\Scripts\Dumpsessions.txt'

    What will be the possible reason... Its urgent

  • sounds like the path to your script may be incorrect, perhaps you could post the command you use to load the script so that me or someone else reading the blog can take a look to figure out whats wrong

  • The purpose of my presentation was to show some common pitfalls and of course to show off windbg and

  • Tess,

    Thanks so much for your awesome labs/tutorials.  I went through all of them and have gotten to the point where I'm trying to debug my own asp.net sessions.

    Running these scripts seems to corrupt my dump.  When I run DumpSessions.txt I get the following:

    0:000> $$><DumpSessions.txt

    Session Address: 015c8420

    Session Size: 208 bytes

    _______________________________________________

    Session Address: 017e05a8

    Session Size: 208 bytes

    _______________________________________________

    Session Address: 017ed374

    Session Size: 208 bytes

    _______________________________________________

    Session Address: 01823aa4

    Session Size: 208 bytes

    _______________________________________________

    Session Address: 01826570

    Session Size: 208 bytes

    _______________________________________________

    Session Address: 05693848

    Session Size: 6466088 bytes

    _______________________________________________

    Session Address: 05859788

    Session Size: 208 bytes

    _______________________________________________

    Session Address: ------------------------------

    Session Size: 1 bytes

    _______________________________________________

    This last entry will hang until I break in Windbg.  If I try to run DumpSessionVar on my large session, I get a few good variable outputs and then it starts dumping

    ================================================

    Session Variable

    ================================================

    Sessionvar Size: not bytes

    Session Variable: 0167e128

    ****

    <Note: this object has an invalid CLASS field>

    Invalid object

    If I try to run either of the scripts, or even a !dumpheap after that, I get the following error message:

    0:018> !dumpheap -stat

    The garbage collector data structures are not in a valid state for traversal. It is either in the "plan phase," where objects are being moved around, or we are at the initialization or shutdown of the gc heap. Commands related to displaying, finding or traversing objects as well as gc heap segments may not work properly. !dumpheap and !verifyheap may incorrectly complain of heap consistency errors.

    Error requesting GC Heap data

    Unable to build snapshot of the garbage collector state

    Any idea?  My dump is from an IIS5 aspnet_wp.exe process with .net 2.0 runtime...

  • Hi Womp,

    based on the output of the 2nd script it looks like your version is not matching the version that the script is written for so the offsets are a little bit off... so you would have to change the offsets in the scripts to match your version...

    either way, to get back to a good state you can unload and reload the sos extension, i.e.   .unload sos (or path to sos) and then .load sos (or path to sos) again

  • Well I figured out the first part of the problem...

    !dumpheap ... -short was printing out a line of "---------" at the end of the output, which was being taken as one of the sessions in the for loop.  Here's some fixed up code for the script:

    .foreach (CurrentSession {!dumpheap -type System.Web.SessionState.InProcSessionState -short}){

     .if ('${CurrentSession}' == '------------------------------')

     {

      $$.printf /D "Not an Address\n";

     }

    .else

    {

    $$ Increment # of sessions

    r @$t0 = @$t0+1

    .printf /D "Session Address:\t<?dml?><exec cmd=\"$$>a< DumpSessionVar.txt ${CurrentSession}\">${CurrentSession}</exec>\n";

    .printf "Session Timeout:\t%d\n", poi(${CurrentSession}+0xc);

    .foreach /pS 2 /ps 99 (token {!objsize ${CurrentSession}}){.printf "Session Size:\t${token} bytes\n"}

    .printf "_______________________________________________\n";

     $$.printf /D "An address\n";

    }

    }

    Now DumpSessionVar.txt gets quite a bit further before bogging down with the invalid class field errors.... but it still doesn't complete properly and ends up in some kind of corrupted state....

Page 1 of 2 (18 items) 12
Leave a Comment
  • Please add 7 and 1 and type the answer here:
  • Post