@TessFerrandez
In my last post I wrote a script to dump out all the ASP.NET requests on the heap. Since one of the most common memory issues I encounter is too much cache or session state I figured that showing you how to retreive session data would be good.
A word of caution, since this script uses !dumpheap (to dump out the objects on the heap) and !objsize (to figure out the size of an object including the size of its membervariables) it may take a long time to execute if you have a very large dump, which is typically the case when you are worried about how much you store in session state.
There is a lot more data that you can get, other than what I am displaying, and the script can pretty easily be modified to display cache objects instead so if you find some cool use or modification of it, please post it in the comments so all of us can benefit from it.
Note: This script only works if you use in-proc session state since it is doing its magic by dumping out all the System.Web.SessionState.InProcSessionState items on the heap. On the other hand, if you have high memory usage due to session state it is probably the in-proc session state items you will be worried about. Btw, if this is what you are running into, check out the post index for other posts on ASP.NET Session memory issues.
Output and comments on output
This script is divided into two parts.
Part #1 (DumpSessions.txt) displays all the System.Web.SessionState.InProcSessionState items with the session address as a DML (Debugger Markup Language) link so that you can click it to get more detailed info.
So in this case we have 3 active sessions, or at least 3 sessions that are still on the heap, the session timeout is set to 20 minutes and the first session holds an amazing ~268 MB (actually more like 256,001 MB, but what's a few MB in the sceme of things).
Part #2 (DumpSessionVar.txt) dumps out more detailed information about each sessions, i.e. the session variables for that particular session and this is what gets executed when you click the DML link.
So in this case the 256 MB session contains 3 session variables. UserName = Joe Smith (88 Bytes), UserType = User (80 Bytes) and ProductsOrdered = ArrayList (268,435,600 bytes). The address of the object stored in session is printed so you can dump it out or explore it further...
An idea for a modification of the script would be to print out all the session variables at once, or add some parsing to get the total size of all sessions. Oh well, there is endless possiblities here...
Finding the data
All in-proc sessions are represented on the heap as System.Web.SessionState.InProcSessionState objects.
0:013> !do 03093d38 Name: System.Web.SessionState.InProcSessionState MethodTable: 663ac580 EEClass: 663ac510 Size: 48(0x30) bytes GC Generation: 1 (C:\WINDOWS\assembly\GAC_32\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll) Fields: MT Field Offset Type VT Attr Value Name 663ecfbc 4001edc 4 ...ateItemCollection 0 instance 03093b18 _sessionItems 663a3390 4001edd 8 ...ObjectsCollection 0 instance 00000000 _staticObjects 791018e0 4001ede c System.Int32 1 instance 20 _timeout 79107584 4001edf 18 System.Boolean 1 instance 0 _locked 791084f8 4001ee0 1c System.DateTime 1 instance 03093d54 _utcLockDate 791018e0 4001ee1 10 System.Int32 1 instance 1 _lockCookie 663aad40 4001ee2 24 ...ReadWriteSpinLock 1 instance 03093d5c _spinLock 791018e0 4001ee3 14 System.Int32 1 instance 0 _flags
and given a particular System.Web.SessionState.InProcSessionState object we can get to the information we want like this...
Size of the session
0:013> !objsize 03093d38 sizeof(03093d38) = 268,437,272 ( 0x10000718) bytes (System.Web.SessionState.InProcSessionState)
Individual Session Variables
We get to the individual session variables by dumping out the _sessionItems member variable, following the steps below...
Note, the number of session variables in the session is available from the _size membervariable of the ArrayList
0:013> !do 03093b18 Name: System.Web.SessionState.SessionStateItemCollection MethodTable: 66411bb4 EEClass: 66411b3c Size: 60(0x3c) bytes GC Generation: 2 (C:\WINDOWS\assembly\GAC_32\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll) Fields: MT Field Offset Type VT Attr Value Name 79107584 4001173 24 System.Boolean 1 instance 0 _readOnly 79105cd4 4001174 4 ...ections.ArrayList 0 instance 03093b60 _entriesArray 791138d4 4001175 8 ...IEqualityComparer 0 instance 03049ce0 _keyComparer 79101634 4001176 c ...ections.Hashtable 0 instance 03093b78 _entriesTable 7a761668 4001177 10 ...e+NameObjectEntry 0 instance 00000000 _nullKeyEntry 7a78e3b0 4001178 14 ...se+KeysCollection 0 instance 00000000 _keys 7910ec98 4001179 18 ...SerializationInfo 0 instance 00000000 _serializationInfo 791018e0 400117a 20 System.Int32 1 instance 4 _version 790fc35c 400117b 1c System.Object 0 instance 00000000 _syncRoot 79115538 400117c 538 ...em.StringComparer 0 shared static defaultComparer >> Domain:Value 0019b498:NotInit 001cc780:NotInit << 79107584 4001f2b 25 System.Boolean 1 instance 1 _dirty 664228fc 4001f2c 28 ...n+KeyedCollection 0 instance 00000000 _serializedItems 79100f74 4001f2d 2c System.IO.Stream 0 instance 00000000 _stream 791018e0 4001f2e 34 System.Int32 1 instance 0 _iLastOffset 790fc35c 4001f2f 30 System.Object 0 instance 03093b54 _serializedItemsLock 79101634 4001f2a c54 ...ections.Hashtable 0 shared static s_immutableTypes >> Domain:Value 0019b498:NotInit 001cc780:0708be1c << 0:013> !do 03093b60 Name: System.Collections.ArrayList MethodTable: 79105cd4 EEClass: 79105c28 Size: 24(0x18) bytes GC Generation: 2 (C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll) Fields: MT Field Offset Type VT Attr Value Name 7912ad90 40008df 4 System.Object[] 0 instance 03093c90 _items 791018e0 40008e0 c System.Int32 1 instance 3 _size 791018e0 40008e1 10 System.Int32 1 instance 3 _version 790fc35c 40008e2 8 System.Object 0 instance 00000000 _syncRoot 7912ad90 40008e3 1c0 System.Object[] 0 shared static emptyArray >> Domain:Value 0019b498:03031ff8 001cc780:030371e4 << 0:013> dc 03093c90 03093c90 7912ad90 00000004 790fc35c 03093c80 ...y....\..y.<.. 03093ca0 03093cb0 03093ce4 00000000 00000000 .<...<.......... 03093cb0 7a761668 030939c8 03093c64 00000000 h.vz.9..d<...... 03093cc0 79105cd4 31180038 00000000 020bb7a8 .\.y8..1........ 03093cd0 020bb7a8 00000000 00194d40 00000000 ........@M...... 03093ce0 00000000 7a761668 03093a14 03093cc0 ....h.vz.:...<.. 03093cf0 00000000 790fcb30 0000001a 00000019 ....0..y........ 03093d00 0061006a 006e0030 00710078 007a006e j.a.0.n.x.q.n.z. 0:013> !do 03093c80 Name: System.Collections.Specialized.NameObjectCollectionBase+NameObjectEntry MethodTable: 7a761668 EEClass: 7a7ce36c Size: 16(0x10) bytes GC Generation: 2 (C:\WINDOWS\assembly\GAC_MSIL\System\2.0.0.0__b77a5c561934e089\System.dll) Fields: MT Field Offset Type VT Attr Value Name 790fcb30 400117d 4 System.String 0 instance 030939a4 Key 790fc35c 400117e 8 System.Object 0 instance 03093c40 Value 0:013> !do 030939a4 Name: System.String MethodTable: 790fcb30 EEClass: 790fca90 Size: 34(0x22) bytes GC Generation: 2 (C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll) String: UserName Fields: MT Field Offset Type VT Attr Value Name 791018e0 4000096 4 System.Int32 1 instance 9 m_arrayLength 791018e0 4000097 8 System.Int32 1 instance 8 m_stringLength 790fe534 4000098 c System.Char 1 instance 55 m_firstChar 790fcb30 4000099 10 System.String 0 shared static Empty >> Domain:Value 0019b498:790d81bc 001cc780:790d81bc << 7912b1d8 400009a 14 System.Char[] 0 shared static WhitespaceChars >> Domain:Value 0019b498:030303f4 001cc780:03034558 << 0:013> !do 03093c40 Name: System.String MethodTable: 790fcb30 EEClass: 790fca90 Size: 36(0x24) bytes GC Generation: 2 (C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll) String: Joe Smith Fields: MT Field Offset Type VT Attr Value Name 791018e0 4000096 4 System.Int32 1 instance 10 m_arrayLength 791018e0 4000097 8 System.Int32 1 instance 9 m_stringLength 790fe534 4000098 c System.Char 1 instance 4a m_firstChar 790fcb30 4000099 10 System.String 0 shared static Empty >> Domain:Value 0019b498:790d81bc 001cc780:790d81bc << 7912b1d8 400009a 14 System.Char[] 0 shared static WhitespaceChars >> Domain:Value 0019b498:030303f4 001cc780:03034558 <<
Running the scripts
To run the scripts copy the text below and put it in two text files somewhere on your harddrive. I've put mine in c:\tools\extensions\DumpSessions.txt and c:\tools\extensions\DumpSessionVar.txt. If you put them somewhere else, make sure you change the path in the DumpSessions.txt script so that you get to the right script file when you click the DML links.
To run it, open up your dump in windbg and load sos.dll, and then run $><c:\tools\extensions\DumpSessions.txt
Script code
DumpSessions.txt
$$ $$ Dumps all sessions on the heap $$ $$ $$ Written by: Tess $$ $$ Run as: $><c:\tools\extensions\DumpSessions.txt $$ $$ CLEAR ALL ALIASES (VARIABLES) $$ ---------------------------------------------------------------------------------- ad /q * r @$t0=0; r @$t1=0; $$ GET ALL SESSION ITEMS .foreach (CurrentSession {!dumpheap -type System.Web.SessionState.InProcSessionState -short}){ $$ Increment # of sessions r @$t0 = @$t0+1 .printf /D "Session Address:\t<?dml?><exec cmd=\"$$>a< c:\\tools\\extensions\\DumpSessionVar.txt ${CurrentSession}\">${CurrentSession}</exec>\n"; .printf "Session Timeout:\t%d\n", poi(${CurrentSession}+0xc); .foreach /pS 2 /ps 99 (token {!objsize ${CurrentSession}}){.printf "Session Size:\t${token} bytes\n"} .printf "_______________________________________________\n"; } .printf "Number of Sessions: %d\n\n\n\n\n\n\n\n\n", @$t0;
DumpSessionVar.txt
$$ $$ Dumps details of a particular session $$ $$ $$ Written by: Tess $$ $$ Run as: $$>a<c:\tools\extensions\DumpSessionVar.txt SessionAddress $$ .printf "Session:\t\t${$arg1}\n"; .foreach /pS 2 /ps 99 (token {!objsize ${$arg1}}){.printf "Session Size:\t${token} bytes\n"} $$ Number of session variables r @$t0 = poi(poi(poi(${$arg1}+0x4)+0x4)+0xc); .for(r @$t1=0; @$t1 < @$t0; r @$t1=@$t1+1){ .printf "\n================================================\n" .printf "Session Variable\n" .printf "================================================\n" $$ NameObject Entry (actual session variable) r @$t2 = poi(poi(poi(poi(${$arg1}+0x4)+0x4)+0x4)+0x8+0x4*(@$t1+1)); $$ Size .foreach /pS 2 /ps 99 (token {!objsize @$t2}){.printf "Sessionvar Size:\t${token} bytes\n"} $$ Key .foreach /pS 5 (tk {.foreach /pS 9 (token {!do -nofields poi(@$t2+0x4)}){.printf "${token} "}}) { .printf /D "Session Key:\t<?dml?><col fg=\"emphfg\">${tk}</col>\n" }; $$ Value .printf "Session Variable:\t%p\n****\n", poi(@$t2+0x8); !do poi(@$t2+0x8) } .printf "\n\n\n\n\n\n\n\n\n"; ad /q *
Final words
If you have any ideas for scripts that you think would be useful please let me know and if I have the time and energy I will probably implement and post them...
Have fun,Tess
Link Listing - September 18, 2007
This is very good info, Tess, thanks !
I don't have any suggestions for useful scripts right now, but I do have two questions that have been nagging me since your blog got me started using SoS :
1) How does !objsize exactly work and how reliable is it ? It seems you could have many sessions that each report being 100 MB in size while in fact they are all pointing to the same 100 MB block of data ?
2) Is there an easy way to filter out unreachable objects from !dumpheap etc. ? I'm asking this because I once had a situation where some code was creating hundreds of thousands of objects very fast, and every dump was full of these - it was very difficult to discern which were temporary and which were actual "leaks". I know of !gcroot but it's slow and not really practical to run it on every object. Ideally there would be a !collect command that would force or simulate a GC on the current heap. But perhaps I am thinking of this in entirely the wrong terms...
Well that's it, if you feel like addressing either or both I'd be grateful. Otherwise it's no matter, your blog has already been of tremendous help.
Cheers,
--Jonathan
Hi Jonathan,
1. You are completely right, you cant add upp the result of !objsize since it could potentially be pointing to the same data. i.e. if two items (a and b) both have a pointer to the same object (c) !objsize for a would include the size of c and !objsize for b would also include the size of c.
However if you have a toplevel object d, pointing to a and b, then the size of c and it's membervariables should only be counted once.
The same thing is true if you have a double-link, i.e a->c and c-> (eg. a control points to the parent page and the page has a list of controls which point to the control)
In this case the size of the control will only be counted once...
So in short, the !objsize of an item is pretty reliable, but the sum of all objsizes will exceed the total amount of memory on the heap.
2. No, there isn't an easy way to do this, and yes, gcroot usually takes a long time because it has to traverse a lot of objects...
The only recommendation I can make there is to run
GC.Collect
GC.WaitForPendingFinalizers
before taking the dump, if you are in a testing environment. That way you will clean out as much as possible, leaving you with almost only reacheable objects.
Hope this helps,
Tess
Here is the latest in my link-listing series . Also check out my ASP.NET Tips, Tricks and Tutorials page
Great blog.
Somehow in Java it seem to be much easier...
Awww, my eyes!!!
Never again textual screenshoots in low quality jpeg, please! :D
Png or Gif heve their best use right in such situations :)
... after manadatory amount of complaining ...
Great info, Thanks!!!
:) I completely see what you're saying... I'll update with better pics shortly...
Niktu,
I found the problem with the pics, turns out that the tool i am using for blogging didn't choose to link to the external source but rather stored the pics on the blog but after having turned them into some low res thumbs of the pics. Now it should look a lot better:) I didn't realize this was happening so the pic quality bothered me too, so i'm glad i figured it out...
I was trying to use the script dumpsessions.txt , i have win2000 Advance server on my machine
I m getting following error :
Address expression missing from ':\Aspnetdumps\Scripts\Dumpsessions.txt'
What will be the possible reason... Its urgent
sounds like the path to your script may be incorrect, perhaps you could post the command you use to load the script so that me or someone else reading the blog can take a look to figure out whats wrong
The purpose of my presentation was to show some common pitfalls and of course to show off windbg and
Tess,
Thanks so much for your awesome labs/tutorials. I went through all of them and have gotten to the point where I'm trying to debug my own asp.net sessions.
Running these scripts seems to corrupt my dump. When I run DumpSessions.txt I get the following:
0:000> $$><DumpSessions.txt
Session Address: 015c8420
Session Size: 208 bytes
_______________________________________________
Session Address: 017e05a8
Session Address: 017ed374
Session Address: 01823aa4
Session Address: 01826570
Session Address: 05693848
Session Size: 6466088 bytes
Session Address: 05859788
Session Address: ------------------------------
Session Size: 1 bytes
This last entry will hang until I break in Windbg. If I try to run DumpSessionVar on my large session, I get a few good variable outputs and then it starts dumping
================================================
Session Variable
Sessionvar Size: not bytes
Session Variable: 0167e128
****
<Note: this object has an invalid CLASS field>
Invalid object
If I try to run either of the scripts, or even a !dumpheap after that, I get the following error message:
0:018> !dumpheap -stat
The garbage collector data structures are not in a valid state for traversal. It is either in the "plan phase," where objects are being moved around, or we are at the initialization or shutdown of the gc heap. Commands related to displaying, finding or traversing objects as well as gc heap segments may not work properly. !dumpheap and !verifyheap may incorrectly complain of heap consistency errors.
Error requesting GC Heap data
Unable to build snapshot of the garbage collector state
Any idea? My dump is from an IIS5 aspnet_wp.exe process with .net 2.0 runtime...
Hi Womp,
based on the output of the 2nd script it looks like your version is not matching the version that the script is written for so the offsets are a little bit off... so you would have to change the offsets in the scripts to match your version...
either way, to get back to a good state you can unload and reload the sos extension, i.e. .unload sos (or path to sos) and then .load sos (or path to sos) again
Well I figured out the first part of the problem...
!dumpheap ... -short was printing out a line of "---------" at the end of the output, which was being taken as one of the sessions in the for loop. Here's some fixed up code for the script:
.foreach (CurrentSession {!dumpheap -type System.Web.SessionState.InProcSessionState -short}){
.if ('${CurrentSession}' == '------------------------------')
{
$$.printf /D "Not an Address\n";
}
.else
$$ Increment # of sessions
r @$t0 = @$t0+1
.printf /D "Session Address:\t<?dml?><exec cmd=\"$$>a< DumpSessionVar.txt ${CurrentSession}\">${CurrentSession}</exec>\n";
.printf "Session Timeout:\t%d\n", poi(${CurrentSession}+0xc);
.foreach /pS 2 /ps 99 (token {!objsize ${CurrentSession}}){.printf "Session Size:\t${token} bytes\n"}
.printf "_______________________________________________\n";
$$.printf /D "An address\n";
Now DumpSessionVar.txt gets quite a bit further before bogging down with the invalid class field errors.... but it still doesn't complete properly and ends up in some kind of corrupted state....