This is a continuation of my previous post with the imaginative name Getting started with windbg - part I. I'll be assuming that you've read it, so if you haven't I suggest you check it out first. We're still working with the same sample dump, so I'll pretty much pick up right where we left off.
Last time we used some nice commands from the sos-extension to look at the running callstacks, the requests, the CPU load of the threads, etc. We also dug deeper into the callstack to see what distinguished name we used for a SearchRequest. We'll keep using these commands, but also learn a few new ones.
Now, imagine that we're looking at a specific thread and would like to see all managed objects referenced to by the current stack. Is there a way for us to do this? - There most certainly is. It's called dumpstackobjects, or dso for short. If we run it on the current thread we'll see a listing of objects that are referenced by the callstack. The whole output would look something like this (trimmed down to size):
This is extremely useful when we want to see all the objects referenced by this thread alone. If you want to analyze one of the objects, simply copy the the address from the Object-column and use !dumpobject:
As you might have noticed we have a couple of object arrays on the stack. Look for the System.Object[]-type in the listing above and you'll find them. If you execute !dumpobject on an array you'll only get information about the array itself, not it's contents. To get information about what's in the array we need to use the !dumparray-command, or !da for short.
As you can see the !dumparray-command gives us a bit more information about the object. We can see that it contains System.String data and we get the addresses of the two items in the array. Since they're System.String-objects we can simply use !dumpobject to view their contents.
If you look at the listing above you'll see that the size of the object is listed as 24 bytes. To paraphrase Obi Wan Kenobi: "This is true, from a certain point of view." The 24 bytes are the size of the System.Object[]-object itself. Not it's contents. As you can see when we execute !dumparray, the array only contains two references to two strings. These strings are individual objects and could be 32 MB for all we know, so the 24 bytes are not the total size of the array, but it would still be correct to claim that the System.Object[] is only 24 bytes large.
To get the total size of the object we use the !objsize command:
!objsize will iterate through all the child objects referenced by the object, as well as all the grandchildren and so on. Apparently the total size of the array and it's child objects is 348 bytes.
If there are a lot of child objects it may take some time for !objsize to calculate the total size of the object. You should also be aware that !objsize may not always be as smart as you'd like. If, for example, you have a custom button control that references it's parent aspx-page you'd get the total size of the aspx-page and all it's child controls. In other words: If !objsize claims that the object in question is ridiculously large you might want to manually check what the object references using !dumpobject.
This is another command that I use frequently. It is, however a command that you will want to use with at least one argument. Dumpheap with no arguments will dump all objects on the heap, so I usually begin by using the -stat argument, which in itself will write a lot of info on the screen, but it will at least be summarized. Here' you'll find the trimmed down output of !dumpheap -stat:
As you can see we now get a listing sorted by size of all object-types on the heap. You'll usually find strings down at the bottom since that's what is commonly used the most.
Other useful arguments are -type and -mt (which stands for MethodTable). Using them you're able to see all objects of a specific type. For example. If we want to look at all HttpRequestCreators on the heap (there are one) you'll simply copy it's MethodTable which you'll find in the listing above (14ef4718) and use !dumpheap -mt
This gives us the address of the object and if we'd like to inspect it closer we simply use !dumpobject on that address.
!dumpheap -type works pretty much the same way, except this time you filter the results by class name. !dumpheap -type performs a substring match, so if you write !dumpheap -type System.Web you'll get every object who's class name contains System.Web, which would be a lot.
Other useful arguments are -min and -max which accept a number representing the minimum/maximum number of bytes the object size should be. This can be really useful when troubleshooting string-abuse, etc. Also !dumpheap -stat -min 85000 would list all objects on the large object heap.
I'd now like to use the commands we've covered in a bit more practical scenario. The dump we've been looking at is from a previous case of mine. The application in question was running on a Web garden with two workerprocesses. Session State was handled by a SQL Server. The customer was experiencing performance issues and the problem description was hazy at best. Anyway I had tons of dumps to work with, so I simply poked around to see what I could find. One thing I did pretty early on was to look at caching. According to the customer they weren't using the cache at all, but I usually find it best to double-check this type of thing.
To find out how much data was kept in the cache I first needed to find the System.Web.Caching.Cache class. So I ran !dumpheap -stat -type System.Web.Caching.Cache. Note that I also used the -stat argument. Otherwise I would have gotten a very long list of System.Web.Caching.CacheKeys and System.Web.Caching.CacheEntrys as well. Anyway, here's the result:
Okay, so now I had the MethodTable for the System.Web.Caching.Cache object. Therefore I could now get the address to the object itself. I did this by asking !dumpheap to list all the objects on the heap with that MethodTable. I knew that there was just going to be one hit:
So now I ran !objsize on this object to see how big it was. This took a little time to calculate, since the cache is quite complex and there are a lot of children to iterate through.
So the cache is 266 MB in size. That's quite a lot considering the fact that the customer claimed that they weren't using the cache at all!
To sample what the customer was caching I then took a look at a few of the CacheEntrys. I already had the MethodTable for the System.Web.Caching.CacheEntry from when I ran !dumpheap -type System.Web.Caching.Cache -stat (above), so I could use that to retrieve all CacheEntrys.
Another valid command that would have given me the exact same output would off course have been !dumpheap -type System.Web.Caching.CacheEntry.
Okay, so now I had a long list of CacheEntrys. To sample the contents I just picked an address and examined it by using !dumpobject
This dumped the CacheEntry and it's properties. I figured the most interesting piece of information would be the _value, so I simply copied the address of that property (look in the Value column) and used !dumpobject again.
Here I found something interesting. The value stored was in fact an InProcSessionState object, which -in case you didn't know this before- is stored in the Cache. This meant that the claim that the application was using SQL Server Session state was incorrect.
As it turned out the customer had temporarily switched to In-process for a brief test, but forgotten to switch back again. Had the application been live they would have spotted this in no time, but since they were stress testing they weren't really paying attention to what the server was returning as long as it returned something. Unknowingly running session state in-process compromised the stress-test in a number of ways. For example:
This was in no way the final solution for their performance issue. There were a lot more things we had to deal with, but I think it's a really nice example of how to use only three commands (!dumpheap, !objsize & !dumpobject) to dig up some really useful information.
Later! / Johan