A Taste of Logic : Debugging

Creating Strings - A Case Study

jamesche — Tue, 11 Dec 2007 22:45:27 GMT

I recently worked an issue with a customer who was experiencing high CPU. Performance Monitor data showed that the % of Time in GC was 46%. We generally recommend that you not be in GC more than 5% of the time, so this was a point of interest.

One of the common causes of very frequent GC is GC.Collect() being called. I checked the # of Induced GC Perfmon counter and it was greater than 8,000 with a process uptime of only a few hours. This was obviously a clear sign that someone was calling GC.Collect() a lot! ASP.NET will call GC.Collect() at times, but not nearly this often.

We found a DLL that was causing that problem and fixed it. Good news. Induced GC was now 0. Bad news. CPU and Time in GC were both still high.

Many issues that we work are like this; like peeling an onion. We resolve one part of the issue only to discover other problems coming to the surface. This was just such an issue. Once we resolved induced GCs, a pattern became clear.

Have a look at the Perfmon data shown below. The color-key appears below the image:

Red: % of Time in GC
Blue: % Processor Time
Tan: # of Bytes in All Heaps
Black: Large Object Heap Size

The first thing to notice is that things started out okay and then went bad very quickly. One of the things that can cause that pattern is a problem call stack. (In fact, that's the issue here. More soon.) We can also see that something is creating a lot of objects and that's causing considerable memory pressure. That's causing us to go into GC often, but even when we're not in GC, you can see that CPU remains high.

The first thing I decided to do was to check the Large Object Heap to see what we had there.

0:046> !dumpheap -min 85000 -stat

Statistics:

MT Count TotalSize Class Name

0x000f3198 69 87,714,456 Free

0x03084300 123 158,192,160 System.String

Total 192 objects, Total size: 245,906,616

This tells me that the memory we see in the LOH is taken up by strings. In fact, when I looked at some of these strings in the debugger, they turned out to be XML data generated by the customer.

I then checked to see where one of these big strings was rooted. It was rooted in a thread, so I checked that stack and saw this:

0:053> kL

ChildEBP RetAddr

0635b890 791c134c mscorsvr!wstrcopy+0x26

0635b8a4 035ac67c mscorsvr!COMString::FillStringChecked+0x37

This tells me that we're making a copy of a string, so I checked the managed stack. When I did, I found a couple of interesting problems. First of all, the customer had a function call that was recursive and had hundreds of iterations already. Secondly, I could tell that each iteration of that function created more strings via wstrcopy. This is a clear sign of string concatenation, and indeed, in another dump I could see concat being called.

In the end, the customer's allocation pattern caused memory pressure and caused us to go into GC much more often than we should have. (In particular, the customer was creating a large number of objects on almost a per-request basis.) Because of this, the customer experienced high CPU and poor application performance.

I hope this helps others who might see the same kind of thing in production applications.

Jim

Getting to the Root Cause of SqlExceptions

jamesche — Thu, 01 Mar 2007 02:18:00 GMT

It's pretty common for me to work cases where something bad happens when an ASP.NET application is accessing a database. The nature of my job dictates that troubleshooting on these applications is most often performed in a production environment via post mortem debugging. Fortunately, our toolset for doing that kind of work is really great.

If you aren't familiar with how to load up a dump in Windbg, set up symbols, etc., check out my post on that topic.

Finding out exactly what errors occurred when hitting a database is easy to do, but it's time consuming. Let's have a look at a particular SqlException that occurred when accessing a database.

0:000> !do 106fd670
Name: System.Data.SqlClient.SqlException
MethodTable 0x0343232c
EEClass 0x034ffef4
Size 68(0x44) bytes
GC Generation: 0
mdToken: 0x020001c4 (c:\windows\assembly\gac\system.data\1.0.5000.0__b77a5c561934e089\system.data.dll)
FieldDesc*: 0x03432228
MT Field Offset Type Attr Value Name
0x79b96824 0x400001d 0x4 CLASS instance 0x00000000 _className
0x79b96824 0x400001e 0x8 CLASS instance 0x00000000 _exceptionMethod
0x79b96824 0x400001f 0xc CLASS instance 0x00000000 _exceptionMethodString
0x79b96824 0x4000020 0x10 CLASS instance 0x1c2ef88c _message
0x79b96824 0x4000021 0x14 CLASS instance 0x00000000 _innerException
0x79b96824 0x4000022 0x18 CLASS instance 0x00000000 _helpURL
0x79b96824 0x4000023 0x1c CLASS instance 0x106fd8c8 _stackTrace
0x79b96824 0x4000024 0x20 CLASS instance 0x00000000 _stackTraceString
0x79b96824 0x4000025 0x24 CLASS instance 0x00000000 _remoteStackTraceString
0x79b96824 0x4000026 0x2c System.Int32 instance 0 _remoteStackIndex
0x79b96824 0x4000027 0x30 System.Int32 instance -2146232060 _HResult
0x79b96824 0x4000028 0x28 CLASS instance 0x00000000 _source
0x79b96824 0x4000029 0x34 System.Int32 instance 0 _xptrs
0x79b96824 0x400002a 0x38 System.Int32 instance -532459699 _xcode
0x0343232c 0x4000fb6 0x3c CLASS instance 0x106fd6b4 _errors

If you look at the _message field, you'll see this:

0:000> !do 0x1c2ef88c
Name: System.String
MethodTable 0x79b94638
EEClass 0x79b94984
Size 44(0x2c) bytes
GC Generation: 2
mdToken: 0x0200000f (c:\windows\microsoft.net\framework\v1.1.4322\mscorlib.dll)
String: System error.

This is a generic error that tells you absolutely nothing. Essentially, it's telling you that there was an error. The key to finding out what that error actually was is the _errors field in the exception. (It's the last field listed in the exception above.) The _errors field is a SqlErrorsCollection as shown here:

0:000> !do 0x106fd6b4
Name: System.Data.SqlClient.SqlErrorCollection
MethodTable 0x0343249c
EEClass 0x034fff58
Size 12(0xc) bytes
GC Generation: 0
mdToken: 0x020001c3 (c:\windows\assembly\gac\system.data\1.0.5000.0__b77a5c561934e089\system.data.dll)
FieldDesc*: 0x034323bc
MT Field Offset Type Attr Value Name
0x0343249c 0x4000fb5 0x4 CLASS instance 0x106fd6c0 errors

The errors field here is an ArrayList of SqlError objects. Let's have a look at that.

0:000> !do 0x106fd6c0
Name: System.Collections.ArrayList
MethodTable 0x79ba2ee4
EEClass 0x79ba3020
Size 24(0x18) bytes
GC Generation: 0
mdToken: 0x02000100 (c:\windows\microsoft.net\framework\v1.1.4322\mscorlib.dll)
FieldDesc*: 0x79ba3084
MT Field Offset Type Attr Value Name
0x79ba2ee4 0x4000362 0x4 CLASS instance 0x106fd6d8 _items
0x79ba2ee4 0x4000363 0xc System.Int32 instance 1 _size
0x79ba2ee4 0x4000364 0x10 System.Int32 instance 1 _version
0x79ba2ee4 0x4000365 0x8 CLASS instance 0x00000000 _syncRoot

Now, to get to the actual errors in that ArrayList, you need to dump the _items field, and to do that, you need to use the -v switch as follows:

0:000> !do -v 0x106fd6d8
Name: System.Object[]
MethodTable 0x020a209c
EEClass 0x020a2018
Size 80(0x50) bytes
GC Generation: 0
Array: Rank 1, Type CLASS
Element Type: System.Object
Content: 16 items
------ Will only dump out valid managed objects ----
Address MT Class Name
0x106fd780 0x0343263c System.Data.SqlClient.SqlError
----------

Alright, now we're getting somewhere. We have the specific SqlError instance that is associated with the SqlException that occurred. Let's dump that.

0:000> !do 0x106fd780
Name: System.Data.SqlClient.SqlError
MethodTable 0x0343263c
EEClass 0x03710010
Size 36(0x24) bytes
GC Generation: 0
mdToken: 0x020001c2 (c:\windows\assembly\gac\system.data\1.0.5000.0__b77a5c561934e089\system.data.dll)
FieldDesc*: 0x03432520
MT Field Offset Type Attr Value Name
0x0343263c 0x4000fad 0x4 CLASS instance 0x182f7d34 source
0x0343263c 0x4000fae 0x14 System.Int32 instance -2 number
0x0343263c 0x4000faf 0x1c System.Byte instance 0 state
0x0343263c 0x4000fb0 0x1d System.Byte instance 10 errorClass
0x0343263c 0x4000fb1 0x8 CLASS instance 0x1c2eb298 server
0x0343263c 0x4000fb2 0xc CLASS instance 0x14384218 message
0x0343263c 0x4000fb3 0x10 CLASS instance 0x106fd73c procedure
0x0343263c 0x4000fb4 0x18 System.Int32 instance 0 lineNumber

Now we have something we can use. We have the server, the error message, and the procedure. If we look at the error message, here's what we see:

0:000> !do 0x14384218
Name: System.String
MethodTable 0x79b94638
EEClass 0x79b94984
Size 248(0xf8) bytes
GC Generation: 2
mdToken: 0x0200000f (c:\windows\microsoft.net\framework\v1.1.4322\mscorlib.dll)
String: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.

That's a heck of a lot more useful than "System error," but it was a lot of work! You can save yourself some effort by scripting this. Each dump of an object contains an offset for each field. The field exists at that offset. In the SqlError above, the message field is at an offset of 0xc.

0x0343263c 0x4000fb2 0xc CLASS instance 0x14384218 message

Therefore, you can get the message of this exception by dumping the address located at an offset of 0xc from the address of the SqlError itself. You can do that like this:

0:000> !do poi(0x106fd780+c)
Name: System.String
MethodTable 0x79b94638
EEClass 0x79b94984
Size 248(0xf8) bytes
GC Generation: 2
mdToken: 0x0200000f (c:\windows\microsoft.net\framework\v1.1.4322\mscorlib.dll)
String: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.

Look familiar? This is an important concept because it means that you can take the original SqlException object and follow the trail all the way to the actual error message.

What if you want to just dump out details of all SQL errors that are on the heap? That's easy to, and the easiest way to do it is to write a script that you can then run inside of Windbg. What appears below is just such a script:

.logopen ${$CurrentDumpPath}\SqlErrors.txt
.printf "Dumping all SqlError objects on the heap..."
.printf "\n\n"

.foreach (se {!dumpheap -short -mt 0x0343263c})
{

.echo "=============================================================="
.echo "SqlError : ${se}"
.printf "Server Instance: "
du poi(${se}+8)+c
.printf "Procedure: "
du poi(${se}+10)+c
.printf "Error Message: "
du poi(${se}+c)+c
.printf "\n\n"

}
.logclose

The first thing this script does is open a log file in the location where the dump file exists. That way, you'll automatically have a text file containing all of the errors. It then uses a .foreach token to loop through all of the System.Data.SqlClient.SqlError objects. (In other words, all objects in MethodTable 0x0343263c.) For each one of those suckers, it dumps out the different fields using the offsets we looked at before.

All of these fields are text values, and text in the Framework is represented using Unicode. That's why I use the du command (dump Unicode) to dump that out. You'll also notice that each one is dumping out the address+c. Unicode strings are at an offset of 12 from the address of the string, so you have to dump out the string starting at address+c.

If you save the above into a text file and then copy it into the same folder where the debuggers are installed, you can then run it from within Windbg like this:

$><dumpsqlerrors.txt

That will give you output similar to this:

0:000> $><dumpsqlerrors.txt
Opened log file 'C:\Users\jamesche\Desktop\SqlErrors.txt'
Dumping all SqlError objects on the heap...

==============================================================
SqlError : 0x106fd780
Server Instance: 1c2eb2a4 "DBASESVR"
Procedure: 106fd748 "ConnectionRead (recv())."
Error Message: 14384224 "Timeout expired. The timeout pe"
14384264 "riod elapsed prior to completion"
143842a4 " of the operation or the server "
143842e4 "is not responding."

==============================================================
SqlError : 0x10721d38
Server Instance: 1c2eb2a4 "DBASESVR"
Procedure: 10721d00 "ConnectionRead (recv())."
Error Message: 14384224 "Timeout expired. The timeout pe"
14384264 "riod elapsed prior to completion"
143842a4 " of the operation or the server "
143842e4 "is not responding."

==============================================================
SqlError : 0x185266a8
Server Instance: 1c2eb2a4 "DBASESVR"
Procedure: 18526670 "ConnectionRead (recv())."
Error Message: 14384224 "Timeout expired. The timeout pe"
14384264 "riod elapsed prior to completion"
143842a4 " of the operation or the server "
143842e4 "is not responding."

==============================================================
SqlError : 0x18549968
Server Instance: 18549948 "SQL-01"
Procedure: 1c2d0230 ""
Error Message: 185498d0 "Changed language setting to us_e"
18549910 "nglish."

==============================================================
SqlError : 0x1c42e894
Server Instance: 1c2eb2a4 "DBASESVR"
Procedure: 1c42e85c "ConnectionRead (recv())."
Error Message: 14384224 "Timeout expired. The timeout pe"
14384264 "riod elapsed prior to completion"
143842a4 " of the operation or the server "
143842e4 "is not responding."

==============================================================
SqlError : 0x1c47faa8
Server Instance: 1c2eb2a4 "DBASESVR"
Procedure: 1c47fa70 "ConnectionRead (recv())."
Error Message: 14384224 "Timeout expired. The timeout pe"
14384264 "riod elapsed prior to completion"
143842a4 " of the operation or the server "
143842e4 "is not responding."

Closing open log file C:\Users\jamesche\Desktop\DumpSqlErrors.txt

The layout of the SqlError object isn't static. I wrote this script for ASP.NET 1.1 SP1.

Generating a Dump In Your Code Using DebugBreak

jamesche — Wed, 12 Apr 2006 20:29:00 GMT

There are many issues that pop up that really require a user mode dump in order to successfully diagnose root cause. I will occasionally visit the ASP.NET newsgroups, and when I do, I read only those posts with a subject that interests me. I find that I gravitate towards crashes, high memory issues, hangs . . . in other words, those issues that require a user-mode debug in order to reach resolution.

Hangs, high memory, crashes, and other such issues lend themselves well to getting a dump easily. Adplus is a great tool for such things, and if you're dealing with any of the aforementioned issues, getting a dump is a no-brainer. However, there are many issues that require dump analysis that aren't so straight-forward. I'm working just such an issue right now.

I have a high-visibility customer experiencing a pretty serious problem. Their Web application is occasionally allowing one person to see another person's data. When this happens, the nature of the issue causes a System.Net.WebException to occur. In order to really dig into what's going on, we need to look into the down and dirty details of the HttpRequest and HttpResponse object at precisely the moment that the issue occurs. (When I say "precisely", I mean just that. A tenth of a second too late and we're toast.) We need a dump at just that moment.

We can use an adplus configuration file to dump when a WebException occurs, but that causes us to dump like crazy for every single occurrence of this exception type. That won't do. The application is throwing that exception for other reasons and we fill up disk space pretty quickly with dump files that are useless to us. We need a way to focus on just the specific WebException that's occurring when the problem occurs. Fortunately, we know the page that is executing when the error occurs. We can use that to get a dump by making an explicit call to kernel32!DebugBreak via P/Invoke. I have never tried this before, but the concept is a proven technique. No doubt about it; this is the way to go in this case.

In this particular case, the customer is making an HttpWebRequest call that sometimes results in the WebException we're interested in. The first step is to wrap the WebRequest call in a try...catch block and catch the WebException.

try
{
    // make the HttpWebRequest call

}
catch(System.Net.WebException e)
{
    // call DebugBreak
    DebugBreak();
}

Now we need to add the P/Invoke call to DebugBreak.

[System.Runtime.InteropServices.DllImport("kernel32.dll", SetLastError=true)]
internal static extern void DebugBreak();

In order for all of this to work, we need to use Adplus to hook up the debugger (CDB in this case) and monitor for the BreakpointException breakpoint. I'll use an Adplus configuration file to do that. It looks like this.

<ADPlus>
  <Exceptions>
    <Option> NoDumpOnFirstChance </Option>
    <Config>
      <Code> bpec </Code>
      <Actions1> FullDump </Actions1>
      <ReturnAction1> gh </ReturnAction1>
    </Config>
</Exceptions>
</ADPlus>

This is a pretty simple config file. The <Code> section defines a BreakpointException via the use of bpec. <Actions1> indicates the action we will take on first-chance exceptions that match the code. <ReturnAction1> specifies what we'll do after we dump. In this case, after dumping, we're executing a "gh" command which means "go / consider the exception handled." I've saved this config file into the same directory where the Debugging Tools for Windows are installed and named it debugbreak.cfg.

Now we need to run Adplus with the following command line.

cscript adplus.vbs -crash -pn w3wp.exe -c debugbreak.cfg -o c:\dbrkdump

Everything's set up now and we're ready to capture a dump when the problem occurs.

You can use this same technique anytime that you need to make your code cause a dump to be created. In fact, if you are writing code that you think might produce issues where a dump is required, you can put such code in your application in various places so that you can create a dump in various situations. If you decide to do that, it's advisable to check a Registry key or use some other method to determine if you should be calling DebugBreak.

Out of memory! Are you nuts!?

jamesche — Sat, 21 Jan 2006 03:00:00 GMT

I may have said this before (my memory's not what it was prior to decade number 4), but the most common issues I debug are memory issues. These range from high memory issues (categorized as 800MB of more memory in use by the worker process) to OutOfMemory exceptions, or OOM. High memory cases don't usually accompany any questions from customers other than "what the heck is taking so much memory?", but OOM conditions are often accompanied by "Why am I seeing OOM when the process is using 1GB of memory on a box with 4GB of memory available?" That's a good question, and fortunately, that question has a good answer as well.

Before we go into why you might see an OOM exception, you first need to understand the basics of how we allocate memory in the .NET world. We allocate memory in large contiguous blocks. Any objects you create are created in our heaps that are located within the memory segments we've allocated. As long as there is sufficient free space in the managed heap for your object, we won't allocate more memory from the OS. As objects are freed from gen 0, 1, and 2, we compact the heap so that you don't run into any problems with memory fragmentation in those heaps. The large object heap (LOH), which is where any object that's 85K or larger lives, is never compacted. However, we do maintain a list of free blocks in the LOH segments, and we'll reuse free blocks if we can. Therefore, fragmentation in the LOH is rarely an issue.

With that out of the way, suppose you need to allocate an object that is 40K in size and we don't have enough memory on the gen 0 heap to create that object. We'll then go to the OS and attempt to allocate another contiguous segment. If we're unable to get a segment of contiguous memory from the OS, we'll throw an OOM exception.

Another scenario: suppose that GC runs and we free all of the objects within a particular segment. We will then release that segment back to the OS. Now let's say that 5 minutes later, we need to grow the heap again. We go back to the OS to get a new segment, but some other process has placed a 1K allocation right at the beginning of that segment that we released. What happens? Well, if that 1k allocation fragments your memory and we're unable to find contiguous space . . . you guessed it. An OOM occurs.

So how do you avoid seeing this kind of thing? The first thing you can do is try and avoid a high memory situation. Almost all of the OOM cases we see involve high memory as well. That's because as you approach 80% of available memory for the process (and that's 2GB on a 32-bit machine without the 3GB switch enabled), the likelihood of seeing an OOM begins to increase exponentially. The reason for this (based on the information above) is pretty obvious. If you have reached the 80% mark, the chances of us finding a large, contiguous chunk of memory go down pretty substantially due to fragmentation. How do you avoid a high memory situation? There are a lot of answers to that question, but a few are:

Don't create a large number of objects on a per-request basis
Don't store large objects in Session
Don't cache user-specific data
Call Dispose() on objects that provide a Dispose() method
Call Marshal.ReleaseComObject in a loop on your COM objects (FinalReleaseComObject without the loop in 2.0)

I don't mean for that to be an exhaustive list, but you get the drift.

Another thing that you want to do to avoid OOM situations is to avoid fragmentation. If system memory is fragmented, it becomes more difficult for the CLR to find contiguous memory to grow the managed heap. There are also a lot of causes of memory fragmentation, but I'd have to say that one of the most common we see is caused by dynamic assemblies. (See my blog entry on the Debug attribute.) Other causes of dynamic assemblies are:

XML serialization (We have an article on this)
Script blocks in XSL transforms (We have an article on this, too)
Heavy use of regular expressions
Creating your own dynamic assemblies en masse

There are a lot of other reasons for fragmentation as well, but these are among the most common.

I hope this helps to explain why OOM conditions occur and how you can avoid them.

Getting the PID and TID of a COM Call

jamesche — Fri, 30 Dec 2005 22:40:00 GMT

There are times when I may be debugging a latency issue that involves a COM component running in a dllhost process. Depending on where we are in that process, I may need to debug the dllhost process in addition to the aspnet_wp or w3wp process. So how do you know which dllhost process to debug, and once you do know the process, how do you know which thread your COM call is executing on? Getting that information isn't hard, but it's not something that you'll be able to figure out on your own.

Finding the Thread Making the COM Call

The first thing you need to do is locate the thread in the aspnet_wp or w3wp process that is executing the COM call. What you want to do is look for a native call to rpcrt4!I_RpcSendReceive. To do that, use the "~*kp" command in the debugger. This will dump out the stack for all threads.

You can see the thread that you should target in the output below.

26 Id: 560.a34 Suspend: 0 Teb: 7ff6c000 Unfrozen
ChildEBP RetAddr
01bffb68 77d5f2a1 NTDLL!NtRequestWaitReplyPort+0xb
01bffb94 77d5d675 rpcrt4!LRPC_CCALL::SendReceive+0x11e
01bffba0 7cf05eb8 rpcrt4!I_RpcSendReceive+0x2c
01bffbc0 7cf05d4a OLE32!any_handle <PERF>
01bffbd8 7cf05c57 OLE32!any_handle <PERF>
01bffc18 7cf05eae OLE32!any_handle <PERF>
01bffc88 7ce35b67 OLE32!any_handle <PERF>
01bffce0 77d9a063 OLE32!CPagedVector::GetTableWithSect+0x1f7
01bffcfc 77d9a011 rpcrt4!NdrProxySendReceive+0x4c
01bfff44 77d99db8 rpcrt4!NdrClientCall2+0x4f5
01bfff60 77d4183f rpcrt4!ObjectStublessClient+0x76
01bfff70 787f5818 rpcrt4!ObjectStubless+0xf
01bfffb4 7c57b388 comsvcs!CEventDispatcher::GetEventServerInfoThread(void * pData = 0xffffffff)+0x118
01bfffec 00000000 KERNEL32!BaseThreadStart+0x52

I've bolded the call that you want to look for in the stack above. Once you locate that stack, switch to that thread. In this case, it's thread 26, so I'll issue the "~26s" command to the debugger to switch to that thread.

Digging Into OLE32 Calls

The next step is not so easy to do because you won't have access to private symbols for OLE32.dll. That means that you don't have type information. (It also explains why the above stack shows several calls to OLE32!any_handle.) Before we go any further, let's first dump out the stack above with arguments using the kv command. I'll actually use the kvL command so that source paths are not included so that the output will be a bit cleaner.

0:026> kvL
ChildEBP RetAddr Args to Child
01bffb68 77d5f2a1 00000858 001297a8 001297a8 NTDLL!NtRequestWaitReplyPort+0xb (FPO: [3,0,0])
01bffb94 77d5d675 0012b098 01bffbc0 7cf05eb8 rpcrt4!LRPC_CCALL::SendReceive+0x11e (FPO: [Non-Fpo])
01bffba0 7cf05eb8 0012b098 0012923c 0012923c rpcrt4!I_RpcSendReceive+0x2c (FPO: [Non-Fpo])
01bffbc0 7cf05d4a 00000000 0012923c 00000000 OLE32!any_handle <PERF> (OLE32+0xe5eb8)
01bffbd8 7cf05c57 01bffc14 0012923c 0007eee8 OLE32!any_handle <PERF> (OLE32+0xe5d4a)
01bffc18 7cf05eae 0012923c 01bffd4c 01bffd08 OLE32!any_handle <PERF> (OLE32+0xe5c57)
01bffc88 7ce35b67 0012923c 01bffd4c 01bffd08 OLE32!any_handle <PERF> (OLE32+0xe5eae)
01bffce0 77d9a063 0012923c 01bffd4c 01bffd08 OLE32!CPagedVector::GetTableWithSect+0x1f7 (FPO: [Non-Fpo])
01bffcfc 77d9a011 00120d14 01bffd98 05030117 rpcrt4!NdrProxySendReceive+0x4c (FPO: [Non-Fpo])
01bfff44 77d99db8 7874c538 7874c5aa 01bfff78 rpcrt4!NdrClientCall2+0x4f5 (FPO: [Non-Fpo])
01bfff60 77d4183f 00000018 00000003 7886cf0c rpcrt4!ObjectStublessClient+0x76 (FPO: [Non-Fpo])
01bfff70 787f5818 00120d14 00000000 01bfffa8 rpcrt4!ObjectStubless+0xf
01bfffb4 7c57b388 7886cec8 0197f1fc 00000000 comsvcs!CEventDispatcher::GetEventServerInfoThread+0x118 (FPO: [Uses EBP] [1,7,4]) (CONV: stdcall)
01bfffec 00000000 787f5700 7886cec8 00000000 KERNEL32!BaseThreadStart+0x52 (FPO: [Non-Fpo])

Now let's focus on the calls into OLE32. All of the function names you see into OLE32 are wrong because your symbols don't line up. However, don't worry. You still have what you need. The call that you're interested in is the function at offset 0xe5c57. You want to dump out the memory at the address of that function call's first argument, in this case 0012923c. Both this function and the memory address of the first argument are bolded in the output above. To dump that memory, dd the address of the argument as follows:

0:026> dd 0012923c
0012923c 7ce681b0 7ce55750 00000003 0000000a
0012924c 00000000 00000000 0007eee8 0007f7c0
0012925c 0012dd90 0012bda0 7ce6a938 00060005
0012926c 00000000 ffffffff 00000000 7ce681b0
0012927c 7ce55750 00000001 00000124 00000000
0012928c 00000000 0007ee08 00000000 00000000
0012929c 00133100 7ce6a938 00060005 ffffffff
001292ac ffffffff ffffffff 0013a748 f1eef1ee

Now you want to focus on the 7th dword. (I've bolded it in the output above.) That is the address of an internal class in OLE32 that holds the information that you need. Do a dd on that address as follows.

0:026> dd 0007eee8
0007eee8 7cf117a0 0007ee78 00000a9c 000006e8
0007eef8 3a76f9c2 6ac6c553 7ecc4b7f 6e0ad0df
0007ef08 0000c009 ffff0a9c b72cdaa4 0647c9ba
0007ef18 00000082 00000000 00000000 000e2e00
0007ef28 00000000 001273f0 00000006 ffffffff
0007ef38 0007ecf0 001266b4 00000003 00000000
0007ef48 00000000 00000001 00000000 00060005
0007ef58 00000000 f1eef1ee 00000000 00000000

The values you are interested in here are the 3rd and 4th dwords. (I've bolded them.) The 3rd dword is the PID of the dllhost process that this COM call is executing in. In this case, it's hex a9c. You can easily convert that to decimal in the debugger using the ? command as follows.

0:026> ? a9c
Evaluate expression: 2716 = 00000a9c

So now you know that you're looking for the dllhost process with a PID of 2716. The thread that your COM call is executing on is the 4th dword of the output above, in this case 6e8. To find that thread, open the dump of the dllhost process ID 2716 and issue a "~" command. You'll see the thread ID as follows.

0:000> ~
. 0 Id: a9c.464 Suspend: 0 Teb: 7ffde000 Unfrozen
1 Id: a9c.c94 Suspend: 0 Teb: 7ffdd000 Unfrozen
2 Id: a9c.4bc Suspend: 0 Teb: 7ffdc000 Unfrozen
3 Id: a9c.b10 Suspend: 0 Teb: 7ffd9000 Unfrozen
4 Id: a9c.c30 Suspend: 0 Teb: 7ffd8000 Unfrozen
5 Id: a9c.c24 Suspend: 0 Teb: 7ffd7000 Unfrozen
6 Id: a9c.65c Suspend: 0 Teb: 7ffd6000 Unfrozen
7 Id: a9c.398 Suspend: 0 Teb: 7ffd4000 Unfrozen
8 Id: a9c.3dc Suspend: 0 Teb: 7ffaf000 Unfrozen
9 Id: a9c.c2c Suspend: 0 Teb: 7ffd5000 Unfrozen
10 Id: a9c.b68 Suspend: 0 Teb: 7ffae000 Unfrozen
11 Id: a9c.2c0 Suspend: 0 Teb: 7ffad000 Unfrozen
12 Id: a9c.b74 Suspend: 0 Teb: 7ffac000 Unfrozen
13 Id: a9c.6e8 Suspend: 0 Teb: 7ffab000 Unfrozen

Your COM call is executing on thread 13. You now have the information that you need to dig into that thread and see what's going on.

Digging Into Objects

jamesche — Fri, 23 Dec 2005 00:30:00 GMT

One of our engineers asked me yesterday about getting the total size for objects in memory. The debugger is adept at doing such things, but you have to understand how object sizes are interpreted.

Let me start by giving you some background on how I'll typically deal with troubleshooting a high memory situation. The first thing of interest to me is whether we're dealing with a managed leak (.NET code) or a native leak. To discover that, I'll use the !eeheap debugger command.

0:000> !eeheap -gc
Loaded Son of Strike data table version 5 from "C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\mscorsvr.dll"
Number of GC Heaps: 4
Large object heap starts at 0x232d0030
segment begin allocated size
0x232d0000 0x232d0030 0x23bdac68 0x0090ac38(9,481,272)
Heap Size 0x16980074(379,060,340)
------------------------------
GC Heap Size 0x575ef130(1,465,839,920)

I've edited out much of the output here for the sake of brevity, but you can clearly see that this output illustrates an enormous amount of memory being used by the managed heaps. (The -gc switch shows us what's in the GC heaps or managed heaps.) In this case, we're looking at over 1GB of memory in use. Based on that, I know that I need to dig into managed objects in order to locate the source of the problem.

In this post, I'm not going to dig into the particular dump I've shown above because it's an actual customer dump that I'm working on. Instead, I will show a simplified example based off of a simple example application. Don't worry. The concepts are actually more clearly illustrated from the example dump I will use.

What's using my memory?

Once you know that your memory is being sucked up by managed objects, you'll need to dig into the managed heaps to find out where your memory is being used. The first step in doing that is to dump out the heap statistics using !dumpheap -stat.

0:013> !dumpheap -stat
Loaded Son of Strike data table version 5 from "C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\mscorwks.dll"
total 108,014 objects
Statistics:
MT         Count TotalSize Class Name
0x79c025fc 1      12         System.UnloadWorker
0x79bfc98c 1      12         System.Runtime.Remoting.Activation.ActivationListener
0x79bfc7c4 1      12         System.Runtime.Remoting.Activation.LocalActivator
0x79bfc390 1      12         System.Reflection.DefaultMemberAttribute
0x79bf48ac 1      12         System.Runtime.Remoting.Messaging.IllogicalCallContext
0x033c7c24 10     200        Staticmethod.Sclass
0x00be209c 13,563 714,108    System.Object[]
0x79b925c8 21,893 2,659,728 System.String
0x00be2c3c 78     10,705,064 System.Byte[]
Total 108,014 objects, Total size: 16,259,364

This output shows you all of the objects in the managed heap ordered by the amount of memory used by that type. The first column you see is the address of the method table (MT) followed by the count of objects, the total size of all of the objects (which is what we'll dig into soon), and the class name. Naturally I have cut down the size of the output dramatically for this example, but what we need is displayed above.

In this example, it's clear that the majority of memory is taken up by System.Byte[], about 10MB. I'm interested in seeing where those byte arrays are coming from and the debugger will let you do that pretty easily using the !gcroot command. However, !gcroot works on the address of an object, and right now we only have the address of a method table. To find out the address of the objects within the method table, use !dumpheap with the -mt switch and pass it the method table address that you got from the !dumpheap -stat command.

0:013> !dumpheap -mt 0x00be2c3c
Address    MT         Size       Gen
0x00f02c54 0x00be2c3c 172        2       System.Byte[]
0x00f02efc 0x00be2c3c 12         2       System.Byte[]
0x00f062e8 0x00be2c3c 28         2       System.Byte[]
0x00f06438 0x00be2c3c 76         2       System.Byte[]
0x00f094a8 0x00be2c3c 172        2       System.Byte[]
0x00f09628 0x00be2c3c 172        2       System.Byte[]

. . . more output here . . .

0x01f0c020 0x00be2c3c 1,048,592 3       System.Byte[]
0x0200c030 0x00be2c3c 1,048,592 3       System.Byte[]
0x0210c050 0x00be2c3c 1,048,592 3       System.Byte[]
0x0220c070 0x00be2c3c 1,048,592 3       System.Byte[]
0x0230c090 0x00be2c3c 1,048,592 3       System.Byte[]
0x0240c0a0 0x00be2c3c 1,048,592 3       System.Byte[]
0x0250c0c0 0x00be2c3c 1,048,592 3       System.Byte[]
0x0260c0e0 0x00be2c3c 1,048,592 3       System.Byte[]
0x0270c100 0x00be2c3c 1,048,592 3       System.Byte[]
0x0280c120 0x00be2c3c 1,048,592 3       System.Byte[]
total 78 objects

You can clearly see a pattern here. Most of the memory from byte arrays is taken up by byte arrays that are almost exactly 1MB in size. (Notice also that this ouput shows you what generation an object is in. The large byte arrays are in the large object heap which is denoted here by gen 3.) In fact, many times when you have a memory issue, you will see patterns like this, but they are nowhere near as obvious as this. :)

Now that we have a better idea as to which byte arrays are the culprits, it's time to see what created them by using !gcroot on the address as I discussed above. We'll pick the first large one.

0:013> !gcroot 0x01f0c020
Scan Thread 1 (0x1720)
Scan Thread 5 (0x13f0)
Scan Thread 8 (0xff8)
Scan HandleTable 0x156aa0
Scan HandleTable 0x15a4b8
Scan HandleTable 0x235c68
HANDLE(Strong):44f1148:Root:0xf56500(System.Web.NativeFileChangeNotification)->
0xf564e4(System.Web.DirMonCompletion)->0xf56214(System.Web.DirectoryMonitor)->
0xf56318(System.Web.FileMonitor)->0xf56348(System.Collections.Hashtable)->
0xf5637c(System.Collections.Hashtable/bucket[])->0xf5467c(System.Web.FileChangesMonitor)->
0xf55da4(System.Web.FileChangeEventHandler)->0xf532d0(System.Web.HttpRuntime)->
0xf5496c(System.Web.Caching.CacheSingle)->0xf54a1c(System.Web.Caching.CacheExpires)->
0xf54c68(System.Object[])->0xf55728(System.Web.Caching.ExpiresBucket)->0x13700e8(System.Web.Caching.ExpiresEntry[])->
0x1370098(System.Web.Caching.CacheEntry)->0x1370068(System.Web.SessionState.InProcSessionState)->
0xfbfd38(System.Web.SessionState.SessionDictionary)->0xfbff80(System.Collections.Hashtable)->
0xfc924c(System.Collections.Hashtable/bucket[])->
0xfc0bdc(System.Collections.Specialized.NameObjectCollectionBase/NameObjectEntry)->
0xfc0b9c(Staticmethod.Sclass)->0x1f0c020(System.Byte[])

The !gcroot command walks the object tree and tells you where an object is rooted. (A discussion of this is outside of the scope of what I'm discussing here.) You can see that we scanned the threads as well as handletables. Each app domain had a handletable where objects can also be rooted. In this case, our byte array is rooted in a handletable and the output shows some interesting things worth discussing.

Let's look at this output starting at the CacheEntry object. You can see this byte array rooted in a CacheEntry object, then in InProcSessionState, and eventually, in a Staticmethod.Sclass object. What this tells us is that the Staticmethod.Sclass object is being stored in InProc Session state and that it is rooting the byte array. (In ASP.NET InProc Session state, objects are stored in cache which is why you see this rooted in a CacheEntry.)

Because of this ouput, I'm interested in having a look into the Staticmethod.Sclass object. If you refer back to the output of !dumpheap -stat, you'll see that there are 10 of these objects for a total of 200 bytes:

0x033c7c24 10 200 Staticmethod.Sclass

Let's use !dumpheap -mt to dump out these individual objects just as we did before.

0:013> !dumpheap -mt 0x033c7c24
Address    MT         Size Gen
0x00f0f164 0x033c7c24 20   2 Staticmethod.Sclass
0x00fc0b9c 0x033c7c24 20   2 Staticmethod.Sclass
0x00fc0c08 0x033c7c24 20   2 Staticmethod.Sclass
0x00fc0c48 0x033c7c24 20   2 Staticmethod.Sclass
0x00fc4230 0x033c7c24 20   2 Staticmethod.Sclass
0x00fc427c 0x033c7c24 20   1 Staticmethod.Sclass
0x00fc42bc 0x033c7c24 20   1 Staticmethod.Sclass
0x00fc42fc 0x033c7c24 20   1 Staticmethod.Sclass
0x00fc433c 0x033c7c24 20   1 Staticmethod.Sclass
0x00fc93a8 0x033c7c24 20   1 Staticmethod.Sclass
total 10 objects

Sure enough, there are 10 of these and each is 20 bytes for a total of 200 bytes. However, if you were to use the !objsize command to get the size of one of these objects, you might be surprised to see how large it actually is.

0:013> !objsize 0x00f0f164
sizeof(0xf0f164) = 1,048,640 (0x100040) bytes (Staticmethod.Sclass)

What the heck? Which one of these commands is giving me wrong information? Actually, neither. The output that you see from !dumpheap -mt shows you the size of ONLY the Staticmethod.Sclass object. It doesn't include the size of members of that class because those members are listed elsewhere in the output. Let's dig into this object size output a bit more.

You can get a good idea of what makes up the 1MB or so used by this object by simply using the -v switch for !objsize like so.

0:013> !objsize 0x00f0f164 -v
sizeof(0xf0f164) = 1,048,640 (0x100040) bytes (Staticmethod.Sclass)
sizeof(0xfc0bb0) = 32 ( 0x20) bytes (System.String)
sizeof(0x230c090) = 1,048,588 (0x10000c) bytes (System.Byte[])

Now things make a bit more sense. We can see from this ouput that the Staticmethod.Sclass object has a couple of member variables; a byte array that is 1,048,588 bytes and a string that's 32 bytes. Add the original 20 bytes to those two and you end up with 1,048,640, the value that !objsize returned. Pretty cool!

Note: It's worth noting that in normal circumstances, you would want to use a command like db to dump out these byte arrays and see what's in them. In this case, we're only interested in discussing the details of getting the size of objects, so I won't go into that in this post. Besides, I happen to know that there's nothing in these byte arrays. :)

Now for another cool trick. Suppose that you wanted to dump out the !objsize output for all of these. In this example, that's not too hard. After all, there are only 10 of them. However, suppose there were 3,000 of them. That would be quite an undertaking if you had to do it manually, but you don't have to. You can use the .foreach token in the debugger to do it for you! Check this out.

0:013> .foreach (obj {!dumpheap -mt 0x033c7c24 -short}) {!objsize obj}
sizeof(0xf0f164) = 1,048,640 (0x100040) bytes (Staticmethod.Sclass)
sizeof(0xfc0b9c) = 1,048,640 (0x100040) bytes (Staticmethod.Sclass)
sizeof(0xfc0c08) = 1,048,640 (0x100040) bytes (Staticmethod.Sclass)
sizeof(0xfc0c48) = 1,048,640 (0x100040) bytes (Staticmethod.Sclass)
sizeof(0xfc4230) = 1,048,640 (0x100040) bytes (Staticmethod.Sclass)
sizeof(0xfc427c) = 1,048,640 (0x100040) bytes (Staticmethod.Sclass)
sizeof(0xfc42bc) = 1,048,640 (0x100040) bytes (Staticmethod.Sclass)
sizeof(0xfc42fc) = 1,048,640 (0x100040) bytes (Staticmethod.Sclass)
sizeof(0xfc433c) = 1,048,640 (0x100040) bytes (Staticmethod.Sclass)
sizeof(0xfc93a8) = 1,048,640 (0x100040) bytes (Staticmethod.Sclass)

The format of this command is pretty simple. The "obj" is simply a variable that is populated by the output from the command in curly brackets immediately after it. In this case, I'm running !dumpheap -mt on the method table of the Staticmethod.Sclass objects. The -short switch causes the command to ONLY output the address of the object and no other details. Here's what that command outputs.

0:013> !dumpheap -mt 0x033c7c24 -short
0xf0f164
0xfc0b9c
0xfc0c08
0xfc0c48
0xfc4230
0xfc427c
0xfc42bc
0xfc42fc
0xfc433c
0xfc93a8

As you can see, this is the perfect kind of output if what we really want to do is loop through each one of these addresses and run a command on each. That's the purpose of the -short switch. I use it often, and now you see why.

The second set of curly brackets contains a second command to run on each "obj" variable, in this case, the !objsize command. The result is the size output of each object as you can see in the listing above.

Back to our theoretical example of 3,000 objects. You now know how to use the .foreach token to get the size of each, but you are still left with using a calculator to add them all up. Once again, the debugger comes to your rescue. You can use a feature of the debugger called "pseudo-registers" to store the cumulative size of objects and then output the total value once they've all been calculated. This allows you to actually add up the total values as you dump them. The command to do that is a bit trickier. Here it is.

0:013> r $t2 = 0; .foreach ( obj {!dumpheap -short -mt 0x033c7c24} ) { r $t2 = $t2 + 1; .foreach( theSize { !objsize -short obj } ) { r $t1 = $t1 + theSize; ? theSize } }; .echo Total Size:; ?$t1
Evaluate expression: 1048640 = 00100040
Evaluate expression: 1048640 = 00100040
Evaluate expression: 1048640 = 00100040
Evaluate expression: 1048640 = 00100040
Evaluate expression: 1048640 = 00100040
Evaluate expression: 1048640 = 00100040
Evaluate expression: 1048640 = 00100040
Evaluate expression: 1048640 = 00100040
Evaluate expression: 1048640 = 00100040
Evaluate expression: 1048640 = 00100040
Total Size:
Evaluate expression: 10486400 = 00a00280

This command uses the pseudo-registers I mentioned above to keep track of the objects. There are 20 pseudo-registers, $t0 - $t19. This command begins by assigning the value of 0 to the $t2 pseudo-register. (r $t2 = 0) After we assign that value, we loop through the objects as we did before. However, in this case, we keep track of the total size using a new variable (theSize). We increment the $t1 pseudo-register with each iteration. Finally, we print out the total size using the .echo command to print the value of the $t1 pseudo-register.

I realize that this may be getting a little confusing. Fortunately, the debugger comes with excellent documentation. I urge you to have a look. Pseudo-registers are a powerful feature of the debugger. The only drawback is that it may become tedious to remember these commands. In a future post, I'll show you how to set up aliases so that you can type a simple command and execute complex instructions that you've written. It's cool, I promise.

One more important thing to keep in mind. Getting the size of a complex object can take a long time, so you'll want to carefully decide whether or not you have the time for a command of this level of complexity to complete. I worked a case once where a customer had millions of a particular object. After running the debugger on it for over a day, I did some math and determined that it was going to take 36 years for the command to complete! The moral? Do your math first!

Happy debugging, and Merry Christmas!

Jim

Are you in GC?

jamesche — Tue, 20 Dec 2005 20:49:00 GMT

When you're debugging, you're really just looking into internal memory structures in a process. When it comes to debugging an ASP.NET application, you are oftentimes dumping out method tables, objects, etc. The ability to do so requires a known address. That's no problem most of the time. However, if you're in a GC when a dump is taken, we may be in the process of moving some of those structures around for compaction. In that case, you may see what looks like managed heap corruption when in fact you're seeing the result of the compaction phase of GC. For that reason, it's often convenient to know if you're in the process of GC when you crack open a dump. To do so, you can check GcHeap::GcInProgress.

First thing you'll want to do is make sure that you have a good symbol path. You can check your symbol path by running the .sympath command in Windbg as so:

0:000> .sympath
Symbol search path is: srv*c:\\symbols*http://msdl.microsoft.com/download/symbols

You can set your symbol path inWindbg using the same command like so:

.sympath srv*c:\\symbols*http://msdl.microsoft.com/download/symbols

What this command does is set up a local symbol server in the c:\symbols directory. When Windbg needs symbols for a particular module, it checks the c:\symbols directory for those symbols. If they're there, great. If not, it downloads the symbols from the Microsoft public symbol server (http://msdl.microsoft.com/download/symbols) and stores them in the c:\symbols folder. Note that it can take a bit of time for symbols to download.

Now you'll need to know whether you are running workstation or server GC. In most cases, this is determined by whether you are running a multi-proc box. (Hyperthreaded boxes do count as multi-proc.) A multi-proc box will usually be running server GC (mscorsvr.dll) and a single-proc box will usually be running workstation GC (mscorwks.dll). You can double-check to see which you have loaded using the lmv command in Windbg.

0:000> lmv mmscorsvr
start end module name
791b0000 79419000 mscorsvr (deferred)

Image path: C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\mscorsvr.dll
Image name: mscorsvr.dll
Timestamp: Fri Feb 18 14:58:55 2005 (4216570F)
CheckSum: 0026A38C
ImageSize: 00269000
File version: 1.1.4322.2300
Product version: 1.1.4322.2300
File flags: 20 (Mask 3F) Special
File OS: 4 Unknown Win32
File type: 2.0 Dll
File date: 00000000.00000000
Translations: 0409.04e4
CompanyName: Microsoft Corporation
ProductName: Microsoft .NET Framework
InternalName: MSCORSVR.DLL
OriginalFilename: mscorsvr.dll
ProductVersion: 1.1.4322.2300
FileVersion: 1.1.4322.2300
FileDescription: Microsoft .NET Runtime Common Language Runtime - Server
LegalCopyright: Copyright © Microsoft Corporation 1998-2002. All rights reserved.
LegalTrademarks: Microsoft® is a registered trademark of Microsoft Corporation. Windows(TM) is a trademark of Microsoft Corporation
Comments: Microsoft .NET Runtime Common Language Runtime - Server

Notice that the command is lmv m<module_name>. In other words, don't forget the m in front of the module name. (You can also put the 'm' immediately after the 'v' followed by a space and the module name. My muscle memory simply types it the other way.) In the case above, it's clear that I'm running server GC (mscorsvr.dll). Now you're ready to check for GC.

To check for the status of GC, you will use the dd command in the debugger. The dd command allows you to display double-word values. The format for the dd command is dd <address> and in this case, you want to dump out the address of mscorsvr!GCHeap::GcInProgress.

0:000> dd mscorsvr!GCHeap::GcInProgress
793e3218 00000000 00000004 00000028 00000002
793e3228 00000003 00000002 0000000f 00000001
793e3238 00000000 00000001 00000001 00000002
793e3248 7c821ad8 01a31e90 ffffffff 00f0ba0f
793e3258 792f2e08 00000000 00000000 792543ac
793e3268 00000000 00000000 7921d1fd 00000000
793e3278 00000000 792543c4 00000000 00000000
793e3288 791b4151 00000000 00000000 792f3263

Note that dd dumps out 32 DWORDs by default. You are actually interested in the first DWORD here, so you can clean up the output a bit by using a different command:

0:000> dd mscorsvr!GCHeap::GcInProgress L1
793e3218 00000000

Now we have the one DWORD we're interested in and we can clearly see that the value is 0. Therefore, we know that we are not currently in GC.

One note: If you were using workstation GC, you will need to replace mscorsvr with mscorwks in these commands.

Have fun!

Jim

Clarifications on Debugging

jamesche — Tue, 20 Dec 2005 18:46:00 GMT

When I'm talking to ASP.NET developers who are experiencing memory issues, crashes, hangs, etc., I will often say, "Hook up the debugger and see what's going on." More often than not, the response that I get from that recommendation is one of confusion. Most developers think of "the debugger" as Visual Studio or some other source code debugger. In fact, when I say "the debugger", I'm talking about Windbg, a debugger included in the Debugging Tools for Windows.

Windbg allows you to do most of the same debugging tasks that you can perform in Visual Studio, but it has additional capabilities. Those additional capabilities come at the price of being a bit harder to learn how to use. But hey, if you are an ASP.NET developer, you're obviously smart enough to learn new tricks. Learning to use Windbg is definitely a trick worth learning.

Suppose you have an ASP.NET application that consumes large amounts of memory over time. If you want to find out what's taking up so much memory, you can profile the application. However, there are some serious drawbacks to taking that approach. Profiling is expensive from a performance standpoint. In fact, in a production environment, running a profiler is most often not an option. What's a developer to do? I have an answer! You can use the Debugging Tools for Windows to create a user-mode dump file which can then be analyzed with Windbg to determine where you are consuming memory.

First thing you'll need to do is create a dump file. The easiest way to do that is to use ADPlus. Here's an article that will walk you through that process:

286350 How to use ADPlus to troubleshoot "hangs" and "crashes"
http://support.microsoft.com/default.aspx?scid=kb;EN-US;286350

After you've got a dump file, you'll need to set up Windbg to debug it. I wrote an article on this about a year ago. While I certainly wouldn't claim that this article is the best resource available on the subject, it's one that I have handy so it's what you're going to get. :)

892277 Troubleshooting ASP.NET using WinDbg and the SOS extension
http://support.microsoft.com/default.aspx?scid=kb;EN-US;892277

Note that this is all 1.x specific.

The article above is a crash dump analysis, and a very simple one at that. If you want details on how to debug memory issues, your best bet is to review our PAG documentation. It's excellent information for debuggers.

http://msdn2.microsoft.com/en-us/library/ms954594.aspx

From here on out, I'll try to provide you with some cool debugging tricks that come straight from the work that I do everyday. Hopefully it will be useful to you in your debugging endeavors!

Jim