Speaking of which... : Debugging School

Make sure you have an Internet connection when using signed assemblies

JohanS — Thu, 12 Nov 2009 11:08:22 GMT

A customer called in the other day and told me that his web application took a long time to start. My initial thought was (off course) that it was a matter of the classic slow-to start web services I’ve written about earlier.

(What to do about the slow startup of web services)

As I learned more about his problem I found that this was more than just a simple matter of pre-compilation and keeping your w3wp.exe alive.

The Cause

The customer was using authenticode signed assemblies in his web application. When that application calls the assembly for the first time it will want to go on-line and check that the certificate is still valid. So, what if this is an internal server with no Internet access? Well, then we have a problem. The process will spend quite some time trying to check the certificate revocation lists. Eventually it will give up, but for my customer this meant that it hung for at least 20 seconds upon startup.

The Resolution

To resolve this I’d chose one of the following three options:

Grant Internet access to the server

This is will obviously resolve the problem, but may not always be possible.

Review the need for Authenticode signing

If you own the assemblies then you might want to consider not using Authenticode signing at all. A Strong Named Assembly might be sufficient. For more information on Strong Named Assemblies and how to create them, please see the following article on MSDN: How to: Sign an Assembly with a Strong Name

Disable Signature Verification

If all else fails you can disable Signature Verification by adding the following to your machine.config or application.exe.config:

Identifying the problem using windbg

Okay, so if we suspect that this is the problem, how can we verify this?

I got a memory dump from my customer, so I’ll use that to demonstrate.

First of all you want to make sure the dump is taken as the application pool hangs upon the first request. Getting a dump right after the problem has occurred will usually do us no good at all. We need to get the dump as the problem is actually ocurring. Having made sure the dump is fine I open up the dump in windbg and load sos.dll. I then use the !aspxpages command to see the pending requests currently on the server. !aspxpages dumps all the HttpContexts found on the heap so we get a list of all currently executing requests, as well as recently finished ones that haven’t been garbage collected yet.

0:000> .load C:\Windows\Microsoft.NET\Framework\v2.0.50727\sos.dll
0:000> !aspxpages
Going to dump the HttpContexts found in the heap.
Loading the heap objects into our cache.
HttpContext    Timeout Completed     Running ThreadId ReturnCode   Verb RequestPath+QueryString
0x01b41ad4        0            no        11 Sec     XXX        200       /App/
0x01b75f48      110 Sec        no        11 Sec     XXX        401   GET /App/Default.aspx
0x01b7eb40    19200 Sec        no        11 Sec      19        200   GET /App/Default.aspx
Total 3 HttpContext objects

So, the request on thread 19 is the one we want. As we can see in the ThreadId column it is the only active request. Let’s jump to that thread and investigate what it is doing.

0:000> ~19s
eax=01510000 ebx=00000000 ecx=03e2d348 edx=776d9a94 esi=03e2d07c edi=00000000
eip=776d9a94 esp=03e2d034 ebp=03e2d0a4 iopl=0         nv up ei ng nz ac pe cy
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000             efl=00000297
ntdll!KiFastSystemCallRet:
776d9a94 c3              ret
0:019> !clrstack
OS Thread Id: 0x11cc (19)
ESP       EIP
03e2eb8c 776d9a94 [PrestubMethodFrame: 03e2eb8c] App.BasePage.Page_Load(System.Object, System.EventArgs)
03e2eba0 6aada7ff System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr, System.Object, System.Object, System.EventArgs)
03e2ebb0 6c392594 System.Web.Util.CalliEventHandlerDelegateProxy.Callback(System.Object, System.EventArgs)
03e2ebc4 6c38ba84 System.Web.UI.Control.OnLoad(System.EventArgs)
03e2ebd8 6c38bac3 System.Web.UI.Control.LoadRecursive()
03e2ebf0 6c387b74 System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)
03e2ed48 6c3877a4 System.Web.UI.Page.ProcessRequest(Boolean, Boolean)
03e2ed80 6c3876d1 System.Web.UI.Page.ProcessRequest()
03e2edb8 6c387666 System.Web.UI.Page.ProcessRequestWithNoAssert(System.Web.HttpContext)
03e2edc4 6c387642 System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)
03e2edd8 012b02a6 ASP.text.ProcessRequest(System.Web.HttpContext)
03e2ede8 6c38db16 System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
03e2ee1c 6c36132c System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)
03e2ee5c 6c95531f System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)
03e2ee60 6c94b704 [InlinedCallFrame: 03e2ee60]
03e2ef00 6c93613d System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)
03e2ef70 6ca0a7a2 System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
03e2ef74 6ca0a58f [InlinedCallFrame: 03e2ef74]
03e2f4c8 01092314 [NDirectMethodFrameStandalone: 03e2f4c8] System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr, System.Web.RequestNotificationStatus ByRef)
03e2f4d8 6ca0a839 System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
03e2f55c 6ca0a58f System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
03e2f65c 01092314 [ContextTransitionFrame: 03e2f65c]

So we’re in the Page_Load event of the page in question, but apart from that !clrstack doesn’t tell us much. Let’s take a look at the native callstack instead using the kb-command.

0:019> kb20
ChildEBP RetAddr Args to Child
03e2d030 776d9254 76adc244 000006d8 00000000 ntdll!KiFastSystemCallRet
03e2d034 76adc244 000006d8 00000000 03e2d07c ntdll!ZwWaitForSingleObject+0xc
03e2d0a4 74256247 000006d8 00003a98 00000000 kernel32!WaitForSingleObjectEx+0xbe
03e2d0d8 74251f7c 0141f2e4 00000002 00202005 cryptnet!CryptRetrieveObjectByUrlWithTimeout+0x1a5
03e2d1c8 742527a1 0141f2e4 00000002 00202005 cryptnet!CryptRetrieveObjectByUrlW+0xcc
03e2d268 742591be 00000000 0141f2e4 00000002 cryptnet!RetrieveObjectByUrlValidForSubject+0xa0
03e2d2c8 74252556 00000000 00000000 00000001 cryptnet!RetrieveTimeValidObjectByUrl+0x15c
03e2d374 74253fc1 00000082 0141f2d8 00000000 cryptnet!CTVOAgent::GetTimeValidObjectByUrl+0x178
03e2d45c 74253d98 00000003 018aa590 00000002 cryptnet!CTVOAgent::GetTimeValidObject+0x4a8
03e2d48c 74253428 00000003 018aa590 018c8220 cryptnet!FreshestCrlFromCrlGetTimeValidObject+0x2d
03e2d4d0 74259283 00000003 018aa590 018c8220 cryptnet!CryptGetTimeValidObject+0x58
03e2d54c 74253b1c 00000003 018aa590 018aa590 cryptnet!GetTimeValidCrl+0x2cb
03e2d590 74253992 018aa590 018c8220 03e2d5c8 cryptnet!GetBaseCrl+0x34
03e2d63c 757f77d5 00000001 00000001 00000001 cryptnet!MicrosoftCertDllVerifyRevocation+0x163
03e2d6c8 757f7641 00000001 00000001 00000001 crypt32!I_CryptRemainingMilliseconds+0x2aa
03e2d74c 757f7a26 00000001 00000001 00000001 crypt32!CertVerifyRevocation+0xd4
03e2d7f8 757f7838 01899590 00000000 00000000 crypt32!CChainPathObject::CalculateRevocationStatus+0x2d0
03e2d83c 75806c44 01899590 018c3018 01870fc0 crypt32!CChainPathObject::CalculateAdditionalStatus+0x152
03e2d904 758069a5 01899590 018aa590 01870fc0 crypt32!CCertChainEngine::CreateChainContextFromPathGraph+0x23e
03e2d93c 7580be14 018aa590 018aff94 01870fc0 crypt32!CCertChainEngine::GetChainContext+0x46
03e2d974 749172ba 01421338 018aa590 018aff94 crypt32!CertGetCertificateChain+0x72
03e2d9dc 749170fc 40000001 00000000 03e2da10 wintrust!_WalkChain+0x1ae
03e2da18 7491363b 00000000 03e2dbdc 03e2db74 wintrust!WintrustCertificateTrust+0xb9
03e2db30 7491346c 00000000 03e2dbe4 00000000 wintrust!_VerifyTrust+0x253
03e2db54 64025b1b 00000000 03e2dbe4 03e2db74 wintrust!WinVerifyTrust+0x50
03e2dbf8 716b7f4d 014443f8 00000000 00000003 mscorsec!GetPublisher+0xe4
03e2dc50 7145ed3b d60caabc 0188c4a8 00603c48 mscorwks!PEFile::CheckSecurity+0xcb
03e2dc78 7145ec84 d60caa64 00000000 0188c4a8 mscorwks!PEAssembly::DoLoadSignatureChecks+0x3a
03e2dca0 7145f0ca 01879f00 00000000 0188c4a8 mscorwks!PEAssembly::PEAssembly+0x109
03e2df3c 7145f1c5 0188c4a8 00000000 00000000 mscorwks!PEAssembly::DoOpen+0x103
03e2dfd0 71459062 0188c4a8 00000000 00000000 mscorwks!PEAssembly::Open+0x79
03e2e134 71456ace 03e2e15c 00000001 00000000 mscorwks!AppDomain::BindAssemblySpec+0x247

Here we see that we’re currently trying to check the certificate revocation list. This is done on a separate thread so this thread is currently doing nothing but waiting. Our next step would be to find this thread, but first we can find out the name of the .dll that we’re trying to verify. Look at the first argument to GetPublisher.

0:019> du 014443f8
014443f8 "C:\Windows\Microsoft.NET\Framewo"
01444438 "rk\v2.0.50727\Temporary ASP.NET "
01444478 "Files\scstt\57ae16cd\2f8637d5\as"
014444b8 "sembly\dl3\078f3d05\00fba208_c03"
014444f8 "6ca01\Microsoft.Practices.Enterp"
01444538 "riseLibrary.Logging.DLL"

So it’s a Microsoft .dll that is part of the Enterprise Library. Good to know. Now let’s find the thread that is attempting to retrieve the revocation list by running ~*kb and scanning through the threads until we find this one:

0:021> kb2000
ChildEBP RetAddr Args to Child
0427d7f8 776d9254 75502283 0000070c 00000001 ntdll!KiFastSystemCallRet
0427d7fc 75502283 0000070c 00000001 0427d824 ntdll!ZwWaitForSingleObject+0xc
0427d83c 75501fc8 0000070c 00000714 00000001 mswsock!SockWaitForSingleObject+0x19f
0427d928 77801693 00000001 00000000 0427d9d0 mswsock!WSPSelect+0x38c
0427d9a8 72a733e5 00000001 00000000 0427d9d0 ws2_32!select+0x494
0427e1ec 72a730fd 00000714 0427e210 72a72e88 winhttp!ICSocket::Connect_Start+0x3b0
0427e1f8 72a72e88 01828250 72a737bc 00000000 winhttp!CFsm_SocketConnect::RunSM+0x42
0427e210 72a7382b 018b0c60 00000000 00000000 winhttp!CFsm::Run+0x20
0427e234 72a7467d 01828250 018ccf30 0427e258 winhttp!DoFsm+0x2a
0427e244 72a74646 0000ea60 00000005 00000020 winhttp!ICSocket::Connect+0x32
0427e258 72a742b5 0000ea60 00000005 0000ea60 winhttp!ICSocket::Connect+0x13
0427e2a4 72a74038 01434458 0427e2c8 72a72e88 winhttp!HTTP_REQUEST_HANDLE_OBJECT::OpenConnection_Fsm+0x471
0427e2b0 72a72e88 01434458 00000000 00000000 winhttp!CFsm_OpenConnection::RunSM+0x37
0427e2c8 72a7382b 018b0c60 00000000 00000000 winhttp!CFsm::Run+0x20
0427e2ec 72a74625 01434458 018ccf30 0427e324 winhttp!DoFsm+0x2a
0427e2fc 72a745bc 00000000 00000000 018b0c60 winhttp!HTTP_REQUEST_HANDLE_OBJECT::OpenConnection+0x2f
0427e324 72a74526 0141f638 0427e348 72a72e88 winhttp!HTTP_REQUEST_HANDLE_OBJECT::MakeConnection_Fsm+0x9b
0427e330 72a72e88 0141f638 00000000 00000000 winhttp!CFsm_MakeConnection::RunSM+0x37
0427e348 72a7382b 018b0c60 00000000 00000000 winhttp!CFsm::Run+0x20
0427e36c 72a743c1 0141f638 018b0c60 014343c0 winhttp!DoFsm+0x2a
0427e3ac 72a74339 014343c0 0427e3d0 72a72e88 winhttp!HTTP_REQUEST_HANDLE_OBJECT::SendRequest_Fsm+0x93
0427e3b8 72a72e88 014343c0 018ccf30 00000000 winhttp!CFsm_SendRequest::RunSM+0x37
0427e3d0 72a7382b 018b0c60 00000000 00000000 winhttp!CFsm::Run+0x20
0427e3f4 72a73db0 014343c0 018b0c60 01888820 winhttp!DoFsm+0x2a
0427e414 72a76f71 00000000 0427e438 72a72e88 winhttp!HTTP_REQUEST_HANDLE_OBJECT::HttpSendRequest_Start+0x2af
0427e420 72a72e88 01888820 00000001 00000000 winhttp!CFsm_HttpSendRequest::RunSM+0x4c
0427e438 72a73c3e 018b0c60 00000000 00000000 winhttp!CFsm::Run+0x20
0427e480 72a73b8b 01888820 018ccf30 00000000 winhttp!StartFsmChain+0xcf
0427e4c4 72a739d8 018ccf30 00000000 00000000 winhttp!HttpWrapSendRequest+0x18c
0427e544 742599b4 018ccf30 00000000 00000000 winhttp!WinHttpSendRequest+0x19b
0427f5e4 74259741 005f7bc0 018ccf30 01898e60 cryptnet!InetSendAuthenticatedRequestAndReceiveResponse+0x56f
0427f6ec 7425921d 005f7bc0 01898e60 00202005 cryptnet!InetSendReceiveUrlRequest+0x2c8
0427f724 742523e9 01898e60 00000002 00202005 cryptnet!CInetSynchronousRetriever::RetrieveObjectByUrl+0x5f
0427f75c 742521a8 01898e60 00000002 00202005 cryptnet!InetRetrieveEncodedObject+0x64
0427f7b8 74256350 01898e60 00000002 00202005 cryptnet!CObjectRetrievalManager::RetrieveObjectByUrl+0xbb
0427f810 76ad4911 00000000 0427f85c 776be4b6 cryptnet!CryptRetrieveObjectByUrlWithTimeoutThreadProc+0x67
0427f81c 776be4b6 01898db0 4e080acf 00000000 kernel32!BaseThreadInitThunk+0xe
0427f85c 776be489 742562f7 01898db0 ffffffff ntdll!__RtlUserThreadStart+0x23
0427f874 00000000 742562f7 01898db0 00000000 ntdll!_RtlUserThreadStart+0x1b
0:021> du 01898e60
01898e60 "http://crl.microsoft.com/pki/crl"
01898ea0 "/products/CodeSignPCA2.crl"

This is the thread. To finally verify that it is in fact attempting to download the Certificate Revocation List we can check the value of the first argument to cryptnet!CInetSynchronousRetriever, InetRetrieveEncodedObject and CObjectRetrievalManager::RetrieveObjectByUrl or the second argument to cryptnet!InetSendReceiveUrlRequest as I did above.

Conclusion

In this case the customer chose to disable Signature Verification completely rather than granting Internet access to the server.

If you found the windbg-portion of this post hard to understand I can recommend a closer look at my Debugging School (In the menu to the left). Begin with Getting started with windbg - Part I and you should be set to go in no time.

/ Johan

Investigating Locks

JohanS — Fri, 09 Oct 2009 16:49:56 GMT

Consider the following scenario:

You have an ASP.NET application which intermittently responds sluggishly. As the problem occurs memory usage is about average, as is CPU usage, but still certain pages respond slower and slower. The machine acts just as if it is under heavy load, but judging from the CPU it isn't. In fact CPU usage might even decrease rather than increase. When trying to access non .NET content such as plain text/html or images you usually find that response times are just as fast as ever.

When you find that your application has hung in a low CPU state it is usually because of one of the following reasons (in order of likelihood based on my experience) :

External resources responding slowly. E.g. a database query or AD lookup is taking a long time to complete and so the application is simply waiting for another server to respond.
One of the threads of the application has entered a critical section and now several other threads are waiting to enter the same critical section.
A classic deadlock. Thread A waits for thread B which waits for thread C which waits for thread A.

Quite often you'll see a mixture of items 1 and 2. A thread enters a critical section, calls a remote database, or similar, and all other threads attempting to enter the same critical section will have to nicely stay in line until the struggling database has responded. Still, it is quite possible that the critical section is waiting for a local event to finish, but then you'd usually see a higher CPU load on the machine. (Though not necessarily in the w3wp.exe process.)

What are Critical Sections?

In case you've forgotten or simply not worked that much with critical sections it might be necessary to refresh your memory on this.

A Critical Section is a segment of code that you do not want to be executed more than one at a time. In order to ensure this you use the lock keyword to ensure that a certain block of code is able to complete with no interruption from any other threads. To illustrate this I've created the following code sample:

public class FileIO
{
    private static object myLock = new object();

	public FileIO()
	{
	}

    public string performLockOperation()
    {
        string strResult = "The Lock Operation started on " + System.DateTime.Now.ToLongTimeString();
        lock (myLock)
        {
            // Let the thread sleep for 10 seconds
            // This way we simulate a File IO operation that takes some time to complete.
            System.Threading.Thread.Sleep(10000);
        }
        strResult += " and ended on " + System.DateTime.Now.ToLongTimeString();
        return strResult;
    }

    public string performNoLockOperation()
    {
        string strResult = "The No Lock Operation started on " + System.DateTime.Now.ToLongTimeString();
        System.Threading.Thread.Sleep(10000);
        strResult += " and ended on " + System.DateTime.Now.ToLongTimeString();
        return strResult;
    }

}

Let's take a quick look at the parts of this class that I've written, called FileIO:

myLock

The class has a static variable called myLock that I will be using to identify the lock I'm about to create. This is why the variable is static. Otherwise I'd be creating new locks for each instance of the FileIO class leaving the lock operation completely redundant. Now that I use a static variable the lock spans all instances of the FileIO class.

constructor

The constructor is empty. Move along. Nothing to see here.

performLockOperation

This is the most interesting part of the class. It is the function we'll be using to actually reproduce the potential problems. The function returns a string which will contain information about when the call to the function was made and when it completed. We store this data in a string called strResult which we will eventually be returning as the result of the function call.

Having declared this variable and put the initial time of the function call in it we then create our lock. This is simply done with the following syntax:

lock (object)
{
}

Any code within this block will first make sure that there is not already a lock taken on this object. If there is, then they will pause execution and wait for the lock to become available before continuing execution.

Inside our lock we then simulate a slow File IO operation by letting the thread sleep for 10 seconds.

After that we append the current time to the result and return it.

performNoLockOperation

This function is identical to the previous one, except this time we do not have a critical section. This function is included just for comparison.

The potential problem

Now, let's get down to business. If we have a web page that makes a call to the performLockOperation-function and that page is requested at the same time by four different clients. What would be the response times?

The answer is 10, 20, 30 & 40 seconds.

Here's why:

The thread handling the first request enters the critical section, effectively locking it for everyone else. When it is complete 10 seconds have passed due to the Sleep-call.
One of the other threads enters the critical section once the first request is finished. It has already waited for 10 seconds, and the thread will now sleep for an additional 10 seconds.
As the second thread finishes one of the remaining two enters the critical section. It has now waited for 20 seconds and will sleep for an additional 10.
The final request is able to enter the critical section after having waited for 30 seconds. Once it is complete 40 seconds have passed.

If we'd been using the performNoLockOperation instead then all requests would have responded after 10 seconds.

Now, I know that 10 seconds is an exceptionally long time to let the thread sleep, but this was all for illustration purposes. Imagine instead that the sleeping time was 1 second and the number of requests was 200...

Troubleshooting this issue

Okay, so what would it look like when this problem occurs? Let's have a look. If you haven't looked at my posts on getting started with Windbg, then I suggest reading them before continuing. (Part I & Part II)

First of all I fire up WinDbg, then I attach to the w3wp.exe. I then use "~*e!clrstack" to look at the managed threads of the w3wp.exe application. Here is a snippet of what I get:

OS Thread Id: 0xce0 (24)
Child-SP         RetAddr          Call Site
000000000447e0f0 000007ff001e13d2 FileIO.performLockOperation()
000000000447e170 000007feee323ec9 _Default.Page_Load(System.Object, System.EventArgs)
000000000447e1b0 000007fee7956aea System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr, System.Object, System.Object, System.EventArgs)
000000000447e1e0 000007fee794d2c4 System.Web.Util.CalliEventHandlerDelegateProxy.Callback(System.Object, System.EventArgs)
000000000447e210 000007fee794d322 System.Web.UI.Control.OnLoad(System.EventArgs)
000000000447e250 000007fee79498ac System.Web.UI.Control.LoadRecursive()
000000000447e2a0 000007fee7948db0 System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)
000000000447e370 000007fee7948cdb System.Web.UI.Page.ProcessRequest(Boolean, Boolean)
000000000447e3d0 000007fee7948c70 System.Web.UI.Page.ProcessRequest()
000000000447e430 000007ff001e0c09 System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)
000000000447e490 000007fee7950117 ASP.default_aspx.ProcessRequest(System.Web.HttpContext)
000000000447e4c0 000007fee791449b System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
000000000447e570 000007fee7ffbd41 System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)
000000000447e610 000007fee7fed132 System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)
000000000447e7a0 000007fee7fcf599 System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)
000000000447e7f0 000007fee80f5344 System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)
000000000447e910 000007fee80f65bb System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
000000000447ea80 000007fee80f4994 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
000000000447eae0 000007fef72701ea DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)
000000000447f310 000007fee80f5424 DomainNeutralILStubClass.IL_STUB(IntPtr, System.Web.RequestNotificationStatus ByRef)
000000000447f3f0 000007fee80f65bb System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
000000000447f560 000007fee80f4994 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
000000000447f5c0 000007fef727043b DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)
OS Thread Id: 0x15ec (25)
Unable to walk the managed stack. The current thread is likely not a 
managed thread. You can run !threads to get a list of managed threads in
the process
OS Thread Id: 0x1958 (26)
Child-SP         RetAddr          Call Site
0000000004c1dd10 000007ff001e13d2 FileIO.performLockOperation()
0000000004c1dd90 000007feee323ec9 _Default.Page_Load(System.Object, System.EventArgs)
0000000004c1ddd0 000007fee7956aea System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr, System.Object, System.Object, System.EventArgs)
0000000004c1de00 000007fee794d2c4 System.Web.Util.CalliEventHandlerDelegateProxy.Callback(System.Object, System.EventArgs)
0000000004c1de30 000007fee794d322 System.Web.UI.Control.OnLoad(System.EventArgs)
0000000004c1de70 000007fee79498ac System.Web.UI.Control.LoadRecursive()
0000000004c1dec0 000007fee7948db0 System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)
0000000004c1df90 000007fee7948cdb System.Web.UI.Page.ProcessRequest(Boolean, Boolean)
0000000004c1dff0 000007fee7948c70 System.Web.UI.Page.ProcessRequest()
0000000004c1e050 000007ff001e0c09 System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)
0000000004c1e0b0 000007fee7950117 ASP.default_aspx.ProcessRequest(System.Web.HttpContext)
0000000004c1e0e0 000007fee791449b System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
0000000004c1e190 000007fee7ffbd41 System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)
0000000004c1e230 000007fee7fed132 System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)
0000000004c1e3c0 000007fee7fcf599 System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)
0000000004c1e410 000007fee80f5344 System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)
0000000004c1e530 000007fee80f65bb System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
0000000004c1e6a0 000007fee80f4994 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
0000000004c1e700 000007fef72701ea DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)
0000000004c1ef30 000007fee80f5424 DomainNeutralILStubClass.IL_STUB(IntPtr, System.Web.RequestNotificationStatus ByRef)
0000000004c1f010 000007fee80f65bb System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
0000000004c1f180 000007fee80f4994 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
0000000004c1f1e0 000007fef727043b DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)
OS Thread Id: 0x1e28 (27)
Child-SP         RetAddr          Call Site
000000000492dcf0 000007ff001e13d2 FileIO.performLockOperation()
000000000492dd70 000007feee323ec9 _Default.Page_Load(System.Object, System.EventArgs)
000000000492ddb0 000007fee7956aea System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr, System.Object, System.Object, System.EventArgs)
000000000492dde0 000007fee794d2c4 System.Web.Util.CalliEventHandlerDelegateProxy.Callback(System.Object, System.EventArgs)
000000000492de10 000007fee794d322 System.Web.UI.Control.OnLoad(System.EventArgs)
000000000492de50 000007fee79498ac System.Web.UI.Control.LoadRecursive()
000000000492dea0 000007fee7948db0 System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)
000000000492df70 000007fee7948cdb System.Web.UI.Page.ProcessRequest(Boolean, Boolean)
000000000492dfd0 000007fee7948c70 System.Web.UI.Page.ProcessRequest()
000000000492e030 000007ff001e0c09 System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)
000000000492e090 000007fee7950117 ASP.default_aspx.ProcessRequest(System.Web.HttpContext)
000000000492e0c0 000007fee791449b System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
000000000492e170 000007fee7ffbd41 System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)
000000000492e210 000007fee7fed132 System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)
000000000492e3a0 000007fee7fcf599 System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)
000000000492e3f0 000007fee80f5344 System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)
000000000492e510 000007fee80f65bb System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
000000000492e680 000007fee80f4994 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
000000000492e6e0 000007fef72701ea DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)
000000000492ef10 000007fee80f5424 DomainNeutralILStubClass.IL_STUB(IntPtr, System.Web.RequestNotificationStatus ByRef)
000000000492eff0 000007fee80f65bb System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
000000000492f160 000007fee80f4994 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
000000000492f1c0 000007fef727043b DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)
OS Thread Id: 0x1c38 (28)
Child-SP         RetAddr          Call Site
0000000004d0e0b0 000007ff001e13d2 FileIO.performLockOperation()
0000000004d0e130 000007feee323ec9 _Default.Page_Load(System.Object, System.EventArgs)
0000000004d0e170 000007fee7956aea System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr, System.Object, System.Object, System.EventArgs)
0000000004d0e1a0 000007fee794d2c4 System.Web.Util.CalliEventHandlerDelegateProxy.Callback(System.Object, System.EventArgs)
0000000004d0e1d0 000007fee794d322 System.Web.UI.Control.OnLoad(System.EventArgs)
0000000004d0e210 000007fee79498ac System.Web.UI.Control.LoadRecursive()
0000000004d0e260 000007fee7948db0 System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)
0000000004d0e330 000007fee7948cdb System.Web.UI.Page.ProcessRequest(Boolean, Boolean)
0000000004d0e390 000007fee7948c70 System.Web.UI.Page.ProcessRequest()
0000000004d0e3f0 000007ff001e0c09 System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)
0000000004d0e450 000007fee7950117 ASP.default_aspx.ProcessRequest(System.Web.HttpContext)
0000000004d0e480 000007fee791449b System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
0000000004d0e530 000007fee7ffbd41 System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)
0000000004d0e5d0 000007fee7fed132 System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)
0000000004d0e760 000007fee7fcf599 System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)
0000000004d0e7b0 000007fee80f5344 System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)
0000000004d0e8d0 000007fee80f65bb System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
0000000004d0ea40 000007fee80f4994 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
0000000004d0eaa0 000007fef72701ea DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)
0000000004d0f2d0 000007fee80f5424 DomainNeutralILStubClass.IL_STUB(IntPtr, System.Web.RequestNotificationStatus ByRef)
0000000004d0f3b0 000007fee80f65bb System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
0000000004d0f520 000007fee80f4994 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
0000000004d0f580 000007fef727043b DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32

So, what does this tell us?

Well, obviously we have four threads that call my function FileIO.performLockOperation() from Page_Load.

Please note that we don't have any additional callstack, and since the performLockOperation-function was named by me it could just as well have been named "foo". In order to find out that these threads are actually waiting for a lock we need to look at the native callstack using the kb-command. Let's take a look at thread 24:

0:029> ~24kb
ntdll!NtDelayExecution+0xa
kernel32!SleepEx+0x84
mscorwks!EESleepEx+0x2d
mscorwks!Thread::UserSleep+0x71
mscorwks!ThreadNative::Sleep+0xf9
App_Code_tvh3usvo!FileIO.performLockOperation()+0x8a
App_Web_uywrbugj!_Default.Page_Load(System.Object, System.EventArgs)+0x32
System_Web_RegularExpressions_ni!System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr, System.Object, System.Object, System.EventArgs)+0x19
System_Web_ni!System.Web.Util.CalliEventHandlerDelegateProxy.Callback(System.Object, System.EventArgs)+0x2a
System_Web_ni!System.Web.UI.Control.OnLoad(System.EventArgs)+0x84
System_Web_ni!System.Web.UI.Control.LoadRecursive()+0x42
System_Web_ni!System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)+0x97c
System_Web_ni!System.Web.UI.Page.ProcessRequest(Boolean, Boolean)+0xa0
System_Web_ni!System.Web.UI.Page.ProcessRequest()+0x5b
System_Web_ni!System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)+0xf0
App_Web_uywrbugj!ASP.default_aspx.ProcessRequest(System.Web.HttpContext)+0x9
System_Web_ni!System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()+0x257
System_Web_ni!System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)+0xab
System_Web_ni!System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)+0x501
System_Web_ni!System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)+0x72

Apparently we entered FileIO.performLockOperation, and then called sleep, (which is what we're doing in this repro in order to simulate a slow IO operation.) So, what do the other threads look like? Here's thread 26:

0:029> ~26kb
ntdll!ZwWaitForMultipleObjects+0xa
kernel32!WaitForMultipleObjectsEx+0x10b
mscorwks!WaitForMultipleObjectsEx_SO_TOLERANT+0xc1
mscorwks!Thread::DoAppropriateAptStateWait+0x41
mscorwks!Thread::DoAppropriateWaitWorker+0x191
mscorwks!Thread::DoAppropriateWait+0x5c
mscorwks!CLREvent::WaitEx+0xbe
mscorwks!AwareLock::EnterEpilog+0xc9
mscorwks!AwareLock::Enter+0x72
mscorwks!AwareLock::Contention+0x1fb
mscorwks!JITutil_MonContention+0xdf
App_Code_tvh3usvo!FileIO.performLockOperation()+0x7f
App_Web_uywrbugj!_Default.Page_Load(System.Object, System.EventArgs)+0x32
System_Web_RegularExpressions_ni!System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr, System.Object, System.Object, System.EventArgs)+0x19
System_Web_ni!System.Web.Util.CalliEventHandlerDelegateProxy.Callback(System.Object, System.EventArgs)+0x2a
System_Web_ni!System.Web.UI.Control.OnLoad(System.EventArgs)+0x84
System_Web_ni!System.Web.UI.Control.LoadRecursive()+0x42
System_Web_ni!System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)+0x97c
System_Web_ni!System.Web.UI.Page.ProcessRequest(Boolean, Boolean)+0xa0
System_Web_ni!System.Web.UI.Page.ProcessRequest()+0x5b

This is one of the threads waiting for the critical section. It then found that it was unable to get a lock and so began waiting for the critical section on thread 24 to end.

Resolving the problem

This is the tricky part. There is no quick-fix for this. We can use the debugger to identify the bottleneck, but after that we'll need to review our design in order to get this working smoothly. A couple of generic things to try would be the following:

Review what you're doing inside the critical section. Is it really necessary to perform all of the operations you're doing, or would it be possible to do some/all of them outside the scope of the critical section?
Identify the reason for the lock taking so much time. Can it be resolved by load-balancing or clustering the resource we're waiting for?
If we're waiting for another server, make sure the physical infrastructure is okay. Are network response times within the expected parameters or do you need to fix your wiring, router, etc?
Are all calls to the critical section taking the same amount of time, or are there certain calls that take an exceptional amount of time? Perhaps most calls to the function are just fine, but intermittently a call comes along that tries to get everything from the AD, database, etc. In order to investigate this I'd use the commands outlined in Getting started with windbg -part II. Identify the thread owning the lock and investigate the callstack and stack objects using !clrstack and !dso. Use !dso and !do to query the stack objects and try to find the SQL query or AD query that is causing the thread to take such a long time to finish. Is the query significantly different from the ones waiting to execute, then this is most likely our culprit.

Post Script

A quick thought on locks and their limits:

Locks do not scale well. A lock can only be made within a specific process, so if you're using a web farm, or in any other way more than one w3wp.exe, then you will not be able to lock all processes. In other words. Server 1 can get a lock, and so can Server 2. Both will happily execute their code independent of each other. For products such as SQL server you'd deal with this in your SQL query, but if you're performing other operations, such as File IO you might have to work around this by using the same approach as Ms Word. Create a lock file in a shared space, and before you enter your critical section you simply check if the file exists:

You are entering your critical section. Look for lock file in shared location
If file exists, then wait
If/when file doesn't/no longer exist, create file and enter critical section
Make sure you have solid exception handling within the critical section so that you under no circumstances leave the critical section without removing the lock file.
When done, remove lock file

/ Johan

Walkthrough - Troubleshooting a native memory leak

JohanS — Thu, 21 Feb 2008 18:38:50 GMT

Problem:

A customer called in. They had a Web Service running on a single IIS6. Memory usage would slowly increase and not be released. As a workaround they'd currently set the application pool to recycle at 500 MB, causing a few failed requests upon each restart.

I thought I'd describe how I went about troubleshooting this scenario.

The first dump

The customer had already taken a memory dump using DebugDiag, so obviously I asked them to upload it. It was a huge dump. 1.64 GB! I opened it up in windbg and ran the following command:

!eeheap -gc

I haven't mentioned !eeheap before. It gives me a nice summary of what's on the managed heap(s).

0:000> !eeheap -gc
Number of GC Heaps: 4
------------------------------
Heap 0 (0x000dfaf0)
generation 0 starts at 0x1124b1c0
generation 1 starts at 0x111c3684
generation 2 starts at 0x102d0030
ephemeral segment allocation context: none
segment    begin       allocated     size
0x102d0000 0x102d0030 0x1165033c 0x138030c(20,448,012)
Large object heap starts at 0x202d0030
segment    begin       allocated     size
0x202d0000 0x202d0030 0x20636810 0x003667e0(3,565,536)
Heap Size 0x16e6aec(24,013,548)
------------------------------
Heap 1 (0x000dfee8)
generation 0 starts at 0x15365b5c
generation 1 starts at 0x152f2138
generation 2 starts at 0x142d0030
ephemeral segment allocation context: none
segment    begin       allocated     size
0x142d0000 0x142d0030 0x1577d230 0x14ad200(21,680,640)
Large object heap starts at 0x252d0030
segment    begin       allocated     size
0x252d0000 0x252d0030 0x25391190 0x000c1160(790,880)
Heap Size 0x156e360(22,471,520)
------------------------------
Heap 2 (0x000e09f8)
generation 0 starts at 0x19480e94
generation 1 starts at 0x193ec5b0
generation 2 starts at 0x182d0030
ephemeral segment allocation context: none
segment    begin       allocated     size
0x182d0000 0x182d0030 0x1985ed34 0x158ed04(22,605,060)
Large object heap starts at 0x222d0030
segment    begin       allocated     size
0x222d0000 0x222d0030 0x222e8748 0x00018718(100,120)
Heap Size 0x15a741c(22,705,180)
------------------------------
Heap 3 (0x000e14e0)
generation 0 starts at 0x1d0f887c
generation 1 starts at 0x1d056738
generation 2 starts at 0x1c2d0030
ephemeral segment allocation context: none
segment    begin       allocated     size
0x1c2d0000 0x1c2d0030 0x1d56af68 0x129af38(19,509,048)
Large object heap starts at 0x232d0030
segment    begin       allocated     size
0x232d0000 0x232d0030 0x23319578 0x00049548(300,360)
Heap Size 0x12e4480(19,809,408)
------------------------------
GC Heap Size 0x54e06e8(88,999,656)

So, as we can see the managed memory is pretty evenly distributed in the 4 managed heaps. If you look down at the bottom you'll see that total memory usage is at 88 MB. (This is a figure I would have gotten by running !dumpheap -stat as well. So in case you're wondering; that would have worked just as fine.) Anyway, if only ~90 MB is in the managed heap, then the remaining 1.55 GB must be on the native heap.

New dumps

Troubleshooting the native heap is a bit harder. The sos extension and windbg is a really nice team so working without them can be tough. Fortunately there is a nice feature included in DebugDiag that lets it perform an automatic analysis. This is usually the best way to get information from a native leak. Sometimes you may want to get a little additional information from windbg as well, but most of the time DebugDiag's auto analysis will get you a long way.

LeakTrack

DebugDiag comes with a handy little dll called LeakTrack. LeakTrack attaches to the process in question and monitors memory allocations and their related call stacks. Since the dll is injected into the process it can have a slight impact on performance, but if you want to troubleshoot the process you really have no other options.

There are two ways to start monitoring a process with LeakTrack:

Create a "Memory and Handle Leak"-rule using the wizard normally displayed upon start up.
Cancel the wizard. Go to the "Processes"-tab. Right-click the w3wp.exe you wish to monitor and select "Monitor for leaks".

LeakTrack will begin monitoring from the moment you inject it. Obviously, it won't be able to get any data for events that have already occurred, so injecting LeakTrack when memory usage is already at it's peak won't really do much good. Instead I usually recycle the process and attach LeakTrack as soon as possible.

This time I chose to let the customer create a "Memory and Handle Leak"-rule using the wizard. This rule then created dumps at certain intervals which the customer uploaded to me.

Analyzing the dumps

So the next step is to let DebugDiag analyze the dumps. Here's how you do it:

Run DebugDiag
Cancel the wizard
Go to the "Advanced Analysis"-tab
"Add Data Files" - point to the dump
Choose "Memory Pressure Analysis" in the "Available Analysis Scripts" list.
Make sure your dump is selected in the "Data Files" list.
"Start Analysis"

The usual suspects

Once DebugDiag has finished analyzing the dump you will get a new browser window with a report. The report I got showed the following right at the top:

So what does this mean? It means that LeakTrack has monitored the allocations made by a certain third party dll. It has also monitored if that memory has been properly released. All in all the .dll now has almost 150 MB of allocations that are unaccounted for. It would be safe to assume that this is our culprit. Further down in the report we have a link called "Leak analysis report". Clicking it will bring us to the following graphical representation:

As you can see the .dll in question is right at the top. But wait a minute! According to the graph mscorsvr has ~9 MB of outstanding allocations as well. Does this mean that mscorsvr is leaking too, though not as much?

The answer is no. As I mentioned before; LeakTrack will monitor all allocations and check that the memory is properly released as well. All we know is that mscorsvr has allocated 8,75 MB of memory that hasn't been released yet. There's nothing to suggest it won't release it eventually. The odds of the stray 148 MB eventually being released, on the other hand, are a lot slimmer in my opinion.

What causes the leak?

Since the dll responsible for the allocations is third party I can't find out exactly what causes the leak within the third party dll. What I can find out is what calls in the customer's code lead us down the leaking code path in the dll. LeakTrack monitors the callstacks related to memory allocations. Looking at the report I was able to find some nice callstacks that the customer should keep an eye on. It might be possible for them to tweak their code slightly in order to avoid the scenario.

3rdparty!IFMX_IsInTransaction+5671
3rdparty!SQLEndTran+e456
odbc32!SQLFetch+160
[DEFAULT] [hasThis] Boolean System.Data.Odbc.OdbcDataReader.Read()
[DEFAULT] [hasThis] I4 System.Data.Common.DbDataAdapter.FillLoadDataRow(Class System.Data.Common.SchemaMapping)
[DEFAULT] [hasThis] I4 System.Data.Common.DbDataAdapter.FillFromReader(Object,String,Class System.Data.IDataReader,I4,I4,Class System.Data.DataColumn,Object)
[DEFAULT] [hasThis] Void System.Data.DataSet.set_EnforceConstraints(Boolean)
[DEFAULT] [hasThis] I4 System.Data.Common.DbDataAdapter.Fill(Class System.Data.DataSet,String,Class System.Data.IDataReader,I4,I4)
[DEFAULT] [hasThis] Void System.Data.DataSet.set_EnforceConstraints(Boolean)
[DEFAULT] [hasThis] I4 System.Data.Common.DbDataAdapter.FillFromCommand(Object,I4,I4,String,Class System.Data.IDbCommand,ValueClass System.Data.CommandBehavior)
[DEFAULT] [hasThis] I4 System.Data.Common.DbDataAdapter.Fill(Class System.Data.DataSet,I4,I4,String,Class System.Data.IDbCommand,ValueClass System.Data.CommandBehavior)
[DEFAULT] [hasThis] I4 System.Data.Common.DbDataAdapter.Fill(Class System.Data.DataSet)
[DEFAULT] [hasThis] Class System.Data.DataSet Company.App.Server.DAL.DAO.Informix.AbstractDAO.Fill(String,String,I4)
[DEFAULT] [hasThis] String Company.App.Server.DAL.DAO.Informix.ResourceDAO.SelectResourcePriceType(String,String)
[DEFAULT] [hasThis] Class System.Data.DataSet Company.App.Server.DAL.DAO.Informix.AbstractDAO.Fill(String,String,I4)
[DEFAULT] [hasThis] Void Company.App.Server.DAL.DAO.Informix.ResourceDAO..ctor(ByRef Class System.Data.IDbConnection,ByRef Class System.Data.IDbTransaction)
[DEFAULT] [hasThis] String Company.App.Server.Services.Pricing.PriceFinder.GetPriceType(String,String,ByRef Class System.Data.IDbConnection,Class System.Data.IDbTransaction)
[DEFAULT] [hasThis] Class Company.App.Core.Query.Result.PriceResult Company.App.Server.Services.Pricing.PriceFinder.GetResourceDiscountPrices(String,String,ValueClass System.DateTime,String,SZArray R4,String,String,ByRef Class System.Data.IDbConnection,Class System.Data.IDbTransaction)
[DEFAULT] [hasThis] String Company.App.Server.Services.Pricing.PriceFinder.GetPriceType(String,String,ByRef Class System.Data.IDbConnection,Class System.Data.IDbTransaction)
[DEFAULT] [hasThis] Class Company.App.Core.Query.Result.PriceResult Company.App.Server.Facade.BookingDataSource.GetResourceDiscountPrices(String,String,ValueClass System.DateTime,String,SZArray R4,String,String)
[DEFAULT] [hasThis] Class Company.App.Core.Query.Result.PriceResult App.BookingData.GetResourceDiscountPrices(String,String,ValueClass System.DateTime,String,SZArray R4,String,String)
[DEFAULT] [hasThis] Class Company.App.Core.Query.Result.PriceResult Company.App.Server.Facade.BookingDataSource.GetResourceDiscountPrices(String,String,ValueClass System.DateTime,String,SZArray R4,String,String)
mscorsvr!CallDescrWorker+30
mscorsvr!COMMember::InvokeMethod+95a
mscorsvr!COMMember::CreateInstance+358
[DEFAULT] [hasThis] SZArray Object System.Web.Services.Protocols.LogicalMethodInfo.Invoke(Object,SZArray Object)
[DEFAULT] [hasThis] Void System.Web.Services.Protocols.WebServiceHandler.Invoke()
[DEFAULT] [hasThis] SZArray Object System.Web.Services.Protocols.LogicalMethodInfo.Invoke(Object,SZArray Object)
[DEFAULT] [hasThis] Void System.Web.Services.Protocols.WebServiceHandler.CoreProcessRequest()
[DEFAULT] [hasThis] Void System.Web.Services.Protocols.WebServiceHandler.Invoke()
[DEFAULT] [hasThis] Void System.Web.Services.Protocols.SyncSessionlessHandler.ProcessRequest(Class System.Web.HttpContext)
[DEFAULT] [hasThis] Void System.Web.Services.Protocols.WebServiceHandler.CoreProcessRequest()
[DEFAULT] [hasThis] Void System.Web.HttpApplication/CallHandlerExecutionStep.System.Web.HttpApplication+IExecutionStep.Execute()
[DEFAULT] [hasThis] Class System.Exception System.Web.HttpApplication.ExecuteStep(Class IExecutionStep,ByRef Boolean)
[DEFAULT] [hasThis] Void System.Web.HttpApplication.ResumeSteps(Class System.Exception)
[DEFAULT] [hasThis] Class System.Exception System.Web.HttpApplication.ExecuteStep(Class IExecutionStep,ByRef Boolean)
[DEFAULT] [hasThis] Class System.IAsyncResult System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest(Class System.Web.HttpContext,Class System.AsyncCallback,Object)
[DEFAULT] [hasThis] Void System.Web.HttpApplication.ResumeSteps(Class System.Exception)
[DEFAULT] [hasThis] Void System.Web.HttpRuntime.ProcessRequestInternal(Class System.Web.HttpWorkerRequest)
[DEFAULT] Void System.Web.HttpRuntime.ProcessRequest(Class System.Web.HttpWorkerRequest)
[DEFAULT] [hasThis] Void System.Web.HttpRuntime.ProcessRequestInternal(Class System.Web.HttpWorkerRequest)
[DEFAULT] [hasThis] I4 System.Web.Hosting.ISAPIRuntime.ProcessRequest(I,I4)
[DEFAULT] Void System.Web.HttpRuntime.ProcessRequest(Class System.Web.HttpWorkerRequest)
mscorsvr!ComToComPlusWorker+1e0
mscorsvr!ComToComPlusWorker_Wrapper+38
mscorsvr!Thread::DoADCallBack+5c
mscorsvr!ComToComPlusWorker+65
mscorsvr!ComCallWrapper::NeedToSwitchDomains+24
aspnet_isapi!HttpCompletion::ProcessRequestInManagedCode+17e
aspnet_isapi!HttpCompletion::ProcessCompletion+24
aspnet_isapi!CorThreadPoolWorkitemCallback+13
mscorsvr!ThreadpoolMgr::ExecuteWorkRequest+19
mscorsvr!ThreadpoolMgr::WorkerThreadStart+129
mscorsvr!ThreadpoolMgr::intermediateThreadProc+44
kernel32!GetModuleHandleForUnicodeString+20
kernel32!GetModuleHandleForUnicodeString+20
kernel32!BasepGetModuleHandleExW+17f
ntdll!LdrpGetProcedureAddress+b3
ntdll!LdrpGetProcedureAddress+b3
ntdll!LdrpCallInitRoutine+14
ntdll!LdrpInitializeThread+18f
ntdll!LdrpInitializeThread+18f
ntdll!ZwContinue+c
kernel32!BaseThreadStart+34

Okay, now what?

Well, this is actually more or less where the road ends for me. Had the problem lied within a Microsoft component it would have been up to us to fix it. Had it been within the customer's code I would have been able to use the customers symbols to get a more detailed callstack and provide him with pointers on how to resolve it as well, but since this is a third party component the next step is to forward it to the third-party vendor.

/ Johan

Using WinDbg - Hunting Exceptions

JohanS — Thu, 31 Jan 2008 19:39:36 GMT

Prerequisites

This post will require some basic knowledge of windbg and the sos extension. For this I recommend looking at the following posts:

For more information on Exceptions in general and why they should be avoided I'd like to recommend this post:

Exceptions

Introduction

I thought it was time to write another post on how to use windbg for troubleshooting. A lot of my time is spent locating exceptions in various web applications, so I thought this might be a good topic to cover. I've previously written a post specifically targeting OutOfMemoryExceptions, but I thought I should broaden the terms and make it a bit more general. There are two scenarios that are exceptionally common in my line of work:

Clients are reporting 2nd chance exceptions displayed on screen with the classic "Server Error"- page.
Performance is generally bad, and when we investigate it turns out that there are tons of exceptions being thrown every second.

In this post I'll cover how to investigate what exceptions have been thrown by an application, as well as how to use windbg and adplus to automatically gather specific information for us.

Where to start

Okay, so you have a web application that you've been monitoring and you believe it is throwing a lot of exceptions. You've taken a dump of the process and you're ready to begin the investigation. Where do you start?

!dumpallexceptions (!dae)

If your application is running under the .NET Framework 1.1 you can use the !dumpallexceptions-command (!dae) to get a list of all the exceptions still on the heap. Now, remember that an exception is a managed object, so they will eventually be garbage collected just like everything else. This means that when looking at the heap for exceptions you will only get the exceptions still in memory, not every exception thrown by the application since startup.

Anyway, if you run !dae you'll get a list of exceptions that looks like this:

0:000> !dae
Going to dump the .NET Exceptions found in the heap.
Loading the heap objects into our cache.
Number of exceptions of this type:        1
Exception object: 026200ec
Exception type: System.ExecutionEngineException
Message:
InnerException:
StackTrace (generated):

StackTraceString:
HResult: 80131506
The current thread is unmanaged
-----------------

Number of exceptions of this type:        1
Exception object: 026200a4
Exception type: System.StackOverflowException
Message:
InnerException:
StackTrace (generated):

StackTraceString:
HResult: 800703e9
The current thread is unmanaged
-----------------

Number of exceptions of this type:        1
Exception object: 0262005c
Exception type: System.OutOfMemoryException
Message:
InnerException:
StackTrace (generated):

StackTraceString:
HResult: 8007000e
The current thread is unmanaged
-----------------

Number of exceptions of this type:        1
Exception object: 0b62cf38
Exception type: System.Data.SqlClient.SqlException
Message: Transaction (Process ID 96) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
InnerException:
StackTrace (generated):
    SP       IP       Function
    1DACECA8 1D0A4D4C Company.Database.AuditTrail.Write(System.String, System.Guid, System.String)
    1DACED6C 1D0A4B98 Company.Database.AuditTrail.AddEntry(EntryType, System.Guid, System.String)
    1DACED90 1D0A4B24 Company.Database.DataFunctions.SaveRow(System.String, System.Guid, Boolean)
    1DACEE20 1DDD7A06 FooBase.PersonApplicationOtherQualificationBase.PersonApplicationOtherQualificationBaseManager.BaseSave(System.Guid, Boolean)
    1DACEE34 1DDD79BF Foo.PersonApplicationOtherQualification.PersonApplicationOtherQualificationManager.Save(System.Guid, Boolean)
    1DACEE48 1DDD77F5 UserControls.PersonApplicationOtherQualificationDetail.OnSave()
    1DACEE70 1DDD2701 Company.Web.UI.Page.InvokeUsercontrolTransaction(System.Web.UI.Control, Company.Web.TransactionType)

StackTraceString:
HResult: 80131904
The current thread is unmanaged
-----------------

Number of exceptions of this type:        3
Exception object: 0b62d23c
Exception type: System.Exception
Message: Thrown while invoking Save on object PersonApplicationOtherQualificationDetail1(ASP.sys_modules_person_personapplicationotherqualificationdetail_ascx)
InnerException: System.Data.SqlClient.SqlException, use !PrintException 0b62cf38 to see more
StackTrace (generated):
    SP       IP       Function
    1DACEDF8 1DDD28C8 Company.Web.UI.Page.InvokeUsercontrolTransaction(System.Web.UI.Control, Company.Web.TransactionType)
    1DACEEC4 1DDD262D Company.Web.UI.Page.InvokeUsercontrolTransaction(System.Web.UI.Control, Company.Web.TransactionType)

StackTraceString:
HResult: 80131500
The current thread is unmanaged
-----------------

Number of exceptions of this type:        8
Exception object: 03c6fbb8
Exception type: System.ArgumentNullException
Message: Value cannot be null.
InnerException:
StackTrace (generated):
    SP       IP       Function
    1F15E3E0 795FC73C System.Guid..ctor(System.String)
    1F15E43C 1D0AB97F Pages.PersonHomePage.Page_Load(System.Object, System.EventArgs)

StackTraceString:
HResult: 80004003
The current thread is unmanaged
-----------------

Number of exceptions of this type:       36
Exception object: 02620134
Exception type: System.Threading.ThreadAbortException
Message:
InnerException:
StackTrace (generated):

StackTraceString:
HResult: 80131530
The current thread is unmanaged
-----------------

Number of exceptions of this type:      116
Exception object: 0396692c
Exception type: System.Web.HttpUnhandledException
Message:
InnerException: System.NullReferenceException, use !PrintException 03966878 to see more
StackTrace (generated):
    SP       IP       Function
    1CE6DF58 6614FDB2 System.Web.UI.Page.HandleError(System.Exception)
    1CE6DFA0 6615681A System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)
    1CE6EF10 66154A8A System.Web.UI.Page.ProcessRequest(Boolean, Boolean)
    1CE6EF48 66154967 System.Web.UI.Page.ProcessRequest()
    1CE6EF80 66154887 System.Web.UI.Page.ProcessRequestWithNoAssert(System.Web.HttpContext)
    1CE6EF88 6615481A System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)
    1CE6EF9C 1C741EAE ASP.sys_pages_application_application_aspx.ProcessRequest(System.Web.HttpContext)
    1CE6EFA8 65FF27D4 System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
    1CE6EFDC 65FC15B5 System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)

StackTraceString:
HResult: 80004005
The current thread is unmanaged
-----------------

Number of exceptions of this type:      136
Exception object: 03966878
Exception type: System.NullReferenceException
Message: Object reference not set to an instance of an object.
InnerException:
StackTrace (generated):
    SP       IP       Function
    1CE6EC2C 1C745B38 Pages.Application.Page_Load(System.Object, System.EventArgs)
    00000000 00000001 System.EventHandler.Invoke(System.Object, System.EventArgs)
    1CE6ED34 66143A84 System.Web.UI.Control.OnLoad(System.EventArgs)
    1CE6ED44 66143AD0 System.Web.UI.Control.LoadRecursive()
    1CE6ED58 66155106 System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)

StackTraceString:
HResult: 80004003
The current thread is unmanaged
-----------------

Total 303 exceptions

As you can see the !dae-command lists all exception types found on the heap. If possible it also gives us a callstack for each exception. Please note, however, that this doesn't mean that the call stack is more or less the same for all exceptions. In the sample above you might have ~20 different callstacks leading to the 136 NullReferenceExceptions you see.

Unfortunately the !dae command is not available in version 2.0 of sos.dll. Still, it's quite easy to get (more or less) the same result by using the !dumpheap command. If we just type "!dumpheap -type Exception" we'll get a list of all objects with the string "Exception" in their class name. This is almost as good.

0:000> !dumpheap -type Exception -stat
------------------------------
Heap 0
total 79 objects
------------------------------
Heap 1
total 76 objects
------------------------------
Heap 2
total 91 objects
------------------------------
Heap 3
total 92 objects
------------------------------
total 338 objects
Statistics:
      MT    Count    TotalSize Class Name
790ff624        3           36 System.Text.DecoderExceptionFallback
790ff5d8        3           36 System.Text.EncoderExceptionFallback
790f9ad4        1           72 System.ExecutionEngineException
790f9a30        1           72 System.StackOverflowException
790f998c        1           72 System.OutOfMemoryException
653c8d04        1           76 System.Data.SqlClient.SqlException
790f984c        3          216 System.Exception
66414de0       18          216 System.Web.HttpApplication+CancelModuleException
7911bc7c       11          352 System.UnhandledExceptionEventHandler
7911a3b0        8          608 System.ArgumentNullException
790f9b78       36         2592 System.Threading.ThreadAbortException
663d9268      116         9744 System.Web.HttpUnhandledException
7915cf40      136         9792 System.NullReferenceException
Total 338 objects

As you can see, this gives us almost the same information except for the callstacks.

Knowing what to ignore

When analyzing data it is always good to know how to filter the information.

The ever-present exceptions

There are a three exceptions that are created as soon as the worker process starts. This means that you will always see them on the heap even if they haven't been thrown at all.:

System.ExecutionEngineException
System.StackOverflowException
System.OutOfMemoryException

So why are they created if we haven't thrown them? - Any guesses?

The answer is quite simple: If you run into a situation where you need to throw any of these exceptions you will probably be in a state where you can't create them. For example, you've run out of memory and are no longer able to allocate even the tiniest string. How would you then be able to allocate enough memory to create a new exception?

So, provided that there's still only one of each on the heap, you can most likely ignore these three exceptions. When it comes to ExecutionEngineExceptions and OutOfMemoryExceptions you will probably have a pretty good idea that this is what you're looking for, and finding a StackOverflowException isn't that hard. If you run !clrstack and find a callstack of 200+ lines you can be more or less certain that this is your problem.

System.Threading.ThreadAbortException

Usually when you see a ThreadAbortExceptions it is because you've called Response.Redirect.

Whenever you call Response.Redirect, this will also result in a call to Response.End. This will terminate the thread prematurely, resulting in a System.Threading.ThreadAbortException. See the callstack below for an example.

    SP       IP       Function
    1ED6F37C 793D74D0 mscorlib_ni!System.Threading.Thread.Abort(System.Object)+0x2c
    1ED6F390 6600CA8C System_Web_ni!System.Web.HttpResponse.End()+0x5c
    1ED6F3A4 6600B8C3 System_Web_ni!System.Web.HttpResponse.Redirect(System.String, Boolean)+0x1f3
    1ED6F3B8 6600B6B7 System_Web_ni!System.Web.HttpResponse.Redirect(System.String)+0x7
    1ED6F3BC 1DDD2E1D Company_Web!Company.Web.UI.Page.RedirectToPreviousPage()+0x125

Obviously I'm not saying you should discard all System.Threading.ThreadAbortExceptions as irrelevant. Even if you have no reason to believe that ThreadAbortExceptions are a major concern it's always a good idea to investigate a few of them. Take a minute or two to confirm that there is an underlying call to Response.End caused by a Response.Redirect. Once you think that you have enough statistical data to imply that the ThreadAbortExceptions are caused by Redirects you can move on.

Examining the Exceptions

Okay, so say we want to look at the callstacks for the System.Data.SqlClient.SqlException, well first of all we need the address for it. As you might remember, this is easily obtained by using !dumpheap without the -stat option.

0:000> !dumpheap -type System.Data.SqlClient.SqlException
------------------------------
Heap 0
Address       MT     Size
total 0 objects
------------------------------
Heap 1
Address       MT     Size
total 0 objects
------------------------------
Heap 2
Address       MT     Size
0b62cf38 653c8d04       76
total 1 objects
------------------------------
Heap 3
Address       MT     Size
total 0 objects
------------------------------
total 1 objects
Statistics:
      MT    Count    TotalSize Class Name
653c8d04        1           76 System.Data.SqlClient.SqlException
Total 1 objects

Now we have the address for the exception. In order to investigate the exception we could use !dumpobject, but there is another command I want to use first.

!printexception (!pe)

Running the !printexception command on the address of an exception will give us some neat information on the exception in question. Here's the result of running !printexception on the SqlException:

0:000> !pe 0b62cf38
Exception object: 0b62cf38
Exception type: System.Data.SqlClient.SqlException
Message: Transaction (Process ID 96) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
InnerException:
StackTrace (generated):
    SP       IP       Function
    1DACECA8 1D0A4D4C Company_Database_1d3c0000!Company.Database.AuditTrail.Write(System.String, System.Guid, System.String)+0x194
    1DACED6C 1D0A4B98 Company_Database_1d3c0000!Company.Database.AuditTrail.AddEntry(EntryType, System.Guid, System.String)+0x48
    1DACED90 1D0A4B24 Company_Database_1d3c0000!Company.Database.DataFunctions.SaveRow(System.String, System.Guid, Boolean)+0x5e4
    1DACEE20 1DDD7A06 FooBase_1d560000!FooBase.PersonApplicationOtherQualificationBase.PersonApplicationOtherQualificationBaseManager.BaseSave(System.Guid, Boolean)+0x2e
    1DACEE34 1DDD79BF Foo_1cfd0000!Foo.PersonApplicationOtherQualification.PersonApplicationOtherQualificationManager.Save(System.Guid, Boolean)+0x27
    1DACEE48 1DDD77F5 Foo_1cfd0000!UserControls.PersonApplicationOtherQualificationDetail.OnSave()+0x65
    1DACEE70 1DDD2701 Company_Web_1d090000!Company.Web.UI.Page.InvokeUsercontrolTransaction(System.Web.UI.Control, Company.Web.TransactionType)+0x1d1

StackTraceString:
HResult: 80131904
The current thread is unmanaged

This is good stuff. The command was even able to generate a callstack for us. (This may not always be the case, since the callstack may very well have gone out of scope.)

!dumpobject (!do) still has its uses

I wouldn't say that !printexception is a complete replacement for !dumpobject when it comes to examining exceptions. !Printexception will fit the exception into a standard template, and since some exceptions may contain more data than others we sometimes want to use !dumpobject as well. The SqlException has a property called _errors that contains a System.Data.SqlClient.SqlErrorCollection that we might want to look at. This is not in the listing above, so we need to use !dumpobject to look at it.

0:000> !do 0b62cf38
Name: System.Data.SqlClient.SqlException
MethodTable: 653c8d04
EEClass: 6540a0d0
Size: 76(0x4c) bytes
(C:\WINDOWS\assembly\GAC_32\System.Data\2.0.0.0__b77a5c561934e089\System.Data.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
790f9244 40000b5        4        System.String 0 instance 00000000 _className
79107d4c 40000b6        8 ...ection.MethodBase 0 instance 00000000 _exceptionMethod
790f9244 40000b7        c        System.String 0 instance 00000000 _exceptionMethodString
790f9244 40000b8       10        System.String 0 instance 0b62cdfc _message
79112734 40000b9       14 ...tions.IDictionary 0 instance 0b62cf84 _data
790f984c 40000ba       18     System.Exception 0 instance 00000000 _innerException
790f9244 40000bb       1c        System.String 0 instance 00000000 _helpURL
790f8a7c 40000bc       20        System.Object 0 instance 0b62d030 _stackTrace
790f9244 40000bd       24        System.String 0 instance 00000000 _stackTraceString
790f9244 40000be       28        System.String 0 instance 00000000 _remoteStackTraceString
790fdb60 40000bf       34         System.Int32 1 instance        0 _remoteStackIndex
790f8a7c 40000c0       2c        System.Object 0 instance 00000000 _dynamicMethods
790fdb60 40000c1       38         System.Int32 1 instance -2146232060 _HResult
790f9244 40000c2       30        System.String 0 instance 00000000 _source
790fcfa4 40000c3       3c        System.IntPtr 1 instance        0 _xptrs
790fdb60 40000c4       40         System.Int32 1 instance -532459699 _xcode
653c8b28 40017e0       44 ...qlErrorCollection 0 instance 0b62cd90 _errors

There we have it. Now we can continue using !dumpobject to investigate it even further if we wish.

Inner exceptions

If we take a look at one of the HttpUnhandledExceptions we find that it has an inner exception. It is even nice enough to let us know how to find out more about it.

0:000> !pe 10544f64
Exception object: 10544f64
Exception type: System.Web.HttpUnhandledException
Message:
InnerException: System.NullReferenceException, use !PrintException 10544df8 to see more
StackTrace (generated):
    SP       IP       Function
    1E3BE1D8 6614FDB2 System_Web_ni!System.Web.UI.Page.HandleError(System.Exception)+0x3e6
    1E3BE220 6615681A System_Web_ni!System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)+0x1b3a
    1E3BF190 66154A8A System_Web_ni!System.Web.UI.Page.ProcessRequest(Boolean, Boolean)+0xd6
    1E3BF1C8 66154967 System_Web_ni!System.Web.UI.Page.ProcessRequest()+0x57
    1E3BF200 66154887 System_Web_ni!System.Web.UI.Page.ProcessRequestWithNoAssert(System.Web.HttpContext)+0x13
    1E3BF208 6615481A System_Web_ni!System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)+0x32
    1E3BF21C 1C741EAE App_Web__ekpvebx!ASP.sys_pages_application_application_aspx.ProcessRequest(System.Web.HttpContext)+0x1e
    1E3BF228 65FF27D4 System_Web_ni!System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()+0x130
    1E3BF25C 65FC15B5 System_Web_ni!System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)+0x41

This means that the System.NullReferenceException mentioned lead to the HttpUnhandledException we're currently investigating. So if we want to find the root cause we'll need to investigate the inner exception as well.

Extra credit

If you've looked at my "Advanced commands"-post a while back you saw some examples of the .foreach command. This is a great command to use, for example if you want to see the callstack for all System.ArgumentNullExceptions. Instead of manually iterating through all the exceptions we can now dump them all at once, check their callstacks, etc.

0:000> .foreach(myVariable {!dumpheap -type System.ArgumentNullException -short}){!pe myVariable;.echo *************}
Exception object: 03c6fbb8
Exception type: System.ArgumentNullException
Message: Value cannot be null.
InnerException:
StackTrace (generated):
    SP       IP       Function
    1F15E3E0 795FC73C mscorlib_ni!System.Guid..ctor(System.String)+0x2a14bc
    1F15E43C 1D0AB97F Foo_1cfd0000!Pages.PersonHomePage.Page_Load(System.Object, System.EventArgs)+0x77

StackTraceString:
HResult: 80004003
The current thread is unmanaged
*************
Exception object: 08378e24
Exception type: System.ArgumentNullException
Message: Value cannot be null.
InnerException:
StackTrace (generated):
    SP       IP       Function
    1CE6EC60 795FC73C mscorlib_ni!System.Guid..ctor(System.String)+0x2a14bc
    1CE6ECBC 1D0AB97F Foo_1cfd0000!Pages.PersonHomePage.Page_Load(System.Object, System.EventArgs)+0x77

StackTraceString:
HResult: 80004003
The current thread is unmanaged
*************
Exception object: 084c0b30
Exception type: System.ArgumentNullException
Message: Value cannot be null.
InnerException:
StackTrace (generated):
    SP       IP       Function
    1E3BEEE0 795FC73C mscorlib_ni!System.Guid..ctor(System.String)+0x2a14bc
    1E3BEF3C 1D0AB97F Foo_1cfd0000!Pages.PersonHomePage.Page_Load(System.Object, System.EventArgs)+0x77

StackTraceString:
HResult: 80004003
The current thread is unmanaged
*************
Exception object: 08522f84
Exception type: System.ArgumentNullException
Message: Value cannot be null.
InnerException:
StackTrace (generated):
    SP       IP       Function
    1E3BEEE0 795FC73C mscorlib_ni!System.Guid..ctor(System.String)+0x2a14bc
    1E3BEF3C 1D0AB97F Foo_1cfd0000!Pages.PersonHomePage.Page_Load(System.Object, System.EventArgs)+0x77

StackTraceString:
HResult: 80004003
The current thread is unmanaged
*************
Exception object: 0c036bf8
Exception type: System.ArgumentNullException
Message: Value cannot be null.
InnerException:
StackTrace (generated):
    SP       IP       Function
    1DC7EB60 795FC73C mscorlib_ni!System.Guid..ctor(System.String)+0x2a14bc
    1DC7EBBC 1D0AB97F Foo_1cfd0000!Pages.PersonHomePage.Page_Load(System.Object, System.EventArgs)+0x77

StackTraceString:
HResult: 80004003
The current thread is unmanaged
*************
Exception object: 105d7f60
Exception type: System.ArgumentNullException
Message: Value cannot be null.
InnerException:
StackTrace (generated):
    SP       IP       Function
    1F39F360 795FC73C mscorlib_ni!System.Guid..ctor(System.String)+0x2a14bc
    1F39F3BC 1D0AB97F Foo_1cfd0000!Pages.PersonHomePage.Page_Load(System.Object, System.EventArgs)+0x77

StackTraceString:
HResult: 80004003
The current thread is unmanaged
*************
Exception object: 106206fc
Exception type: System.ArgumentNullException
Message: Value cannot be null.
InnerException:
StackTrace (generated):
    SP       IP       Function
    1F39F360 795FC73C mscorlib_ni!System.Guid..ctor(System.String)+0x2a14bc
    1F39F3BC 1D0AB97F Foo_1cfd0000!Pages.PersonHomePage.Page_Load(System.Object, System.EventArgs)+0x77

StackTraceString:
HResult: 80004003
The current thread is unmanaged
*************
Exception object: 1077a864
Exception type: System.ArgumentNullException
Message: Value cannot be null.
InnerException:
StackTrace (generated):
    SP       IP       Function
    1E3BEEE0 795FC73C mscorlib_ni!System.Guid..ctor(System.String)+0x2a14bc
    1E3BEF3C 1D0AB97F Foo_1cfd0000!Pages.PersonHomePage.Page_Load(System.Object, System.EventArgs)+0x77

StackTraceString:
HResult: 80004003
The current thread is unmanaged
*************
Unknown option: ------------------------------
*************

Well this post was even longer than usual.

I hope you found it of value, and I'll gladly listen to any comments, feedback or wishes on future topics you might have.

/ Johan

Using WinDbg - Advanced commands

JohanS — Wed, 23 Jan 2008 17:46:00 GMT

Did you know you can build your own advanced commands using for each, if, etc? The complete list of control tokens are:

.if
.else
.elseif
.foreach
.for
.while
.do
.break
.continue
.catch
.leave
.printf
.block

Using these command tokes you can send quite advanced instructions to the debugger that not only will make your job a lot easier, but also impress your manager immensely. :)

.foreach

Let's begin with an easy example. Imagine you want to investigate all strings on the heap that are 6500 bytes or more. To list them you'd simply type !dumpheap -type System.String -min 6500. This will give you the following information:

0:000> !dumpheap -type System.String -min 6500
------------------------------
Heap 0
Address       MT     Size
790da154 790f9244     9280
0264c4d0 790f9244    32788
total 2 objects
------------------------------
Heap 1
Address       MT     Size
total 0 objects
------------------------------
Heap 2
Address       MT     Size
0b62e790 790f9244    11284
total 1 objects
------------------------------
Heap 3
Address       MT     Size
0e6839d0 790f9244    32788
0e717904 790f9244    32788
0fb2a320 790f9244     6828
total 3 objects
------------------------------
total 6 objects
Statistics:
      MT    Count    TotalSize Class Name
790f9244        6       125756 System.String
Total 6 objects

So far, so good. The problem is that in order to investigate each string you'd have to run !dumpobject (!do) on every address. This might be acceptable now that we're only dealing with 6 strings, but what if it were 25, or 100? I don't know if you're aware of this, but if you pass the -short argument to !dumpheap it will give you the minimum information (just the addresses of the objects in question):

0:000> !dumpheap -type System.String -min 6500 -short
790da154
0264c4d0
0b62e790
0e6839d0
0e717904
0fb2a320
------------------------------

Now, let's use this information in a .foreach-clause:

0:000> .foreach(myVariable {!dumpheap -type System.String -min 6500 -short}){!do myVariable;.echo *************}
Name: System.String
MethodTable: 790f9244
EEClass: 790f91a4
Size: 9280(0x2440) bytes
(C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
String:
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWGRkAiEPDxYCHwEFZFhYWFhYWFhYWFhYWFhYWFhYWFhYWF
hYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhkZAInDw8WCh8CBQFFHwMFCjIyLzExLzIwMDcfBAUBVh8FBQFFHwYFASpkZAIpD2QWBGYPZBYEAg
ETC...

Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
790fdb60 4000096        4         System.Int32 1 instance     4632 m_arrayLength
790fdb60 4000097        8         System.Int32 1 instance     4631 m_stringLength
790fad38 4000098        c          System.Char 1 instance       3c m_firstChar
790f9244 4000099       10        System.String 0   shared   static Empty
    >> Domain:Value 000d5eb8:790d57b4 000fb4c0:790d57b4 000ca848:790d57b4 1d8334d8:790d57b4 <<
79122994 400009a       14        System.Char[] 0   shared   static WhitespaceChars
    >> Domain:Value 000d5eb8:026203f0 000fb4c0:02624504 000ca848:026745f0 1d8334d8:026dcef4 <<
*************
Name: System.String
MethodTable: 790f9244
EEClass: 790f91a4
Size: 32786(0x8012) bytes
(C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
String:
g8PFgIfAQUHYWFhYWFhYWRkAgMPDxYCHwEFCWFhYWFhYWFhYWRkAgQPDxYCHwEFCjI4LzEyLzIwMDdkZAIvD2QWAmY
PZBYCZg9kFgICAw8PFgIfAQUFWFhYWFhkZAIxDw8WAh8JZ2QWBGYPZBYEAgEPZBYCZg9kFgQCAQ8WAh8IBQMxcHgWA
gIBDw8WAh8JaGRkAgMPZBYEAgEPDxYCHwEFCkFkZCBSZWNvcmRkZAIDDw8WAh8ABRZ+L0ltYWdlcy9UaXRsZS9OZXc
ETC...

Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
790fdb60 4000096        4         System.Int32 1 instance    16385 m_arrayLength
790fdb60 4000097        8         System.Int32 1 instance    10960 m_stringLength
790fad38 4000098        c          System.Char 1 instance       3c m_firstChar
790f9244 4000099       10        System.String 0   shared   static Empty
    >> Domain:Value 000d5eb8:790d57b4 000fb4c0:790d57b4 000ca848:790d57b4 1d8334d8:790d57b4 <<
79122994 400009a       14        System.Char[] 0   shared   static WhitespaceChars
    >> Domain:Value 000d5eb8:026203f0 000fb4c0:02624504 000ca848:026745f0 1d8334d8:026dcef4 <<
*************
Name: System.String
MethodTable: 790f9244
EEClass: 790f91a4
Size: 11282(0x2c12) bytes
(C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
String:
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWGRkAiEPDxYCHwEFZFhYWFhYWFhYWFhYWFhYWFhYWFhYWFa
YWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWF
hYWFhYWFhYWFhYWFhYWFhkZAInDw8WCh8CBQFFHwMFCjIyLzExLzIwMDcfBAUBVh8FBQFFHwYFASpkZAIpD2QWBGYPZ
ETC...

Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
790fdb60 4000096        4         System.Int32 1 instance     5633 m_arrayLength
790fdb60 4000097        8         System.Int32 1 instance     3092 m_stringLength
790fad38 4000098        c          System.Char 1 instance       5b m_firstChar
790f9244 4000099       10        System.String 0   shared   static Empty
    >> Domain:Value 000d5eb8:790d57b4 000fb4c0:790d57b4 000ca848:790d57b4 1d8334d8:790d57b4 <<
79122994 400009a       14        System.Char[] 0   shared   static WhitespaceChars
    >> Domain:Value 000d5eb8:026203f0 000fb4c0:02624504 000ca848:026745f0 1d8334d8:026dcef4 <<
*************
Name: System.String
MethodTable: 790f9244
EEClass: 790f91a4
Size: 32786(0x8012) bytes
(C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
String:
SRU5UIFBBR0U6IDMgb2YgMTVkZAILD2QWAmYPZBYcAgcPDxYCHwEFZFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWF
hYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYW
FhYWFhkZAINDw8WCh4BRQUBRR4CVFQFCjIyLzExLzIwMDceAlRWBQFWHgJURQUBRR4CVFIFASpkZAITDw8WCh8CBQFF
ETC...

Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
790fdb60 4000096        4         System.Int32 1 instance    16385 m_arrayLength
790fdb60 4000097        8         System.Int32 1 instance    10960 m_stringLength
790fad38 4000098        c          System.Char 1 instance       3c m_firstChar
790f9244 4000099       10        System.String 0   shared   static Empty
    >> Domain:Value 000d5eb8:790d57b4 000fb4c0:790d57b4 000ca848:790d57b4 1d8334d8:790d57b4 <<
79122994 400009a       14        System.Char[] 0   shared   static WhitespaceChars
    >> Domain:Value 000d5eb8:026203f0 000fb4c0:02624504 000ca848:026745f0 1d8334d8:026dcef4 <<
*************
Name: System.String
MethodTable: 790f9244
EEClass: 790f91a4
Size: 32786(0x8012) bytes
(C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
String:
0b25Ib21lBR9BcHBsaWNhdGlvbk5hdmlnYXRpb246aWJ0TG9nb3V0BTdfY3RsMDpQZXJzb25BcHBsaWNhdGlvbkRlZ3J
lZUxpc3Q6U3lzdGVtVGl0bGU6RWRpdEltYWdlBT5fY3RsMDpQZXJzb25BcHBsaWNhdGlvbkRlZ3JlZURpcGxvbWFMaXN
0OlN5c3RlbVRpdGxlOkVkaXRJbWFnZQVDX2N0bDA6UGVyc29uQXBwbGljYXRpb25PdGhlclF1YWxpZmljYXRpb25MaXN
ETC...

Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
790fdb60 4000096        4         System.Int32 1 instance    16385 m_arrayLength
790fdb60 4000097        8         System.Int32 1 instance    10960 m_stringLength
790fad38 4000098        c          System.Char 1 instance       3c m_firstChar
790f9244 4000099       10        System.String 0   shared   static Empty
    >> Domain:Value 000d5eb8:790d57b4 000fb4c0:790d57b4 000ca848:790d57b4 1d8334d8:790d57b4 <<
79122994 400009a       14        System.Char[] 0   shared   static WhitespaceChars
    >> Domain:Value 000d5eb8:026203f0 000fb4c0:02624504 000ca848:026745f0 1d8334d8:026dcef4 <<
*************
Name: System.String
MethodTable: 790f9244
EEClass: 790f91a4
Size: 6826(0x1aaa) bytes
(C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
String:
/wEPDwULLTEyMTQ5MDkyMjgPZBYCAgEPZBYGAgcPZBYEAgEPDxYCHghJbWFnZVVybAUrfi9pbWFnZXMvbmF2ZAILD2QW
aWdhdGlvbi9wYWdlcy9QYWdlMlByb2dyZXNzLmdpZmRkAgIPDxYCHgRUZXh0BRVDVVJSRU5UIFBBR0U6IDMgb2YgMTVk
ZAILD2QWAmYPZBYcAgcPDxYCHwEFZFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
ETC...

Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
790fdb60 4000096        4         System.Int32 1 instance     3405 m_arrayLength
790fdb60 4000097        8         System.Int32 1 instance     3404 m_stringLength
790fad38 4000098        c          System.Char 1 instance       2f m_firstChar
790f9244 4000099       10        System.String 0   shared   static Empty
    >> Domain:Value 000d5eb8:790d57b4 000fb4c0:790d57b4 000ca848:790d57b4 1d8334d8:790d57b4 <<
79122994 400009a       14        System.Char[] 0   shared   static WhitespaceChars
    >> Domain:Value 000d5eb8:026203f0 000fb4c0:02624504 000ca848:026745f0 1d8334d8:026dcef4 <<
*************
Unknown option: ------------------------------
*************

Let's analyze the exact syntax. Here's the command

.foreach(myVariable {!dumpheap -type System.String -min 6500 -short}){!do myVariable;.echo *************}

"myVariable" is, as the name implies, the name of the variable that I wish to use for the data generated by the first set of commands. The second set of commands is what I wish to execute for each token. First I run !do on the variable, and then I use the .echo-command to print a separator in order to make it a bit easier on the eyes.

There are additional parameters you can use. For example you can choose to skip every n number of variables, or specify a text file to be parsed and used as tokens instead. Take a look at the windbg documentation if you're interested.

.shell

I first saw this being used by my colleague Doug Stewart, who is a genius with these things.

0:000> .shell -i - -ci "!iisinfo.clientconns" FIND /c "Request active"
40
.shell: Process exited

What it does is, it runs !iisinfo.clientonns and uses the MS-DOS FIND-command to count the number of times the string "Request active" appears. Off course you could use it to search for certain strings from any type of output, like ".shell -i - -ci "!do 0b62e790" FIND /c /i "<table"" or whatever suits your needs.

Let's take a quick look at the syntax.

When it comes to ".shell" the -i option is mandatory. It specifies the file we want to use for input. In this scenario we don't want to use a file, so we add another hyphen, resulting in the following syntax: ".shell -i -"

We then add the -ci option, which states that we should treat the following commands as an input file instead. ".shell -i - -ci "!iisinfo.clientconns""

Finally we state what shell command we wish to run using this input. ".shell -i - -ci "!iisinfo.clientconns" FIND /c "Request active"".

Naturally we can use any complicated command we wish in the statement, so instead of !iisinfo.clientconns we could run one of our .foreach-loops instead.

/ Johan

Getting started with windbg - part II

JohanS — Mon, 26 Nov 2007 16:49:50 GMT

This is a continuation of my previous post with the imaginative name Getting started with windbg - part I. I'll be assuming that you've read it, so if you haven't I suggest you check it out first. We're still working with the same sample dump, so I'll pretty much pick up right where we left off.

More useful commands

Last time we used some nice commands from the sos-extension to look at the running callstacks, the requests, the CPU load of the threads, etc. We also dug deeper into the callstack to see what distinguished name we used for a SearchRequest. We'll keep using these commands, but also learn a few new ones.

!dumpstackobjects (!dso)

Now, imagine that we're looking at a specific thread and would like to see all managed objects referenced to by the current stack. Is there a way for us to do this? - There most certainly is. It's called dumpstackobjects, or dso for short. If we run it on the current thread we'll see a listing of objects that are referenced by the callstack. The whole output would look something like this (trimmed down to size):

0:050> !dso
OS Thread Id: 0x1e8c (50)
ESP/REG Object   Name
17a9e534 0741f860 System.RuntimeType
17a9e6b8 271fdfe0 System.DirectoryServices.Protocols.LdapConnection
17a9e6bc 27246f20 System.DirectoryServices.Protocols.SEC_WINNT_AUTH_IDENTITY_EX
17a9e740 271fdfe0 System.DirectoryServices.Protocols.LdapConnection
17a9e744 27246f20 System.DirectoryServices.Protocols.SEC_WINNT_AUTH_IDENTITY_EX
17a9e764 27246f20 System.DirectoryServices.Protocols.SEC_WINNT_AUTH_IDENTITY_EX
17a9e768 271fdfe0 System.DirectoryServices.Protocols.LdapConnection
17a9e780 271fdfe0 System.DirectoryServices.Protocols.LdapConnection
17a9e784 27246e38 System.DirectoryServices.Protocols.SearchRequest
17a9e794 271fdf14 System.DirectoryServices.Protocols.LdapDirectoryIdentifier
17a9e7a8 27246ef8 System.Collections.ArrayList
17a9e7bc 27246ef8 System.Collections.ArrayList
17a9e7c8 271fdfe0 System.DirectoryServices.Protocols.LdapConnection
17a9e8a4 27246e38 System.DirectoryServices.Protocols.SearchRequest
17a9e8d0 27246ed8 System.Object[]    (System.Object[])
17a9e8e0 073ff6b8 System.String    cn
17a9e8e4 271fdfe0 System.DirectoryServices.Protocols.LdapConnection
17a9e8f4 27246d00 System.String    CN=Dummy,CN=Accounts,CN=useradm,DC=Dummy,DC=net
17a9e8f8 271fdfe0 System.DirectoryServices.Protocols.LdapConnection
17a9e8fc 27246e38 System.DirectoryServices.Protocols.SearchRequest
17a9e910 03380310 System.String
17a9e914 27246e24 System.Object[]    (System.String[])
17a9e918 272399a8 System.String    CN=OID-Dummy-ABC123,CN=Dummy,CN=Accounts,CN=useradm,DC=Dummy,DC=net
17a9e91c 27246ddc System.String    (CN=OID-Dummy-ABC123)
...etc...

This is extremely useful when we want to see all the objects referenced by this thread alone. If you want to analyze one of the objects, simply copy the the address from the Object-column and use !dumpobject:

0:050> !do 271fdfe0
Name: System.DirectoryServices.Protocols.LdapConnection
MethodTable: 14a2040c
EEClass: 149daf08
Size: 56(0x38) bytes
(C:\WINDOWS\assembly\GAC_MSIL\System.DirectoryServices.Protocols\2.0.0.0__b03f5f7f11d50a3a\System.DirectoryServices.Protocols.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
14a2078c 40000c3        4 ...NetworkCredential 0 instance 00000000 directoryCredential
14a2144c 40000c4        8 ...ificateCollection 0 instance 271fe018 certificatesCollection
1202af88 40000c5       10      System.TimeSpan 1 instance 271fdff0 connectionTimeOut
1466fe50 40000c6        c ...rectoryIdentifier 0 instance 271fdf14 directoryIdentifier
14a2034c 4000236       24         System.Int32 0 instance        2 connectionAuthType
14a223a4 4000237       18 ...dapSessionOptions 0 instance 271fe2d8 options
0fb896d8 4000238       28        System.IntPtr 0 instance 564180944 ldapHandle
120261c8 4000239       2c       System.Boolean 0 instance        0 disposed
120261c8 400023a       2d       System.Boolean 0 instance        0 bounded
120261c8 400023b       2e       System.Boolean 0 instance        0 needRebind
14a22084 400023e       1c ...pResponseCallback 0 instance 271fe03c fd
120261c8 4000243       2f       System.Boolean 0 instance        0 setFQDNDone
120261c8 4000244       30       System.Boolean 0 instance        1 automaticBind
120261c8 4000245       31       System.Boolean 0 instance        1 needDispose
120261c8 4000246       32       System.Boolean 0 instance        1 connected
14a2267c 4000247       20 ...s.QUERYCLIENTCERT 0 instance 271fe394 clientCertificateRoutine
0fd314bc 400023c       20 ...ections.Hashtable 0   shared   static handleTable
    >> Domain:Value 0019daf0:NotInit 11b42540:073fe504 <<
02c36ca0 400023d       24        System.Object 0   shared   static objectLock
    >> Domain:Value 0019daf0:NotInit 11b42540:073fe53c <<
0fd314bc 400023f       28 ...ections.Hashtable 0   shared   static asyncResultTable
    >> Domain:Value 0019daf0:NotInit 11b42540:073fe610 <<
14a21864 4000240       2c ...lResultsProcessor 0   shared   static partialResultsProcessor
    >> Domain:Value 0019daf0:NotInit 11b42540:073fe678 <<
12305e94 4000241       30 ....ManualResetEvent 0   shared   static waitHandle
    >> Domain:Value 0019daf0:NotInit 11b42540:073fe64c <<
14a21954 4000242       34 ...lResultsRetriever 0   shared   static retriever
    >> Domain:Value 0019daf0:NotInit 11b42540:073fe6a8 <<

!dumparray (!da)

As you might have noticed we have a couple of object arrays on the stack. Look for the System.Object[]-type in the listing above and you'll find them. If you execute !dumpobject on an array you'll only get information about the array itself, not it's contents. To get information about what's in the array we need to use the !dumparray-command, or !da for short.

0:050> !do 27239b98
Name: System.Object[]
MethodTable: 02c3896c
EEClass: 02c388ec
Size: 24(0x18) bytes
Array: Rank 1, Number of elements 2, Type CLASS
Element Type: System.String
Fields:
None

0:050> !da 27239b98
Name: System.String[]
MethodTable: 02c3896c
EEClass: 02c388ec
Size: 24(0x18) bytes
Array: Rank 1, Number of elements 2, Type CLASS
Element Methodtable: 02c39310
[0] 272399a8
[1] 27239a44

As you can see the !dumparray-command gives us a bit more information about the object. We can see that it contains System.String data and we get the addresses of the two items in the array. Since they're System.String-objects we can simply use !dumpobject to view their contents.

!objsize

If you look at the listing above you'll see that the size of the object is listed as 24 bytes. To paraphrase Obi Wan Kenobi: "This is true, from a certain point of view." The 24 bytes are the size of the System.Object[]-object itself. Not it's contents. As you can see when we execute !dumparray, the array only contains two references to two strings. These strings are individual objects and could be 32 MB for all we know, so the 24 bytes are not the total size of the array, but it would still be correct to claim that the System.Object[] is only 24 bytes large.

To get the total size of the object we use the !objsize command:

0:050> !objsize 27239b98
sizeof(27239b98) = 348 ( 0x15c) bytes (System.Object[])

!objsize will iterate through all the child objects referenced by the object, as well as all the grandchildren and so on. Apparently the total size of the array and it's child objects is 348 bytes.

If there are a lot of child objects it may take some time for !objsize to calculate the total size of the object. You should also be aware that !objsize may not always be as smart as you'd like. If, for example, you have a custom button control that references it's parent aspx-page you'd get the total size of the aspx-page and all it's child controls. In other words: If !objsize claims that the object in question is ridiculously large you might want to manually check what the object references using !dumpobject.

!dumpheap

This is another command that I use frequently. It is, however a command that you will want to use with at least one argument. Dumpheap with no arguments will dump all objects on the heap, so I usually begin by using the -stat argument, which in itself will write a lot of info on the screen, but it will at least be summarized. Here' you'll find the trimmed down output of !dumpheap -stat:

0:050> !dumpheap -stat
------------------------------
Heap 0
total 2754508 objects
------------------------------
Heap 1
total 2761329 objects
------------------------------
total 5515837 objects
Statistics:
      MT    Count    TotalSize Class Name
16e0a6d8        1           12 System.Collections.Generic.ObjectEqualityComparer`1[[System.Data.ProviderBase.DbConnectionInternal, System.Data]]
16d9cd9c        1           12 System.Xml.Serialization.Configuration.DateTimeSerializationSection+DateTimeSerializationMode
16d9bf30        1           12 System.Diagnostics.OrdinalCaseInsensitiveComparer
16d9112c        1           12 System.Xml.Serialization.NameTable
16d7f664        1           12 System.Xml.Serialization.TempAssemblyCache
163ea85c        1           12 System.Data.Res
1501e4c4        1           12 System.Collections.Generic.ObjectEqualityComparer`1[[System.Web.UI.Control, System.Web]]
14efb138        1           12 System.Net.TimeoutValidator
14ef9964        1           12 System.Net.Cache.HttpRequestCacheLevel
14ef77a8        1           12 Microsoft.Win32.WinInetCache
14ef68e4        1           12 System.Net.WebRequest+WebProxyWrapper
14ef658c        1           12 System.Net.Configuration.ProxyElement+BypassOnLocalValues
14ef63d8        1           12 System.Net.Configuration.ProxyElement+AutoDetectValues
14ef5b68        1           12 System.Net.CaseInsensitiveAscii
14ef5610        1           12 System.Net.HeaderInfoTable
14ef4718        1           12 System.Net.HttpRequestCreator
14db6710        1           12 System.Web.Configuration.MachineKeyValidationConverter
14db3140        1           12 System.Collections.Generic.ObjectEqualityComparer`1[[System.Runtime.Serialization.MemberHolder, mscorlib]]
14b3f4d8        1           12 System.Web.UI.SupportsEventValidationAttribute
...etc...
14a276a8    19578       704808 System.DirectoryServices.Interop.AdsValueHelper
14a2ea24     9196       735680 System.Web.UI.WebControls.Label
14a2e51c    16862       741928 System.Web.UI.WebControls.Style
125778ec    48015       768240 System.Collections.Specialized.NameObjectCollectionBase+NameObjectEntry
120261c8    65842       790104 System.Boolean
14a2ee7c     9198       809424 System.Web.UI.WebControls.Table
14b311c4     9647       810348 System.Web.UI.WebControls.Image
13a2b7dc    34913       837912 System.Web.HttpServerVarsCollectionEntry
14b303a4    10605       848400 System.Web.UI.WebControls.HyperLink
14d8e3d4    77748       932976 Microsoft.Web.UI.WebControls.BaseChildNodeCollection+ActionType
14db90ac    81372       976464 System.Web.UI.WebControls.FontInfo
14b30694    28648      1031328 System.Web.UI.WebControls.TableRow+CellControlCollection
14d8fdbc    38912      1089536 Microsoft.Web.UI.WebControls.TreeNodeCollection
14b3d0bc    86592      1385472 System.Web.UI.Pair
1466c5c4    39305      1414980 System.Web.UI.ControlCollection
14d8e48c    77748      1865952 Microsoft.Web.UI.WebControls.BaseChildNodeCollection+Action
1545061c    38874      2176944 Microsoft.Web.UI.WebControls.TreeNode
14b30eec    52668      2317392 System.Web.UI.WebControls.TableItemStyle
14a2f804    28515      2395260 System.Web.UI.WebControls.TableRow
14a2be6c    40894      2453640 System.Web.UI.LiteralControl
0fd3da00   228792      2745504 System.Int32
14b3e3ac   244793      2937516 System.Web.UI.IndexedString
14a2de94   198580      3177280 System.Web.UI.StateBag
1466c454    80512      3542528 System.Web.UI.Control+OccasionalFields
12302c2c   205849      4116980 System.Collections.Specialized.HybridDictionary
14b30024    52934      4446456 System.Web.UI.WebControls.TableCell
12302f2c   178294      4992232 System.Collections.Specialized.ListDictionary
14a2e284   412762      6604192 System.Web.UI.StateItem
14d8ce64   117078      7024680 Microsoft.Web.UI.WebControls.CssCollection
0fd314bc   132065      7395640 System.Collections.Hashtable
1230319c   422580      8451600 System.Collections.Specialized.ListDictionary+DictionaryNode
1202a58c   380438      9130512 System.Collections.ArrayList
0fd32050   133000     22582944 System.Collections.Hashtable+bucket[]
02c3896c   649842     23275900 System.Object[]
0fd3c12c     3471     36385536 System.Byte[]
001fee20      338     65409920      Free
02c39310   683083    161821000 System.String
Total 5515837 objects
Fragmented blocks larger than 0.5 MB:
    Addr     Size      Followed by
2adf31cc    2.0MB         2aff85d8 System.String
2b006a2c   20.3MB         2c4530d8 System.Net.SocketAddress

As you can see we now get a listing sorted by size of all object-types on the heap. You'll usually find strings down at the bottom since that's what is commonly used the most.

Other useful arguments are -type and -mt (which stands for MethodTable). Using them you're able to see all objects of a specific type. For example. If we want to look at all HttpRequestCreators on the heap (there are one) you'll simply copy it's MethodTable which you'll find in the listing above (14ef4718) and use !dumpheap -mt

0:050> !dumpheap -mt 14ef4718
------------------------------
Heap 0
Address       MT     Size
0342ccf8 14ef4718       12
total 1 objects
------------------------------
Heap 1
Address       MT     Size
total 0 objects
------------------------------
total 1 objects
Statistics:
      MT    Count    TotalSize Class Name
14ef4718        1           12 System.Net.HttpRequestCreator

This gives us the address of the object and if we'd like to inspect it closer we simply use !dumpobject on that address.

!dumpheap -type works pretty much the same way, except this time you filter the results by class name. !dumpheap -type performs a substring match, so if you write !dumpheap -type System.Web you'll get every object who's class name contains System.Web, which would be a lot.

Other useful arguments are -min and -max which accept a number representing the minimum/maximum number of bytes the object size should be. This can be really useful when troubleshooting string-abuse, etc. Also !dumpheap -stat -min 85000 would list all objects on the large object heap.

Putting the tools to use

I'd now like to use the commands we've covered in a bit more practical scenario. The dump we've been looking at is from a previous case of mine. The application in question was running on a Web garden with two workerprocesses. Session State was handled by a SQL Server. The customer was experiencing performance issues and the problem description was hazy at best. Anyway I had tons of dumps to work with, so I simply poked around to see what I could find. One thing I did pretty early on was to look at caching. According to the customer they weren't using the cache at all, but I usually find it best to double-check this type of thing.

Determining the size of the cache

To find out how much data was kept in the cache I first needed to find the System.Web.Caching.Cache class. So I ran !dumpheap -stat -type System.Web.Caching.Cache. Note that I also used the -stat argument. Otherwise I would have gotten a very long list of System.Web.Caching.CacheKeys and System.Web.Caching.CacheEntrys as well. Anyway, here's the result:

0:050> !dumpheap -type System.Web.Caching.Cache -stat
------------------------------
Heap 0
total 665 objects
------------------------------
Heap 1
total 1084 objects
------------------------------
total 1749 objects
Statistics:
      MT    Count    TotalSize Class Name
123056f8        1           12 System.Web.Caching.CacheKeyComparer
1230494c        1           12 System.Web.Caching.Cache
1230500c        1           24 System.Web.Caching.CacheMultiple
1230514c        1           32 System.Web.Caching.CacheMemoryStats
123053b4        1           36 System.Web.Caching.CacheMemoryTotalMemoryPressure
123059bc        2           40 System.Web.Caching.CacheUsage
12304bdc        1           48 System.Web.Caching.CacheCommon
123054f4        1           52 System.Web.Caching.CacheMemoryPrivateBytesPressure
12305874        2           64 System.Web.Caching.CacheExpires
12304e64        2          200 System.Web.Caching.CacheSingle
1255b594       85         1360 System.Web.Caching.CacheDependency+DepFileInfo
123046c4       40         1440 System.Web.Caching.CacheDependency
123042ec       47         1504 System.Web.Caching.CacheItemRemovedCallback
123063fc      832        16640 System.Web.Caching.CacheKey
12306820      732        52704 System.Web.Caching.CacheEntry
Total 1749 objects

Okay, so now I had the MethodTable for the System.Web.Caching.Cache object. Therefore I could now get the address to the object itself. I did this by asking !dumpheap to list all the objects on the heap with that MethodTable. I knew that there was just going to be one hit:

0:050> !dumpheap -mt 1230494c
------------------------------
Heap 0
Address       MT     Size
03392d20 1230494c       12
total 1 objects
------------------------------
Heap 1
Address       MT     Size
total 0 objects
------------------------------
total 1 objects
Statistics:
      MT    Count    TotalSize Class Name
1230494c        1           12 System.Web.Caching.Cache
Total 1 objects

So now I ran !objsize on this object to see how big it was. This took a little time to calculate, since the cache is quite complex and there are a lot of children to iterate through.

0:050> !objsize 03392d20
sizeof(03392d20) = 266640828 ( 0xfe49dbc) bytes (System.Web.Caching.Cache)

So the cache is 266 MB in size. That's quite a lot considering the fact that the customer claimed that they weren't using the cache at all!

What is being cached?

To sample what the customer was caching I then took a look at a few of the CacheEntrys. I already had the MethodTable for the System.Web.Caching.CacheEntry from when I ran !dumpheap -type System.Web.Caching.Cache -stat (above), so I could use that to retrieve all CacheEntrys.

0:050> !dumpheap -mt 12306820
------------------------------
Heap 0
Address       MT     Size
033950bc 12306820       72
033a20d8 12306820       72
033ac79c 12306820       72
033da21c 12306820       72
033f04c4 12306820       72
03428ec8 12306820       72
0344dab4 12306820       72
03815d00 12306820       72
038265d8 12306820       72
....etc...
03af7010 12306820       72
03b291bc 12306820       72
03b2c674 12306820       72
03b6dca0 12306820       72
03b797dc 12306820       72
03b85318 12306820       72
03ba9150 12306820       72
03c258cc 12306820       72
03de43c8 12306820       72
03e160f8 12306820       72
total 382 objects
------------------------------
total 732 objects

Another valid command that would have given me the exact same output would off course have been !dumpheap -type System.Web.Caching.CacheEntry.

Okay, so now I had a long list of CacheEntrys. To sample the contents I just picked an address and examined it by using !dumpobject

0:050> !do 03b2c674
Name: System.Web.Caching.CacheEntry
MethodTable: 12306820
EEClass: 122f6470
Size: 72(0x48) bytes
(C:\WINDOWS\assembly\GAC_32\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
02c39310 4001327        4        System.String 0 instance 03b2c600 _key
0fb8f1f8 4001328        c          System.Byte 0 instance        2 _bits
0fd3da00 4001329        8         System.Int32 0 instance -1314181915 _hashCode
02c36ca0 4001330       10        System.Object 0 instance 03b2c644 _value
120219d0 4001331       1c      System.DateTime 1 instance 03b2c690 _utcCreated
120219d0 4001332       24      System.DateTime 1 instance 03b2c698 _utcExpires
1202af88 4001333       2c      System.TimeSpan 1 instance 03b2c6a0 _slidingExpiration
0fb8f1f8 4001334        d          System.Byte 0 instance        7 _expiresBucket
123062d8 4001335       34 ...g.ExpiresEntryRef 1 instance 03b2c6a8 _expiresEntryRef
0fb8f1f8 4001336        e          System.Byte 0 instance 4294967295 _usageBucket
12306738 4001337       38 ...ing.UsageEntryRef 1 instance 03b2c6ac _usageEntryRef
120219d0 4001338       3c      System.DateTime 1 instance 03b2c6b0 _utcLastUpdate
123046c4 4001339       14 ...g.CacheDependency 0 instance 00000000 _dependency
02c36ca0 400133a       18        System.Object 0 instance 033d8344 _onRemovedTargets
120219d0 400132d      1bc      System.DateTime 1   shared   static NoAbsoluteExpiration
    >> Domain:Value 0019daf0:NotInit 11b42540:03395104 <<
1202af88 400132e      1c0      System.TimeSpan 1   shared   static NoSlidingExpiration
    >> Domain:Value 0019daf0:NotInit 11b42540:03395114 <<
1202af88 400132f      1c4      System.TimeSpan 1   shared   static OneYear
    >> Domain:Value 0019daf0:NotInit 11b42540:03395124 <<

This dumped the CacheEntry and it's properties. I figured the most interesting piece of information would be the _value, so I simply copied the address of that property (look in the Value column) and used !dumpobject again.

0:000> !do 03e160c8
Name: System.Web.SessionState.InProcSessionState
MethodTable: 14dbad5c
EEClass: 14e43af8
Size: 48(0x30) bytes
(C:\WINDOWS\assembly\GAC_32\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
1466c9d8 4001d89        4 ...ateItemCollection 0 instance 1a7f5438 _sessionItems
1292672c 4001d8a        8 ...ObjectsCollection 0 instance 00000000 _staticObjects
0fd3da00 4001d8b        c         System.Int32 0 instance       20 _timeout
120261c8 4001d8c       18       System.Boolean 0 instance        0 _locked
120219d0 4001d8d       1c      System.DateTime 1 instance 03e160e4 _utcLockDate
0fd3da00 4001d8e       10         System.Int32 0 instance        1 _lockCookie
1202bf60 4001d8f       24 ...ReadWriteSpinLock 1 instance 03e160ec _spinLock
0fd3da00 4001d90       14         System.Int32 0 instance        0 _flags

Here I found something interesting. The value stored was in fact an InProcSessionState object, which -in case you didn't know this before- is stored in the Cache. This meant that the claim that the application was using SQL Server Session state was incorrect.

As it turned out the customer had temporarily switched to In-process for a brief test, but forgotten to switch back again. Had the application been live they would have spotted this in no time, but since they were stress testing they weren't really paying attention to what the server was returning as long as it returned something. Unknowingly running session state in-process compromised the stress-test in a number of ways. For example:

Memory usage for the worker process was a lot higher than expected
The load on the network was not as high as it would be under normal circumstances
The response times were no longer accountable
The SQL-server was not stressed to the extent it should have been, so any potential bottlenecks there went unnoticed

This was in no way the final solution for their performance issue. There were a lot more things we had to deal with, but I think it's a really nice example of how to use only three commands (!dumpheap, !objsize & !dumpobject) to dig up some really useful information.

Later! / Johan

Getting started with windbg - part I

JohanS — Tue, 13 Nov 2007 17:20:11 GMT

Okay, I've previously written some random posts about how to set up windbg and how to troubleshoot OutOfMemoryExceptions. I thought I'd take a few steps back and review some of the basics in order to help you get started on using this fantastic tool.

Basic Configuration

Copy sos.dll from the framework directory to the folder where you installed windbg. Make sure you copy it from the same Framework version as the one you wish to investigate. If you'll be working with both 1.1 and 2.0 you can rename the SOS.dlls to SOS11.dll and SOS20.dll or put them in separate folders.
Create a folder where you want to cache all the symbol files. For example: "C:\Symbols".
Start windbg and open the dialogue to configure the symbol path by clicking File -> Symbol File Path.
Enter the path, as well as the address from which you'll want to download missing symbols using the following syntax:
srv*[cache path]*[symbols path]
I'd recommend the following path:
srv*c:\symbols\public*http://msdl.microsoft.com/download/symbols

You should now be set to go. You're now ready to open up a saved dump, or attach to a process

Useful commands

I'll be using a dump from an IIS6-server to demonstrate some useful commands.

The first thing you'll want to do is load SOS. You'll do this using the .load command. The syntax is simple. .load [extension filename]. So if you want to load sos and haven't renamed the .dll you'd simply write:

.load sos

You'll now have all the cool commands from the SOS-extension at your disposal as well as the default windbg ones. Commands from extensions are always preceded by a "!", so if you want to run the help-command for sos you'd write

!help

If you should happen to have two extensions with an identically named command you can always separate them by typing ![extension name].[command] Example:

!sos.help

Okay, now that we know how to run the commands from the extension, try running !help. It should give you the following result.

0:000> !help
-------------------------------------------------------------------------------
SOS is a debugger extension DLL designed to aid in the debugging of managed
programs. Functions are listed by category, then roughly in order of
importance. Shortcut names for popular functions are listed in parenthesis.
Type "!help " for detailed info on that function.

Object Inspection                  Examining code and stacks
-----------------------------      -----------------------------
DumpObj (do)                       Threads
DumpArray (da)                     CLRStack
DumpStackObjects (dso)             IP2MD
DumpHeap                           U
DumpVC                             DumpStack
GCRoot                             EEStack
ObjSize                            GCInfo
FinalizeQueue                      EHInfo
PrintException (pe)                COMState
TraverseHeap                       BPMD

Examining CLR data structures      Diagnostic Utilities
-----------------------------      -----------------------------
DumpDomain                         VerifyHeap
EEHeap                             DumpLog
Name2EE                            FindAppDomain
SyncBlk                            SaveModule
DumpMT                             GCHandles
DumpClass                          GCHandleLeaks
DumpMD                             VMMap
Token2EE                           VMStat
EEVersion                          ProcInfo
DumpModule                         StopOnException (soe)
ThreadPool                         MinidumpMode
DumpAssembly
DumpMethodSig                      Other
DumpRuntimeTypes                   -----------------------------
DumpSig                            FAQ
RCWCleanupList
DumpIL

For more documentation on a specific command, type !help [name of command]

.time

This is not an SOS-command, which is evident by the command not beginning with a "!". Running the .Time command will show you relevant info about the time, as well as system uptime, process uptime and the amount of time spent in kernel & user mode.

0:000> .time
Debug session time: Tue Oct 23 08:38:35.000 2007 (GMT+1)
System Uptime: 4 days 17:48:01.906
Process Uptime: 0 days 0:24:37.000
Kernel time: 0 days 0:04:23.000
User time: 0 days 0:03:28.000

As you can see the system has been up for over 4 days. The process has been running for 24½ minutes and has an accumulated CPU-time of about 8 minutes total. This would give us an average CPU-usage for the process of around 32,5%

!threadpool

We can then use the !Threadpool-command to find out exactly what the CPU-usage was at the time the dump was taken. We'll also get some useful information like the number of work requests in the queue, completion port threads and timers.

0:000> !threadpool
CPU utilization 100%
Worker Thread: Total: 5 Running: 4 Idle: 1 MaxLimit: 200 MinLimit: 2
Work Request in Queue: 16
Unknown Function: 6a2d945d Context: 023ede30
Unknown Function: 6a2d945d Context: 023ee1e8
AsyncTimerCallbackCompletion TimerInfo@11b53760
Unknown Function: 6a2d945d Context: 023ee3a8
Unknown Function: 6a2d945d Context: 023e3040
Unknown Function: 6a2d945d Context: 023ee178
Unknown Function: 6a2d945d Context: 023edfb0
AsyncTimerCallbackCompletion TimerInfo@11b36428
AsyncTimerCallbackCompletion TimerInfo@11b53868
Unknown Function: 6a2d945d Context: 023ee060
Unknown Function: 6a2d945d Context: 023ee290
Unknown Function: 6a2d945d Context: 023eded0
Unknown Function: 6a2d945d Context: 023edd88
Unknown Function: 6a2d945d Context: 023ede98
Unknown Function: 6a2d945d Context: 023ee258
Unknown Function: 6a2d945d Context: 023edfe8
--------------------------------------
Number of Timers: 9
--------------------------------------
Completion Port Thread:Total: 3 Free: 3 MaxFree: 4 CurrentLimit: 2 MaxLimit: 200 MinLimit: 2

So we can see that currently we're using 100% of the CPU, which leads us to the next command.

!runaway

This is a nice command that will list all running threads and their CPU-usage. It's your best friend when troubleshooting a high CPU hang issue.

0:000> !runaway
User Mode Time
Thread       Time
25:1a94      0 days 0:00:39.937
16:1bc0      0 days 0:00:38.390
50:1e8c      0 days 0:00:08.859
52:1e40      0 days 0:00:08.687
20:1c2c      0 days 0:00:08.234
51:1340      0 days 0:00:08.171
21:1bcc      0 days 0:00:06.953
26:13ec      0 days 0:00:06.671
44:131c      0 days 0:00:03.906
22:d8c       0 days 0:00:03.375
33:78c       0 days 0:00:02.656
34:1a8c      0 days 0:00:00.906
29:1f5c      0 days 0:00:00.828
   6:e28       0 days 0:00:00.625
   5:1c78      0 days 0:00:00.546
23:14a4      0 days 0:00:00.484
   4:5ac       0 days 0:00:00.437
45:5dc       0 days 0:00:00.421
   3:13b4      0 days 0:00:00.421
47:19c8      0 days 0:00:00.375
28:1b6c      0 days 0:00:00.250
46:1dac      0 days 0:00:00.156
   7:1dd8      0 days 0:00:00.109
48:cdc       0 days 0:00:00.093
49:1eac      0 days 0:00:00.062
15:1a64      0 days 0:00:00.062
   0:1804      0 days 0:00:00.046
36:4a4       0 days 0:00:00.031
11:1eb4      0 days 0:00:00.031
   1:10b4      0 days 0:00:00.031
31:16ac      0 days 0:00:00.015
14:4ac       0 days 0:00:00.015
   2:186c      0 days 0:00:00.015
59:590       0 days 0:00:00.000
58:294       0 days 0:00:00.000
57:16d0      0 days 0:00:00.000
56:1578      0 days 0:00:00.000
55:1428      0 days 0:00:00.000
54:16d8      0 days 0:00:00.000
53:fd8       0 days 0:00:00.000
43:1b8c      0 days 0:00:00.000
42:1c24      0 days 0:00:00.000
41:1e2c      0 days 0:00:00.000
40:11b0      0 days 0:00:00.000
39:edc       0 days 0:00:00.000
38:1a08      0 days 0:00:00.000
37:171c      0 days 0:00:00.000
35:1254      0 days 0:00:00.000
32:1f9c      0 days 0:00:00.000
30:1ae8      0 days 0:00:00.000
27:190c      0 days 0:00:00.000
24:1d2c      0 days 0:00:00.000
19:1e38      0 days 0:00:00.000
18:ee4       0 days 0:00:00.000
17:fb8       0 days 0:00:00.000
13:1b54      0 days 0:00:00.000
12:1a48      0 days 0:00:00.000
10:f64       0 days 0:00:00.000
   9:1024      0 days 0:00:00.000
   8:1b78      0 days 0:00:00.000

As you can see the total amount of time does not match the total CPU utilization time that we got from the .time command. That's simply because threads get reused and recycled. This means that the total amount of CPU time used by a thread may have been split up over several page requests.

!threads

To get more information about the running threads we can run the !Threads-command. This will list all managed threads in the application, what application domain the thread is currently executing under, etc. The output will look like this:

0:000> !threads
ThreadCount: 48
UnstartedThread: 0
BackgroundThread: 29
PendingThread: 0
DeadThread: 19
Hosted Runtime: no
                                      PreEmptive   GC Alloc           Lock
       ID OSID ThreadOBJ    State     GC       Context       Domain   Count APT Exception
16    1 1bc0 001fccd0   1808220 Enabled 00000000:00000000 0019daf0     0 Ukn (Threadpool Worker)
22    2 d8c 002016f0      b220 Enabled 00000000:00000000 0019daf0     0 MTA (Finalizer)
14    4 4ac 00242e58   880a220 Enabled 00000000:00000000 0019daf0     0 MTA (Threadpool Completion Port)
23    5 14a4 11b39f18    80a220 Enabled 00000000:00000000 0019daf0     0 MTA (Threadpool Completion Port)
24    6 1d2c 11b41ad8      1220 Enabled 00000000:00000000 0019daf0     0 Ukn
25    7 1a94 11b46c70   180b220 Enabled 27240c98:27241fd8 11b42540     1 MTA (Threadpool Worker)
26    9 13ec 12ce2888   200b220 Enabled 2a9f1434:2a9f33c0 11b42540     0 MTA
27    a 190c 12d85eb8   200b220 Enabled 00000000:00000000 11b42540     0 MTA
29    b 1f5c 13df6a50   200b220 Enabled 2ab1da6c:2ab1f1c0 11b42540     0 MTA
30    c 1ae8 12d44a58      b220 Enabled 00000000:00000000 11b42540     0 MTA
31    d 16ac 12e2e008   200b220 Enabled 2a81348c:2a8153c0 11b42540     1 MTA
   5    e 1c78 12da2160       220 Enabled 00000000:00000000 0019daf0     0 Ukn
33    8 78c 11b674c8   200b220 Enabled 2707b818:2707c1d8 11b42540     0 MTA
34   12 1a8c 13f163c8       220 Enabled 00000000:00000000 0019daf0     0 Ukn
36   13 4a4 13eef718   200b220 Enabled 2a7db4a4:2a7dd3c0 11b42540     0 MTA
   4   14 5ac 13ef2008       220 Enabled 00000000:00000000 0019daf0     0 Ukn
42   10 1c24 13f0e950   880b220 Enabled 00000000:00000000 0019daf0     0 MTA (Threadpool Completion Port)
   6   11 e28 13f16008       220 Enabled 00000000:00000000 0019daf0     0 Ukn
   3    f 13b4 13eba008       220 Enabled 00000000:00000000 0019daf0     0 Ukn
43   15 1b8c 140db008   880b220 Enabled 00000000:00000000 0019daf0     0 MTA (Threadpool Completion Port)
44   17 131c 140ceb28   200b220 Enabled 272288c8:27229fd8 11b42540     0 MTA
45   1d 5dc 140cd0a0       220 Enabled 00000000:00000000 0019daf0     0 Ukn
47   20 19c8 1651a008       220 Enabled 00000000:00000000 0019daf0     0 Ukn
XXXX   24    0 16468880   1801820 Enabled 00000000:00000000 0019daf0     0 Ukn (Threadpool Worker)
46   1f 1dac 1650ab48       220 Enabled 00000000:00000000 0019daf0     0 Ukn
XXXX   1a    0 140d5008   1801820 Enabled 00000000:00000000 0019daf0     0 Ukn (Threadpool Worker)
XXXX   16    0 140c5008   1801820 Enabled 00000000:00000000 0019daf0     0 Ukn (Threadpool Worker)
50    3 1e8c 14064420   180b220 Enabled 27246f54:27247fd8 11b42540     1 MTA (Threadpool Worker)
XXXX   35    0 1406e800   1801820 Enabled 00000000:00000000 0019daf0     0 Ukn (Threadpool Worker)
51   36 1340 140df008   180b220 Enabled 2adec9cc:2aded1c0 11b42540     1 MTA (Threadpool Worker)
XXXX   37    0 16566868   1801820 Enabled 00000000:00000000 0019daf0     0 Ukn (Threadpool Worker)
48   38 cdc 16578840       220 Enabled 00000000:00000000 0019daf0     0 Ukn
XXXX   39    0 16566c28   1801820 Enabled 00000000:00000000 0019daf0     0 Ukn (Threadpool Worker)
XXXX   3b    0 1646b8b0   1801820 Enabled 00000000:00000000 0019daf0     0 Ukn (Threadpool Worker)
XXXX   3c    0 16674008   1801820 Enabled 00000000:00000000 0019daf0     0 Ukn (Threadpool Worker)
XXXX   3d    0 16676418   1801820 Enabled 00000000:00000000 0019daf0     0 Ukn (Threadpool Worker)
XXXX   3e    0 16676fb8   1801820 Enabled 00000000:00000000 0019daf0     0 Ukn (Threadpool Worker)
XXXX   3f    0 16674d48   1801820 Enabled 00000000:00000000 0019daf0     0 Ukn (Threadpool Worker)
XXXX   40    0 1667de10   1801820 Enabled 00000000:00000000 0019daf0     0 Ukn (Threadpool Worker)
XXXX   41    0 16680050   1801820 Enabled 00000000:00000000 0019daf0     0 Ukn (Threadpool Worker)
XXXX   42    0 166812e8   1801820 Enabled 00000000:00000000 0019daf0     0 Ukn (Threadpool Worker)
XXXX   43    0 16683e60   1801820 Enabled 00000000:00000000 0019daf0     0 Ukn (Threadpool Worker)
52   44 1e40 165259e8   180b220 Enabled 2adf126c:2adf31c0 11b42540     1 MTA (Threadpool Worker)
XXXX   45    0 165b7c08   1801820 Enabled 00000000:00000000 0019daf0     0 Ukn (Threadpool Worker)
XXXX   46    0 165aa3d8   1801820 Enabled 00000000:00000000 0019daf0     0 Ukn (Threadpool Worker)
XXXX   47    0 165242c8   1801820 Enabled 00000000:00000000 0019daf0     0 Ukn (Threadpool Worker)
XXXX   48    0 165e9500   1801820 Enabled 00000000:00000000 0019daf0     0 Ukn (Threadpool Worker)
49   3a 1eac 165676f0       220 Enabled 00000000:00000000 0019daf0     0 Ukn

The threads with an ID of XXXX have ended and are waiting to be recycled. We can also see that the finalizer thread has an ID of 22. So if we'd seen an unhealthy amount of activity on thread 22 when we ran the !runaway-command we would now have known that we had a finalizer-issue on our hands.

Switching to a specific thread

To go to a specific thread we use the ~-command. The syntax is as follows: ~[number of thread]s. So to switch to thread 50 we would type the following:

0:000> ~50s

We have then switched to thread 50, and can use a lot of other useful commands.

!clrstack

This great command will list the callstack for the current thread. If you want additional information you can add the "-p" switch which will show you parameters and local variables as well.

Below is a sample listing of the clrstack for thread 50.

0:050> !clrstack
OS Thread Id: 0x1e8c (50)
ESP EIP
17a9e750 7d61c828 [NDirectMethodFrameSlim: 17a9e750] System.DirectoryServices.Protocols.Wldap32.ldap_bind_s(IntPtr, System.String, System.DirectoryServices.Protocols.SEC_WINNT_AUTH_IDENTITY_EX, System.DirectoryServices.Protocols.BindMethod)
17a9e768 14df70f9 System.DirectoryServices.Protocols.LdapConnection.BindHelper(System.Net.NetworkCredential, Boolean)
17a9e794 14df6de0 System.DirectoryServices.Protocols.LdapConnection.Bind()
17a9e79c 14df59e9 System.DirectoryServices.Protocols.LdapConnection.SendRequestHelper(System.DirectoryServices.Protocols.DirectoryRequest, Int32 ByRef)
17a9e8b8 14df56e8 System.DirectoryServices.Protocols.LdapConnection.SendRequest(System.DirectoryServices.Protocols.DirectoryRequest, System.TimeSpan)
17a9e8bc 14df5657 [InlinedCallFrame: 17a9e8bc]

So, reading from the bottom up we can see that an LdapConnection called the SendRequest method, which in turn called the SendRequestHelper method, which called the Bind method, and so on.

If we run !clrstack -p we get the following:

0:050> !clrstack -p
OS Thread Id: 0x1e8c (50)
ESP       EIP
17a9e750 7d61c828 [NDirectMethodFrameSlim: 17a9e750] System.DirectoryServices.Protocols.Wldap32.ldap_bind_s(IntPtr, System.String, System.DirectoryServices.Protocols.SEC_WINNT_AUTH_IDENTITY_EX, System.DirectoryServices.Protocols.BindMethod)
17a9e768 14df70f9 System.DirectoryServices.Protocols.LdapConnection.BindHelper(System.Net.NetworkCredential, Boolean)
    PARAMETERS:
        this = 0x271fdfe0
        newCredential =
        needSetCredential =

17a9e794 14df6de0 System.DirectoryServices.Protocols.LdapConnection.Bind()
    PARAMETERS:
        this =

17a9e79c 14df59e9 System.DirectoryServices.Protocols.LdapConnection.SendRequestHelper(System.DirectoryServices.Protocols.DirectoryRequest, Int32 ByRef)
    PARAMETERS:
        this = 0x271fdfe0
        request = 0x27246e38
        messageID = 0x17a9e8ec

17a9e8b8 14df56e8 System.DirectoryServices.Protocols.LdapConnection.SendRequest(System.DirectoryServices.Protocols.DirectoryRequest, System.TimeSpan)
    PARAMETERS:
        this = 0x271fdfe0
        request = 0x27246e38
        requestTimeout =

17a9e8bc 14df5657 [InlinedCallFrame: 17a9e8bc]

We can now look at the parameters, like the DirectoryRequest that was sent to the SendRequest and SendRequestHelper methods. To do this we simply copy the address of the request, (0x27246e38) and use it as an argument for our next command.

!dumpobject (!do)

This is another crucial command. Dumpobject will dump the object at the specified address, so if we send the address of the request as a parameter we will get the request dumped to screen.:

0:050> !do 0x27246e38
Name: System.DirectoryServices.Protocols.SearchRequest
MethodTable: 14b394c4
EEClass: 14d97ce0
Size: 52(0x34) bytes
GC Generation: 0
(C:\WINDOWS\assembly\GAC_MSIL\System.DirectoryServices.Protocols\2.0.0.0__b03f5f7f11d50a3a\System.DirectoryServices.Protocols.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
02c39310 4000102        4        System.String 0 instance 00000000 directoryRequestID
14b398bc 4000103        8 ...ControlCollection 0 instance 27246e90 directoryControlCollection
02c39310 4000111        c        System.String 0 instance 27246d00 dn
12579f5c 4000112       10 ....StringCollection 0 instance 27246eb4 directoryAttributes
02c36ca0 4000113       14        System.Object 0 instance 27246ddc directoryFilter
14b39344 4000114       18         System.Int32 1 instance        1 directoryScope
14b393fc 4000115       1c         System.Int32 1 instance        0 directoryRefAlias
0fd3da00 4000116       20         System.Int32 1 instance        0 directorySizeLimit
1202af88 4000117       28      System.TimeSpan 1 instance 27246e60 directoryTimeLimit
120261c8 4000118       24       System.Boolean 1 instance        0 directoryTypesOnly

Okay, so what's this? Well, it's a System.DirectoryServices.Protocols.SearchRequest object. This means that it has various properties defined by the System.DirectoryServices.Protocols.SearchRequest class. If you want to know more about these properties I suggest you look up the SearchRequest class in msdn. We have the RequestId, the Scope, the DistinguishedName, etc.

So, let's say we want to know what the distinguished name is for this particular request. We boldly assume that the dn-property we see in the listing above is what is called DistinguishedName by MSDN. Simply copy the address of the dn-property (27246d00) and use !dumpobject again. We can see that the property is a System.String, so the output should be pretty clear.

0:050> !do 27246d00
Name: System.String
MethodTable: 02c39310
EEClass: 0fb610ac
Size: 112(0x70) bytes
GC Generation: 0
(C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
String: CN=Dummy,CN=Accounts,CN=useradm,DC=dummy,DC=net
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
0fd3da00 4000096        4         System.Int32 1 instance       48 m_arrayLength
0fd3da00 4000097        8         System.Int32 1 instance       47 m_stringLength
0fb80010 4000098        c          System.Char 1 instance       43 m_firstChar
02c39310 4000099       10        System.String 0   shared   static Empty
    >> Domain:Value 0019daf0:03380310 11b42540:03380310 <<
0fb86d44 400009a       14        System.Char[] 0   shared   static WhitespaceChars
    >> Domain:Value 0019daf0:03380324 11b42540:033855bc <<

Apparently, the distinguished name used was "CN=Dummy,CN=Accounts,CN=useradm,DC=dummy,DC=net". If we want to find out more we simply continue using !dumpobject to examine the objects and their respective values.

In my next post I thought I'd continue using the !dumpobject-command to probe through the w3wp-process, and introduce a couple of other great commands.

To be continued...

/ Johan

How to install Windbg and get your first memory dump

JohanS — Thu, 11 Jan 2007 19:42:00 GMT

Install Windbg

Windbg is the tool for the ASP.NET support engineer. It is free and it's available at www.microsoft.com/whdc/devtools/debugging/default.mspx. The learning curve is steep to say the least, but if you're interested in finding out what is going on behind the scenes in your application, then Windbg is your new best friend. For information on how to configure windbg, please refer to the documentation. Pay special attention to the section concerning symbols.

There is an extension called SOS.dll that you will want to use. You'll find it in the framework directory so for Framework 2.0 look in "C:\Windows\Microsoft.NET\Framework\v2.0.50727". You might want to copy it into the same folder as windbg for easy access.

Get a memory dump

Windbg will allow you to either perform a post mortem analysis on a memory dump or to attach to a process during execution. I mainly deal with memory dumps, since it's a lot easier to request a single file from a customer rather than access to their server. Maybe I'll cover live debugging in another post, but for now we'll just look at dump files.

Vista

If you're running Windows Vista, then you can easily create a dump file from the task manager. Simply open up the "Processes"-tab, right-click the process you wish to dump and select "Create Dump File".

Adplus

For any other system or if you want to specify certain conditions I'd recommend using a script called adplus. It comes with the Windbg installation and is run from the command prompt. Adplus will take a number of arguments, but for basic operation there are two things you need to specify:

When to take the dump
The name or process ID of the process you wish to take a dump of

The dumps generated by adplus will be saved to a subfolder of the folder where you've installed windbg.

For example:

adplus -crash -pn w3wp.exe

This will generate a full memory dump right before any process named w3wp.exe terminates or recycles. This will also generate minidumps on all first chance exceptions.

adplus -crash -pn w3wp.exe -NoDumpOnFirst

Same as above, but without the minidumps.

adplus -hang -p 2960

This will immediately get a full dump of the process with ID 2960. Commonly used when the process has hung, or is generally unresponsive. Hence the name.

Advanced Adplus

If you're trying to pin down the cause of a specific exception, then you can use a config file. This is a sample config file that will create a full memory dump once a System.Runtime.InteropServices.COMException occurs. Simply copy the code below into notepad and save it as MyConfig.cfg.

<ADPlus>
<Settings>
<RunMode> CRASH </RunMode>
</Settings> <PreCommands>
<Cmd> !load clr10\sos</Cmd>
</PreCommands> <Exceptions>
<Option> NoDumpOnFirstChance </Option>
<Option> NoDumpOnSecondChance </Option>
<Config>
   
   <Code> clr </Code>
   <Actions1> Log </Actions1>
   <CustomActions1> !clr10\sos.cce System.Runtime.InteropServices.COMException 1; j ($t1 = 1) '.dump /ma /u c:\dumps\exceptiondump.dmp;gn' ; 'gn' </CustomActions1>
   <ReturnAction1> GN </ReturnAction1>
   <Actions2> Void </Actions2>
   <ReturnAction2> GN </ReturnAction2>
</Config>
</Exceptions>
</ADPlus>

As you can see you can easily adjust the config so that it gets a dump on any other exception. The dump will be saved in c:\dumps, so you should also make sure that this folder exists. When you're ready, run adplus with the following syntax:

adplus -c myconfig.cfg -pn w3wp.exe

Debugging through a Terminal Server session

If you don't have direct access to the server you need to either attach noninvasively or schedule the command you wish to run. This can seem a bit complicated, but there's a pretty good howto written in the knowledge base under the following article: http://support.microsoft.com/default.aspx/kb/323478

Well I guess that's all for now.

/ Johan