Speaking of which... : ASP.NET

Investigating Locks

JohanS — Fri, 09 Oct 2009 16:49:56 GMT

Consider the following scenario:

You have an ASP.NET application which intermittently responds sluggishly. As the problem occurs memory usage is about average, as is CPU usage, but still certain pages respond slower and slower. The machine acts just as if it is under heavy load, but judging from the CPU it isn't. In fact CPU usage might even decrease rather than increase. When trying to access non .NET content such as plain text/html or images you usually find that response times are just as fast as ever.

When you find that your application has hung in a low CPU state it is usually because of one of the following reasons (in order of likelihood based on my experience) :

External resources responding slowly. E.g. a database query or AD lookup is taking a long time to complete and so the application is simply waiting for another server to respond.
One of the threads of the application has entered a critical section and now several other threads are waiting to enter the same critical section.
A classic deadlock. Thread A waits for thread B which waits for thread C which waits for thread A.

Quite often you'll see a mixture of items 1 and 2. A thread enters a critical section, calls a remote database, or similar, and all other threads attempting to enter the same critical section will have to nicely stay in line until the struggling database has responded. Still, it is quite possible that the critical section is waiting for a local event to finish, but then you'd usually see a higher CPU load on the machine. (Though not necessarily in the w3wp.exe process.)

What are Critical Sections?

In case you've forgotten or simply not worked that much with critical sections it might be necessary to refresh your memory on this.

A Critical Section is a segment of code that you do not want to be executed more than one at a time. In order to ensure this you use the lock keyword to ensure that a certain block of code is able to complete with no interruption from any other threads. To illustrate this I've created the following code sample:

public class FileIO
{
    private static object myLock = new object();

	public FileIO()
	{
	}

    public string performLockOperation()
    {
        string strResult = "The Lock Operation started on " + System.DateTime.Now.ToLongTimeString();
        lock (myLock)
        {
            // Let the thread sleep for 10 seconds
            // This way we simulate a File IO operation that takes some time to complete.
            System.Threading.Thread.Sleep(10000);
        }
        strResult += " and ended on " + System.DateTime.Now.ToLongTimeString();
        return strResult;
    }

    public string performNoLockOperation()
    {
        string strResult = "The No Lock Operation started on " + System.DateTime.Now.ToLongTimeString();
        System.Threading.Thread.Sleep(10000);
        strResult += " and ended on " + System.DateTime.Now.ToLongTimeString();
        return strResult;
    }

}

Let's take a quick look at the parts of this class that I've written, called FileIO:

myLock

The class has a static variable called myLock that I will be using to identify the lock I'm about to create. This is why the variable is static. Otherwise I'd be creating new locks for each instance of the FileIO class leaving the lock operation completely redundant. Now that I use a static variable the lock spans all instances of the FileIO class.

constructor

The constructor is empty. Move along. Nothing to see here.

performLockOperation

This is the most interesting part of the class. It is the function we'll be using to actually reproduce the potential problems. The function returns a string which will contain information about when the call to the function was made and when it completed. We store this data in a string called strResult which we will eventually be returning as the result of the function call.

Having declared this variable and put the initial time of the function call in it we then create our lock. This is simply done with the following syntax:

lock (object)
{
}

Any code within this block will first make sure that there is not already a lock taken on this object. If there is, then they will pause execution and wait for the lock to become available before continuing execution.

Inside our lock we then simulate a slow File IO operation by letting the thread sleep for 10 seconds.

After that we append the current time to the result and return it.

performNoLockOperation

This function is identical to the previous one, except this time we do not have a critical section. This function is included just for comparison.

The potential problem

Now, let's get down to business. If we have a web page that makes a call to the performLockOperation-function and that page is requested at the same time by four different clients. What would be the response times?

The answer is 10, 20, 30 & 40 seconds.

Here's why:

The thread handling the first request enters the critical section, effectively locking it for everyone else. When it is complete 10 seconds have passed due to the Sleep-call.
One of the other threads enters the critical section once the first request is finished. It has already waited for 10 seconds, and the thread will now sleep for an additional 10 seconds.
As the second thread finishes one of the remaining two enters the critical section. It has now waited for 20 seconds and will sleep for an additional 10.
The final request is able to enter the critical section after having waited for 30 seconds. Once it is complete 40 seconds have passed.

If we'd been using the performNoLockOperation instead then all requests would have responded after 10 seconds.

Now, I know that 10 seconds is an exceptionally long time to let the thread sleep, but this was all for illustration purposes. Imagine instead that the sleeping time was 1 second and the number of requests was 200...

Troubleshooting this issue

Okay, so what would it look like when this problem occurs? Let's have a look. If you haven't looked at my posts on getting started with Windbg, then I suggest reading them before continuing. (Part I & Part II)

First of all I fire up WinDbg, then I attach to the w3wp.exe. I then use "~*e!clrstack" to look at the managed threads of the w3wp.exe application. Here is a snippet of what I get:

OS Thread Id: 0xce0 (24)
Child-SP         RetAddr          Call Site
000000000447e0f0 000007ff001e13d2 FileIO.performLockOperation()
000000000447e170 000007feee323ec9 _Default.Page_Load(System.Object, System.EventArgs)
000000000447e1b0 000007fee7956aea System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr, System.Object, System.Object, System.EventArgs)
000000000447e1e0 000007fee794d2c4 System.Web.Util.CalliEventHandlerDelegateProxy.Callback(System.Object, System.EventArgs)
000000000447e210 000007fee794d322 System.Web.UI.Control.OnLoad(System.EventArgs)
000000000447e250 000007fee79498ac System.Web.UI.Control.LoadRecursive()
000000000447e2a0 000007fee7948db0 System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)
000000000447e370 000007fee7948cdb System.Web.UI.Page.ProcessRequest(Boolean, Boolean)
000000000447e3d0 000007fee7948c70 System.Web.UI.Page.ProcessRequest()
000000000447e430 000007ff001e0c09 System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)
000000000447e490 000007fee7950117 ASP.default_aspx.ProcessRequest(System.Web.HttpContext)
000000000447e4c0 000007fee791449b System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
000000000447e570 000007fee7ffbd41 System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)
000000000447e610 000007fee7fed132 System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)
000000000447e7a0 000007fee7fcf599 System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)
000000000447e7f0 000007fee80f5344 System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)
000000000447e910 000007fee80f65bb System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
000000000447ea80 000007fee80f4994 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
000000000447eae0 000007fef72701ea DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)
000000000447f310 000007fee80f5424 DomainNeutralILStubClass.IL_STUB(IntPtr, System.Web.RequestNotificationStatus ByRef)
000000000447f3f0 000007fee80f65bb System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
000000000447f560 000007fee80f4994 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
000000000447f5c0 000007fef727043b DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)
OS Thread Id: 0x15ec (25)
Unable to walk the managed stack. The current thread is likely not a 
managed thread. You can run !threads to get a list of managed threads in
the process
OS Thread Id: 0x1958 (26)
Child-SP         RetAddr          Call Site
0000000004c1dd10 000007ff001e13d2 FileIO.performLockOperation()
0000000004c1dd90 000007feee323ec9 _Default.Page_Load(System.Object, System.EventArgs)
0000000004c1ddd0 000007fee7956aea System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr, System.Object, System.Object, System.EventArgs)
0000000004c1de00 000007fee794d2c4 System.Web.Util.CalliEventHandlerDelegateProxy.Callback(System.Object, System.EventArgs)
0000000004c1de30 000007fee794d322 System.Web.UI.Control.OnLoad(System.EventArgs)
0000000004c1de70 000007fee79498ac System.Web.UI.Control.LoadRecursive()
0000000004c1dec0 000007fee7948db0 System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)
0000000004c1df90 000007fee7948cdb System.Web.UI.Page.ProcessRequest(Boolean, Boolean)
0000000004c1dff0 000007fee7948c70 System.Web.UI.Page.ProcessRequest()
0000000004c1e050 000007ff001e0c09 System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)
0000000004c1e0b0 000007fee7950117 ASP.default_aspx.ProcessRequest(System.Web.HttpContext)
0000000004c1e0e0 000007fee791449b System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
0000000004c1e190 000007fee7ffbd41 System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)
0000000004c1e230 000007fee7fed132 System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)
0000000004c1e3c0 000007fee7fcf599 System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)
0000000004c1e410 000007fee80f5344 System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)
0000000004c1e530 000007fee80f65bb System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
0000000004c1e6a0 000007fee80f4994 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
0000000004c1e700 000007fef72701ea DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)
0000000004c1ef30 000007fee80f5424 DomainNeutralILStubClass.IL_STUB(IntPtr, System.Web.RequestNotificationStatus ByRef)
0000000004c1f010 000007fee80f65bb System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
0000000004c1f180 000007fee80f4994 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
0000000004c1f1e0 000007fef727043b DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)
OS Thread Id: 0x1e28 (27)
Child-SP         RetAddr          Call Site
000000000492dcf0 000007ff001e13d2 FileIO.performLockOperation()
000000000492dd70 000007feee323ec9 _Default.Page_Load(System.Object, System.EventArgs)
000000000492ddb0 000007fee7956aea System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr, System.Object, System.Object, System.EventArgs)
000000000492dde0 000007fee794d2c4 System.Web.Util.CalliEventHandlerDelegateProxy.Callback(System.Object, System.EventArgs)
000000000492de10 000007fee794d322 System.Web.UI.Control.OnLoad(System.EventArgs)
000000000492de50 000007fee79498ac System.Web.UI.Control.LoadRecursive()
000000000492dea0 000007fee7948db0 System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)
000000000492df70 000007fee7948cdb System.Web.UI.Page.ProcessRequest(Boolean, Boolean)
000000000492dfd0 000007fee7948c70 System.Web.UI.Page.ProcessRequest()
000000000492e030 000007ff001e0c09 System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)
000000000492e090 000007fee7950117 ASP.default_aspx.ProcessRequest(System.Web.HttpContext)
000000000492e0c0 000007fee791449b System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
000000000492e170 000007fee7ffbd41 System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)
000000000492e210 000007fee7fed132 System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)
000000000492e3a0 000007fee7fcf599 System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)
000000000492e3f0 000007fee80f5344 System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)
000000000492e510 000007fee80f65bb System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
000000000492e680 000007fee80f4994 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
000000000492e6e0 000007fef72701ea DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)
000000000492ef10 000007fee80f5424 DomainNeutralILStubClass.IL_STUB(IntPtr, System.Web.RequestNotificationStatus ByRef)
000000000492eff0 000007fee80f65bb System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
000000000492f160 000007fee80f4994 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
000000000492f1c0 000007fef727043b DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)
OS Thread Id: 0x1c38 (28)
Child-SP         RetAddr          Call Site
0000000004d0e0b0 000007ff001e13d2 FileIO.performLockOperation()
0000000004d0e130 000007feee323ec9 _Default.Page_Load(System.Object, System.EventArgs)
0000000004d0e170 000007fee7956aea System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr, System.Object, System.Object, System.EventArgs)
0000000004d0e1a0 000007fee794d2c4 System.Web.Util.CalliEventHandlerDelegateProxy.Callback(System.Object, System.EventArgs)
0000000004d0e1d0 000007fee794d322 System.Web.UI.Control.OnLoad(System.EventArgs)
0000000004d0e210 000007fee79498ac System.Web.UI.Control.LoadRecursive()
0000000004d0e260 000007fee7948db0 System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)
0000000004d0e330 000007fee7948cdb System.Web.UI.Page.ProcessRequest(Boolean, Boolean)
0000000004d0e390 000007fee7948c70 System.Web.UI.Page.ProcessRequest()
0000000004d0e3f0 000007ff001e0c09 System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)
0000000004d0e450 000007fee7950117 ASP.default_aspx.ProcessRequest(System.Web.HttpContext)
0000000004d0e480 000007fee791449b System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
0000000004d0e530 000007fee7ffbd41 System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)
0000000004d0e5d0 000007fee7fed132 System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)
0000000004d0e760 000007fee7fcf599 System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)
0000000004d0e7b0 000007fee80f5344 System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)
0000000004d0e8d0 000007fee80f65bb System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
0000000004d0ea40 000007fee80f4994 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
0000000004d0eaa0 000007fef72701ea DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)
0000000004d0f2d0 000007fee80f5424 DomainNeutralILStubClass.IL_STUB(IntPtr, System.Web.RequestNotificationStatus ByRef)
0000000004d0f3b0 000007fee80f65bb System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
0000000004d0f520 000007fee80f4994 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
0000000004d0f580 000007fef727043b DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32

So, what does this tell us?

Well, obviously we have four threads that call my function FileIO.performLockOperation() from Page_Load.

Please note that we don't have any additional callstack, and since the performLockOperation-function was named by me it could just as well have been named "foo". In order to find out that these threads are actually waiting for a lock we need to look at the native callstack using the kb-command. Let's take a look at thread 24:

0:029> ~24kb
ntdll!NtDelayExecution+0xa
kernel32!SleepEx+0x84
mscorwks!EESleepEx+0x2d
mscorwks!Thread::UserSleep+0x71
mscorwks!ThreadNative::Sleep+0xf9
App_Code_tvh3usvo!FileIO.performLockOperation()+0x8a
App_Web_uywrbugj!_Default.Page_Load(System.Object, System.EventArgs)+0x32
System_Web_RegularExpressions_ni!System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr, System.Object, System.Object, System.EventArgs)+0x19
System_Web_ni!System.Web.Util.CalliEventHandlerDelegateProxy.Callback(System.Object, System.EventArgs)+0x2a
System_Web_ni!System.Web.UI.Control.OnLoad(System.EventArgs)+0x84
System_Web_ni!System.Web.UI.Control.LoadRecursive()+0x42
System_Web_ni!System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)+0x97c
System_Web_ni!System.Web.UI.Page.ProcessRequest(Boolean, Boolean)+0xa0
System_Web_ni!System.Web.UI.Page.ProcessRequest()+0x5b
System_Web_ni!System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)+0xf0
App_Web_uywrbugj!ASP.default_aspx.ProcessRequest(System.Web.HttpContext)+0x9
System_Web_ni!System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()+0x257
System_Web_ni!System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)+0xab
System_Web_ni!System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)+0x501
System_Web_ni!System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)+0x72

Apparently we entered FileIO.performLockOperation, and then called sleep, (which is what we're doing in this repro in order to simulate a slow IO operation.) So, what do the other threads look like? Here's thread 26:

0:029> ~26kb
ntdll!ZwWaitForMultipleObjects+0xa
kernel32!WaitForMultipleObjectsEx+0x10b
mscorwks!WaitForMultipleObjectsEx_SO_TOLERANT+0xc1
mscorwks!Thread::DoAppropriateAptStateWait+0x41
mscorwks!Thread::DoAppropriateWaitWorker+0x191
mscorwks!Thread::DoAppropriateWait+0x5c
mscorwks!CLREvent::WaitEx+0xbe
mscorwks!AwareLock::EnterEpilog+0xc9
mscorwks!AwareLock::Enter+0x72
mscorwks!AwareLock::Contention+0x1fb
mscorwks!JITutil_MonContention+0xdf
App_Code_tvh3usvo!FileIO.performLockOperation()+0x7f
App_Web_uywrbugj!_Default.Page_Load(System.Object, System.EventArgs)+0x32
System_Web_RegularExpressions_ni!System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr, System.Object, System.Object, System.EventArgs)+0x19
System_Web_ni!System.Web.Util.CalliEventHandlerDelegateProxy.Callback(System.Object, System.EventArgs)+0x2a
System_Web_ni!System.Web.UI.Control.OnLoad(System.EventArgs)+0x84
System_Web_ni!System.Web.UI.Control.LoadRecursive()+0x42
System_Web_ni!System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)+0x97c
System_Web_ni!System.Web.UI.Page.ProcessRequest(Boolean, Boolean)+0xa0
System_Web_ni!System.Web.UI.Page.ProcessRequest()+0x5b

This is one of the threads waiting for the critical section. It then found that it was unable to get a lock and so began waiting for the critical section on thread 24 to end.

Resolving the problem

This is the tricky part. There is no quick-fix for this. We can use the debugger to identify the bottleneck, but after that we'll need to review our design in order to get this working smoothly. A couple of generic things to try would be the following:

Review what you're doing inside the critical section. Is it really necessary to perform all of the operations you're doing, or would it be possible to do some/all of them outside the scope of the critical section?
Identify the reason for the lock taking so much time. Can it be resolved by load-balancing or clustering the resource we're waiting for?
If we're waiting for another server, make sure the physical infrastructure is okay. Are network response times within the expected parameters or do you need to fix your wiring, router, etc?
Are all calls to the critical section taking the same amount of time, or are there certain calls that take an exceptional amount of time? Perhaps most calls to the function are just fine, but intermittently a call comes along that tries to get everything from the AD, database, etc. In order to investigate this I'd use the commands outlined in Getting started with windbg -part II. Identify the thread owning the lock and investigate the callstack and stack objects using !clrstack and !dso. Use !dso and !do to query the stack objects and try to find the SQL query or AD query that is causing the thread to take such a long time to finish. Is the query significantly different from the ones waiting to execute, then this is most likely our culprit.

Post Script

A quick thought on locks and their limits:

Locks do not scale well. A lock can only be made within a specific process, so if you're using a web farm, or in any other way more than one w3wp.exe, then you will not be able to lock all processes. In other words. Server 1 can get a lock, and so can Server 2. Both will happily execute their code independent of each other. For products such as SQL server you'd deal with this in your SQL query, but if you're performing other operations, such as File IO you might have to work around this by using the same approach as Ms Word. Create a lock file in a shared space, and before you enter your critical section you simply check if the file exists:

You are entering your critical section. Look for lock file in shared location
If file exists, then wait
If/when file doesn't/no longer exist, create file and enter critical section
Make sure you have solid exception handling within the critical section so that you under no circumstances leave the critical section without removing the lock file.
When done, remove lock file

/ Johan

Office Automation

JohanS — Tue, 14 Oct 2008 16:15:30 GMT

A very common scenario that keeps sprouting new heads like a hydra is Office Automation. Let me start by saying that this is not supported.

There is a KB-article number 257757 discusses this, and clearly states the following:

Microsoft does not currently recommend, and does not support, Automation of Microsoft Office applications from any unattended, non-interactive client application or component (including ASP, ASP.NET, DCOM, and NT Services), because Office may exhibit unstable behavior and/or deadlock when Office is run in this environment.

Why is this not supported?

The crucial thing to consider is the fact that Microsoft Office is designed to be an end-user, single client product. Automating Microsoft Office in a client application, using the identity and security context of the logged on client is supported, but unattended execution is not.

There are a number of things that can go wrong and article 257757 lists most of them. For arguments sake, let's consider the following scenario:

Your server application starts up and uses CreateObject to create an instance of MS Word. Word tries to read the settings of the current client and since your application is using the Network Service account this presents an immediate problem. There are, for example, certain methods that rely on a default printer being installed. All this can lead to serious problems. You then need to monitor the server, since Office might show a modal dialogue for some reason. The "install on first use" feature of MSI might also kick in, prompting the client to install additional features. All this would hang the current thread. Also, running one instance of Word, might be okay, but what happens when you get 100 more or less concurrent requests and each request starts up an instance of Word? MS Word is not a server, it's an excellent piece of single-client software.

So what should I do instead?

Actually, most of your problems can be resolved using HTML. I've seen applications where lines and lines of complex code were used for generating an Excel document, when it could just as well have been made using a standard HTML-table. Excel would have no problems whatsoever reading a table like the one below:

<table>
    <tr>
        <td><b>Person</b></td>
        <td><b>Age</b></td>
    </tr>
    <tr>
        <td>Pete</td>
        <td>30</td>
    </tr>
    <tr>
        <td>Claire</td>
        <td>40</td>
    </tr>
    <tr>
        <td>Average age</td>
        <td>=AVERAGE(B2:B3)</td>
    </tr>
</table>

As you can see all formatting, and even the functions, would be interpreted correctly.

Obviously Word is great with HTML as well, so most formatting issues can quite easily be resolved using this approach.

So how do I do this?

The solution is quite easy. In order to generate an Excel document all you need to do is to create a table with the data you want, and add the following two lines to the Page_Load event of your page:

Response.ContentType = "application/vnd.ms-excel";
Response.AddHeader("Content-Disposition", "attachment; filename=Data.xls;");

The first line, sets the returned ContentType to Excel, and the second changes the filename of the returned file. This means that the client will be prompted to Open/Save the file, and the filename will be set to "Data.xls" rather than "Default.aspx" (or whatever your original document may be called.)

If you wanted to return a word document you'd set the ContentType to "application/msword" instead.

One little thing to consider is the fact that by default you're probably adding a lot of redundant information to your webpage. You might want to remove all excessive HTML from the page so that you remove all unnecessary headers, stylesheets, viewstate, etc. For an example, please consider the code below:

Response.Clear();
Response.Buffer = true;
Response.ContentType = "application/vnd.ms-excel";
Response.AddHeader("Content-Disposition", "attachment; filename=Data.xls;");
this.EnableViewState = false;
System.IO.StringWriter oSw = new System.IO.StringWriter();
System.Web.UI.HtmlTextWriter oHtml = new System.Web.UI.HtmlTextWriter(oSw);
Table oTable = new Table();
for (int ctr = 1; ctr <= 10; ctr++)
{
    TableRow oRow = new TableRow();
    TableCell oCell1 = new TableCell();
    oCell1.Text = ctr.ToString();
    TableCell oCell2 = new TableCell();
    oCell2.Text = (ctr*10).ToString();
    oRow.Cells.Add(oCell1);
    oRow.Cells.Add(oCell2);
    oTable.Rows.Add(oRow);
}
oTable.RenderControl(oHtml);
Response.Write(oSw.ToString());
Response.End();

Security issues in Office 2007

If you're running Office 2007 you might come across the following error message:

"The file you are trying to open, 'Data.xls', is in a different format than specified by the file extension. Verify that the file is not corrupted and is from a trusted source before opening the file. Do you want to open the file now?

This is completely by design. In this example Excel has noticed that though the file is named .xls it does in fact contain html, so it's warning us about this inconsistency. We might bypass this by actually changing the file extension to htm or, perhaps saving the data as comma separated values (csv) which Excel also supports, but both options would be opened directly in the browser rather than passed on to Excel, so this is a design consideration we have to take into account.

There is really no way of bypassing this security feature other than manually disabling it, client-side, using a registry hack. (The key is HKCU\Software\Microsoft\Office\12.0\Excel\Security\ Add a DWORD named xtensionHardening and set it to 0.) For obvious reasons this is not something that is generally recommended.

Other options

There are also third-party products available for generating MS Office-compatible documents, but there are also some articles in the knowledge base on the subject. For example this little gem which can easily be adopted to ASP.NET as well:

How to use ASP to generate a Rich Text Format (RTF) document to stream to Microsoft Word

/ Johan

Why doesn’t the GC kick in when the worker process is inactive?

JohanS — Tue, 13 May 2008 16:39:48 GMT

I got the following question in my Getting started with windbg – post and I thought it might be worth posting the replies in a separate article:

Hi Johan,

about those threads with an ID of XXXX, should they go away after certain amount of idle time like 2 minutes? I am trouble shooting an application while the application should sit idle (because of no stimulation), however it is still using quite some CPU time and and threadpool threads, where it should not. So I created an adplus dump when it is sitting idle, and I found that there are many threads with ID XXXX, and I created a dump file again after 15 minutes, again, it still have the exact same XXXX threads. All those threads are completion port threads, and I wonder why they did not get recycled?

Also, when doing a '~*e !clrstack', most of the worker threads and completion port threads are showing "Failed to start stack walk: 80004005". Is there a way to show the stack for those threads, because those are the threads I am interested in.

Thank you!

The GC will be triggered when you’re allocating memory. If your application is inactive, then so is the Garbage Collector. Even if you’re making a few, random allocations they still may not be enough to trigger a GC, so this is not at all unexpected.

The “Failed to start stack walk: 80004005”-error is displayed when the thread did contain a managed stack, but no longer does.

/ Johan

What to do about the slow startup of web services

JohanS — Wed, 02 Apr 2008 18:05:16 GMT

Due to the architecture of web services and web applications they can be quite slow to start. For example on my Windows 2003-box the initial localhost-call to a simple “Hello World!”-web service takes approximately 8 seconds, while the next request is more or less immediate.

Why is this?

This isn’t news, really. One of the first things we learned in the early beta stages of ASP.NET was that the first request would take a little extra time due to the Just In Time Compilation (JIT). This was a major change from the classic ASP architecture where everything was interpreted rather than compiled. Upon the first request the .aspx / .asmx file will be compiled (JIT’ed) into Microsoft Immediate Language (MSIL) and the resulting .dll will be moved to it’s proper location.

This compilation will occur every time the application pool starts causing a significantly longer response time on the first request compared to the following requests.

Run-time performance over startup performance

It all comes down to prioritizing run-time performance over a quick startup. Which, in my humble opinion, is a sound choice. There are, however, situations where you may feel that this is to your disadvantage. The other day I got a question from a customer who had a web service that was called quite infrequently and to him this meant that with every other request response times would be horrendous, simply because the application got recompiled.

For each case I get I usually do a quick search on the problem description. I don’t really expect to find the solution this way, since I can only assume that the customer has tried this as well. Instead it will likely present me with a list of troubleshooting steps that the customer has already tried as well as possible solutions that didn’t fit. (Off course I still need to verify this with the customer).

This time I didn’t really need to do this since I was pretty sure what the cause was. Still, I was curious to see how common this problem was. I honestly didn’t think this was a very common scenario, but my quick search on the web revealed the contrary. The few hits that I quickly browsed through suggested writing an application that would ping the web service in order to keep it alive. I must admit I find this quite inventive, but I really don’t think it’s the best approach. :)

What to do

So, what is to be done about this? Well, I’ve already mentioned that there are better solutions than pinging the web service. As I told my customer there are basically two good options:

Precompile the assemblies
Change the settings for automatic recycling on the worker process

Precompiling the assemblies

There is a nice little tool called Native Image Generator. (ngen.exe) It will compile the assembly into native images and installs them into the native image cache. The syntax is pretty straight-forward: ngen.exe install [assembly] The problem is that once you’ve changed the assembly you’d have to precompile it again making updates a bit tedious.

There is also a service-version of the ngen called the Native Image Service. More information about ngen can be found at http://msdn2.microsoft.com/sv-se/magazine/cc163808(en-us).aspx

Changing the recycling settings

The reason why my customer was experiencing this problem was because his application pool would time out and recycle itself due to the default settings in the IIS manager. By default the worker process will recycle after 20 minutes of inactivity, so if no one has pinged your application in that time the next request will cause the application to recompile, resulting in a longer response time. By tweaking this setting in the IIS manager to something more suitable to the current rate of requests, like 40 minutes or maybe even 60 he should also be able to find a nice balance. Actually, considering the low load it would probably be a good idea to turn this setting off completely. Here’s how:

Open up IIS Manager
Locate the Application Pool in question
Right-click it and select “Properties”
Go to the “Performance”-tab
Change, or disable the “Idle timeout” setting

/ Johan

Using IE6 With Visual Studio 2008

JohanS — Wed, 26 Mar 2008 19:04:08 GMT

Here's a little scenario I came across the other day. I've forwarded the information to development, so it's pending further investigation. I still thought it would be a good idea to publish the scenario though.

Visual Studio 2008 + FTP = Possible trouble

It seems like there might be a bug in Visual Studio 2008 when it comes to projects that you access via FTP. I don't know how common it is to use FTP to access & deploy your projects. It would be really interesting to see some genuine statistics, but since FTP is highly insecure I guess it is very rarely used. This, I guess, is probably the reason why this little glitch slipped through in the first place.

File change notification works fine...

Consider the following scenario:

You're working on a web project using FTP
You've downloaded a file to your local cache and you're doing some changes.
Another client changes the file
You attempt to save the file, but the original file on the server has changed so it no longer matches the one in your local cache.

This causes a warning to pop up with the following information:

A more recent version of the file [File Name] has been saved to the web on '[Date]' . Do you want to replace the server file with your local file?

...overwriting the file doesn't

Clicking Yes you'd now expect Visual Studio to simply overwrite the file on the server. Unfortunately this is where the glitch rears it's ugly head. Instead of overwriting the file it gives you the following error message:

Cannot save the file [File Name] to the Webserver. The file [File Name] has been modified by (unknown) on [Date] [Time Zone].

Continuous attempts at saving will have no effect. The error pattern will repeat, displaying the two dialogue boxes over and over again.

Alternative solution

My customer originally fixed this by selecting everything in the editor, copying it to Notepad, canceling the operation in Visual Studio, reopening the connection, pasting the contents from Notepad into the document and saving. Fortunately there's an easier way to resolve it.

Simply click the Refresh-button in the Project Explorer window. Visual Studio will then ask you if you wish to update your cached version of the files you've edited. This may sound like you're going to perform a revert-to-save operation on the pages you've edited, undoing all the changes, but it's actually not the case. Visual Studio will simply refresh the files in the cache, not the ones in the editor. When Visual Studio has finished refreshing the cache it will notice the inconsistency between the files asking you if you want to update the file in the editor as well. At this point you should answer "No", unless you want all your changes undone.

My thoughts about this

Obviously this shouldn't happen. You shouldn't have to refresh the cache in order to overwrite the files. Still, as I mentioned before I believe there are two reasons why this problem managed to slip through the cracks.

FTP is unsafe. Even the password is sent as clear text and can be intercepted by anyone. For this reason I would assume that FTP is quite rarely used.
In order for the problem to occur you need to trigger a file change notification on the file in question. For normal scenarios this would mean that another client accesses the FTP-site at the same time and changes the file as you're working on it. This type of on-the fly editing without any form of source-control would be highly risky in a production environment and you'd most likely use some other means of connection while in development.

Visual Studio 2008 + FTP + Internet Explorer 6 = More trouble

Unfortunately the problem doesn't end there.

It turns out that if you are using Visual Studio 2008, are working on a web project using FTP and have IE6 on your machine, then IE will act as if a file change notification has occurred after 90 seconds wether this is true or not. This means that after 90 seconds all your attempts to save will trigger the behavior described above. The refresh cache-approach still works, but it will quickly become quite tedious.

According to the prerequisites for Visual Studio 2008, Internet Explorer 6 is a minimum requirement, so there is nothing documented on IE7 being a necessity in order to run Visual Studio 2008.

Alternative solution

So far I only know of one way to resolve this. Install IE7. You'll still encounter the first potential problem if you have more than one client working simultaneously on the same application, but as I've already mentioned, this sounds like a very unlikely scenario.

/ Johan

Monitoring Application Pool and Application Restarts

JohanS — Mon, 18 Feb 2008 17:12:00 GMT

This morning I found the following in my inbox:

I had set my web servers running on IIS 6 to recycle if they hit 700 MB (Maximum used Memory).
Can I run a report to know how many times per day or week the W3WP got recycled.
Any suggestions please.

Thanks in anticipation

The quick answer to your question is to use performance monitor:

Start performance monitor
Add a new counter
Performance object: "ASP.NET v[n]"
Counter: "Worker Process Restarts"

So, there's the answer. But while we're on the subject I'd like to mention application restarts as well.

Applications vs. Application pools

Application restarts may lead you think that your entire application pool recycled, while in fact it was only one of the applications hosted by the application pool that restarted. If this is getting confusing, then please remember that there's a difference between application pools and application. An application pool will consist of one or several worker processes and may host one or several applications.

There are two easy ways of monitoring application restarts:

Using performance monitor again
Have ASP.NET write an event to the event log upon shutdown & startup. (Requires ASP.NET 2.0 or later)

Performance monitor

This is probably the easiest, though you won't get too much additional information other than the fact that an application restart occurred. Here's what you do:

Start performance monitor
Add a new counter
Performance object: "ASP.NET v[n]"
Counter: "Application Restarts"

Event log

By altering the root web.config you can get an event for every application shutdown and start up. This is a good way to get more detailed information on why the application shut down.

Open up the root web.config, (located in the %WinDir%\Microsoft.NET\Framework\v2.0.50727\CONFIG directory,) locate the healthMonitoring.rules subkey and add the following:

Now when the application exited for an application-specific reason you'll get an event like this:

Event code: 1002 
Event message: Application is shutting down. Reason: Configuration changed. 
Event time: 2/14/2008 10:00:41 AM 
Event time (UTC): 2/14/2008 9:00:41 AM 
Event ID: a1314c10a0c84222ae2d870d85308304 
Event sequence: 18 
Event occurrence: 1 
Event detail code: 50004 
 
Application information: 
    Application domain: /LM/w3svc/1/ROOT/Test-1-128474532435626182 
    Trust level: Full 
    Application Virtual Path: /Test
    Application Path: c:\inetpub\wwwroot\Test\ 
    Machine name: JOHAN

As you can see the application shut down because the configuration changed. Note that you won't get an event if you manually kill the entire application pool in IIS manager, or similar. One thing that you will get, however, is the following event each and every time the application starts up again:

Event code: 1001 
Event message: Application is starting. 
Event time: 2/14/2008 10:00:47 AM 
Event time (UTC): 2/14/2008 9:00:47 AM 
Event ID: 1f41fd3b17764330ac61804094b0abf0 
Event sequence: 1 
Event occurrence: 1 
Event detail code: 0 
 
Application information: 
    Application domain: /LM/w3svc/1/ROOT/Test-1-128474532435626182 
    Trust level: Full 
    Application Virtual Path: /Test
    Application Path: c:\inetpub\wwwroot\Test\ 
    Machine name: JOHAN

So even if the application shut down for a reason that didn't generate an event, (IISReset, idle server, etc.) you'll at least see that for some reason it had to start up again.

/ Johan

Did you know? - Changing ASP.NET Version restarts IIS

JohanS — Tue, 05 Feb 2008 15:09:23 GMT

Here's a little something I learned the other day.

If you go to the ASP.NET tab and change the ASP.NET version for an application pool this will not only reset the application pool, but the entire IIS. I was a bit surprised at first, but investigating the matter showed that there was a pretty solid architectural decision behind this.

As most "bugs" this turned out to be quite well documented once you know where to look. It's actually mentioned in the aspnet_regiis documentation.

The reason it is documented under aspnet_regiis is because this is also a rather good alternative if you wish to reconfigure an application pool, but not cause an IIS reset.

You do it by running the following two commands:

aspnet_regiis -s w3svc/<instance>/root -norestart
iisapp /a /r

/ Johan

Debugging School

JohanS — Fri, 01 Feb 2008 17:44:48 GMT

As you may or may not have noticed I've started organizing all posts I've made regarding debugging training in a separate list on the left hand side of the page. If you're new to debugging and don't know where to begin, then at least you know where I'd recommend you to start.

I hope you'll find it useful.

Have a great weekend! / Johan

Using WinDbg - Hunting Exceptions

JohanS — Thu, 31 Jan 2008 19:39:36 GMT

Prerequisites

This post will require some basic knowledge of windbg and the sos extension. For this I recommend looking at the following posts:

For more information on Exceptions in general and why they should be avoided I'd like to recommend this post:

Exceptions

Introduction

I thought it was time to write another post on how to use windbg for troubleshooting. A lot of my time is spent locating exceptions in various web applications, so I thought this might be a good topic to cover. I've previously written a post specifically targeting OutOfMemoryExceptions, but I thought I should broaden the terms and make it a bit more general. There are two scenarios that are exceptionally common in my line of work:

Clients are reporting 2nd chance exceptions displayed on screen with the classic "Server Error"- page.
Performance is generally bad, and when we investigate it turns out that there are tons of exceptions being thrown every second.

In this post I'll cover how to investigate what exceptions have been thrown by an application, as well as how to use windbg and adplus to automatically gather specific information for us.

Where to start

Okay, so you have a web application that you've been monitoring and you believe it is throwing a lot of exceptions. You've taken a dump of the process and you're ready to begin the investigation. Where do you start?

!dumpallexceptions (!dae)

If your application is running under the .NET Framework 1.1 you can use the !dumpallexceptions-command (!dae) to get a list of all the exceptions still on the heap. Now, remember that an exception is a managed object, so they will eventually be garbage collected just like everything else. This means that when looking at the heap for exceptions you will only get the exceptions still in memory, not every exception thrown by the application since startup.

Anyway, if you run !dae you'll get a list of exceptions that looks like this:

0:000> !dae
Going to dump the .NET Exceptions found in the heap.
Loading the heap objects into our cache.
Number of exceptions of this type:        1
Exception object: 026200ec
Exception type: System.ExecutionEngineException
Message:
InnerException:
StackTrace (generated):

StackTraceString:
HResult: 80131506
The current thread is unmanaged
-----------------

Number of exceptions of this type:        1
Exception object: 026200a4
Exception type: System.StackOverflowException
Message:
InnerException:
StackTrace (generated):

StackTraceString:
HResult: 800703e9
The current thread is unmanaged
-----------------

Number of exceptions of this type:        1
Exception object: 0262005c
Exception type: System.OutOfMemoryException
Message:
InnerException:
StackTrace (generated):

StackTraceString:
HResult: 8007000e
The current thread is unmanaged
-----------------

Number of exceptions of this type:        1
Exception object: 0b62cf38
Exception type: System.Data.SqlClient.SqlException
Message: Transaction (Process ID 96) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
InnerException:
StackTrace (generated):
    SP       IP       Function
    1DACECA8 1D0A4D4C Company.Database.AuditTrail.Write(System.String, System.Guid, System.String)
    1DACED6C 1D0A4B98 Company.Database.AuditTrail.AddEntry(EntryType, System.Guid, System.String)
    1DACED90 1D0A4B24 Company.Database.DataFunctions.SaveRow(System.String, System.Guid, Boolean)
    1DACEE20 1DDD7A06 FooBase.PersonApplicationOtherQualificationBase.PersonApplicationOtherQualificationBaseManager.BaseSave(System.Guid, Boolean)
    1DACEE34 1DDD79BF Foo.PersonApplicationOtherQualification.PersonApplicationOtherQualificationManager.Save(System.Guid, Boolean)
    1DACEE48 1DDD77F5 UserControls.PersonApplicationOtherQualificationDetail.OnSave()
    1DACEE70 1DDD2701 Company.Web.UI.Page.InvokeUsercontrolTransaction(System.Web.UI.Control, Company.Web.TransactionType)

StackTraceString:
HResult: 80131904
The current thread is unmanaged
-----------------

Number of exceptions of this type:        3
Exception object: 0b62d23c
Exception type: System.Exception
Message: Thrown while invoking Save on object PersonApplicationOtherQualificationDetail1(ASP.sys_modules_person_personapplicationotherqualificationdetail_ascx)
InnerException: System.Data.SqlClient.SqlException, use !PrintException 0b62cf38 to see more
StackTrace (generated):
    SP       IP       Function
    1DACEDF8 1DDD28C8 Company.Web.UI.Page.InvokeUsercontrolTransaction(System.Web.UI.Control, Company.Web.TransactionType)
    1DACEEC4 1DDD262D Company.Web.UI.Page.InvokeUsercontrolTransaction(System.Web.UI.Control, Company.Web.TransactionType)

StackTraceString:
HResult: 80131500
The current thread is unmanaged
-----------------

Number of exceptions of this type:        8
Exception object: 03c6fbb8
Exception type: System.ArgumentNullException
Message: Value cannot be null.
InnerException:
StackTrace (generated):
    SP       IP       Function
    1F15E3E0 795FC73C System.Guid..ctor(System.String)
    1F15E43C 1D0AB97F Pages.PersonHomePage.Page_Load(System.Object, System.EventArgs)

StackTraceString:
HResult: 80004003
The current thread is unmanaged
-----------------

Number of exceptions of this type:       36
Exception object: 02620134
Exception type: System.Threading.ThreadAbortException
Message:
InnerException:
StackTrace (generated):

StackTraceString:
HResult: 80131530
The current thread is unmanaged
-----------------

Number of exceptions of this type:      116
Exception object: 0396692c
Exception type: System.Web.HttpUnhandledException
Message:
InnerException: System.NullReferenceException, use !PrintException 03966878 to see more
StackTrace (generated):
    SP       IP       Function
    1CE6DF58 6614FDB2 System.Web.UI.Page.HandleError(System.Exception)
    1CE6DFA0 6615681A System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)
    1CE6EF10 66154A8A System.Web.UI.Page.ProcessRequest(Boolean, Boolean)
    1CE6EF48 66154967 System.Web.UI.Page.ProcessRequest()
    1CE6EF80 66154887 System.Web.UI.Page.ProcessRequestWithNoAssert(System.Web.HttpContext)
    1CE6EF88 6615481A System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)
    1CE6EF9C 1C741EAE ASP.sys_pages_application_application_aspx.ProcessRequest(System.Web.HttpContext)
    1CE6EFA8 65FF27D4 System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
    1CE6EFDC 65FC15B5 System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)

StackTraceString:
HResult: 80004005
The current thread is unmanaged
-----------------

Number of exceptions of this type:      136
Exception object: 03966878
Exception type: System.NullReferenceException
Message: Object reference not set to an instance of an object.
InnerException:
StackTrace (generated):
    SP       IP       Function
    1CE6EC2C 1C745B38 Pages.Application.Page_Load(System.Object, System.EventArgs)
    00000000 00000001 System.EventHandler.Invoke(System.Object, System.EventArgs)
    1CE6ED34 66143A84 System.Web.UI.Control.OnLoad(System.EventArgs)
    1CE6ED44 66143AD0 System.Web.UI.Control.LoadRecursive()
    1CE6ED58 66155106 System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)

StackTraceString:
HResult: 80004003
The current thread is unmanaged
-----------------

Total 303 exceptions

As you can see the !dae-command lists all exception types found on the heap. If possible it also gives us a callstack for each exception. Please note, however, that this doesn't mean that the call stack is more or less the same for all exceptions. In the sample above you might have ~20 different callstacks leading to the 136 NullReferenceExceptions you see.

Unfortunately the !dae command is not available in version 2.0 of sos.dll. Still, it's quite easy to get (more or less) the same result by using the !dumpheap command. If we just type "!dumpheap -type Exception" we'll get a list of all objects with the string "Exception" in their class name. This is almost as good.

0:000> !dumpheap -type Exception -stat
------------------------------
Heap 0
total 79 objects
------------------------------
Heap 1
total 76 objects
------------------------------
Heap 2
total 91 objects
------------------------------
Heap 3
total 92 objects
------------------------------
total 338 objects
Statistics:
      MT    Count    TotalSize Class Name
790ff624        3           36 System.Text.DecoderExceptionFallback
790ff5d8        3           36 System.Text.EncoderExceptionFallback
790f9ad4        1           72 System.ExecutionEngineException
790f9a30        1           72 System.StackOverflowException
790f998c        1           72 System.OutOfMemoryException
653c8d04        1           76 System.Data.SqlClient.SqlException
790f984c        3          216 System.Exception
66414de0       18          216 System.Web.HttpApplication+CancelModuleException
7911bc7c       11          352 System.UnhandledExceptionEventHandler
7911a3b0        8          608 System.ArgumentNullException
790f9b78       36         2592 System.Threading.ThreadAbortException
663d9268      116         9744 System.Web.HttpUnhandledException
7915cf40      136         9792 System.NullReferenceException
Total 338 objects

As you can see, this gives us almost the same information except for the callstacks.

Knowing what to ignore

When analyzing data it is always good to know how to filter the information.

The ever-present exceptions

There are a three exceptions that are created as soon as the worker process starts. This means that you will always see them on the heap even if they haven't been thrown at all.:

System.ExecutionEngineException
System.StackOverflowException
System.OutOfMemoryException

So why are they created if we haven't thrown them? - Any guesses?

The answer is quite simple: If you run into a situation where you need to throw any of these exceptions you will probably be in a state where you can't create them. For example, you've run out of memory and are no longer able to allocate even the tiniest string. How would you then be able to allocate enough memory to create a new exception?

So, provided that there's still only one of each on the heap, you can most likely ignore these three exceptions. When it comes to ExecutionEngineExceptions and OutOfMemoryExceptions you will probably have a pretty good idea that this is what you're looking for, and finding a StackOverflowException isn't that hard. If you run !clrstack and find a callstack of 200+ lines you can be more or less certain that this is your problem.

System.Threading.ThreadAbortException

Usually when you see a ThreadAbortExceptions it is because you've called Response.Redirect.

Whenever you call Response.Redirect, this will also result in a call to Response.End. This will terminate the thread prematurely, resulting in a System.Threading.ThreadAbortException. See the callstack below for an example.

    SP       IP       Function
    1ED6F37C 793D74D0 mscorlib_ni!System.Threading.Thread.Abort(System.Object)+0x2c
    1ED6F390 6600CA8C System_Web_ni!System.Web.HttpResponse.End()+0x5c
    1ED6F3A4 6600B8C3 System_Web_ni!System.Web.HttpResponse.Redirect(System.String, Boolean)+0x1f3
    1ED6F3B8 6600B6B7 System_Web_ni!System.Web.HttpResponse.Redirect(System.String)+0x7
    1ED6F3BC 1DDD2E1D Company_Web!Company.Web.UI.Page.RedirectToPreviousPage()+0x125

Obviously I'm not saying you should discard all System.Threading.ThreadAbortExceptions as irrelevant. Even if you have no reason to believe that ThreadAbortExceptions are a major concern it's always a good idea to investigate a few of them. Take a minute or two to confirm that there is an underlying call to Response.End caused by a Response.Redirect. Once you think that you have enough statistical data to imply that the ThreadAbortExceptions are caused by Redirects you can move on.

Examining the Exceptions

Okay, so say we want to look at the callstacks for the System.Data.SqlClient.SqlException, well first of all we need the address for it. As you might remember, this is easily obtained by using !dumpheap without the -stat option.

0:000> !dumpheap -type System.Data.SqlClient.SqlException
------------------------------
Heap 0
Address       MT     Size
total 0 objects
------------------------------
Heap 1
Address       MT     Size
total 0 objects
------------------------------
Heap 2
Address       MT     Size
0b62cf38 653c8d04       76
total 1 objects
------------------------------
Heap 3
Address       MT     Size
total 0 objects
------------------------------
total 1 objects
Statistics:
      MT    Count    TotalSize Class Name
653c8d04        1           76 System.Data.SqlClient.SqlException
Total 1 objects

Now we have the address for the exception. In order to investigate the exception we could use !dumpobject, but there is another command I want to use first.

!printexception (!pe)

Running the !printexception command on the address of an exception will give us some neat information on the exception in question. Here's the result of running !printexception on the SqlException:

0:000> !pe 0b62cf38
Exception object: 0b62cf38
Exception type: System.Data.SqlClient.SqlException
Message: Transaction (Process ID 96) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
InnerException:
StackTrace (generated):
    SP       IP       Function
    1DACECA8 1D0A4D4C Company_Database_1d3c0000!Company.Database.AuditTrail.Write(System.String, System.Guid, System.String)+0x194
    1DACED6C 1D0A4B98 Company_Database_1d3c0000!Company.Database.AuditTrail.AddEntry(EntryType, System.Guid, System.String)+0x48
    1DACED90 1D0A4B24 Company_Database_1d3c0000!Company.Database.DataFunctions.SaveRow(System.String, System.Guid, Boolean)+0x5e4
    1DACEE20 1DDD7A06 FooBase_1d560000!FooBase.PersonApplicationOtherQualificationBase.PersonApplicationOtherQualificationBaseManager.BaseSave(System.Guid, Boolean)+0x2e
    1DACEE34 1DDD79BF Foo_1cfd0000!Foo.PersonApplicationOtherQualification.PersonApplicationOtherQualificationManager.Save(System.Guid, Boolean)+0x27
    1DACEE48 1DDD77F5 Foo_1cfd0000!UserControls.PersonApplicationOtherQualificationDetail.OnSave()+0x65
    1DACEE70 1DDD2701 Company_Web_1d090000!Company.Web.UI.Page.InvokeUsercontrolTransaction(System.Web.UI.Control, Company.Web.TransactionType)+0x1d1

StackTraceString:
HResult: 80131904
The current thread is unmanaged

This is good stuff. The command was even able to generate a callstack for us. (This may not always be the case, since the callstack may very well have gone out of scope.)

!dumpobject (!do) still has its uses

I wouldn't say that !printexception is a complete replacement for !dumpobject when it comes to examining exceptions. !Printexception will fit the exception into a standard template, and since some exceptions may contain more data than others we sometimes want to use !dumpobject as well. The SqlException has a property called _errors that contains a System.Data.SqlClient.SqlErrorCollection that we might want to look at. This is not in the listing above, so we need to use !dumpobject to look at it.

0:000> !do 0b62cf38
Name: System.Data.SqlClient.SqlException
MethodTable: 653c8d04
EEClass: 6540a0d0
Size: 76(0x4c) bytes
(C:\WINDOWS\assembly\GAC_32\System.Data\2.0.0.0__b77a5c561934e089\System.Data.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
790f9244 40000b5        4        System.String 0 instance 00000000 _className
79107d4c 40000b6        8 ...ection.MethodBase 0 instance 00000000 _exceptionMethod
790f9244 40000b7        c        System.String 0 instance 00000000 _exceptionMethodString
790f9244 40000b8       10        System.String 0 instance 0b62cdfc _message
79112734 40000b9       14 ...tions.IDictionary 0 instance 0b62cf84 _data
790f984c 40000ba       18     System.Exception 0 instance 00000000 _innerException
790f9244 40000bb       1c        System.String 0 instance 00000000 _helpURL
790f8a7c 40000bc       20        System.Object 0 instance 0b62d030 _stackTrace
790f9244 40000bd       24        System.String 0 instance 00000000 _stackTraceString
790f9244 40000be       28        System.String 0 instance 00000000 _remoteStackTraceString
790fdb60 40000bf       34         System.Int32 1 instance        0 _remoteStackIndex
790f8a7c 40000c0       2c        System.Object 0 instance 00000000 _dynamicMethods
790fdb60 40000c1       38         System.Int32 1 instance -2146232060 _HResult
790f9244 40000c2       30        System.String 0 instance 00000000 _source
790fcfa4 40000c3       3c        System.IntPtr 1 instance        0 _xptrs
790fdb60 40000c4       40         System.Int32 1 instance -532459699 _xcode
653c8b28 40017e0       44 ...qlErrorCollection 0 instance 0b62cd90 _errors

There we have it. Now we can continue using !dumpobject to investigate it even further if we wish.

Inner exceptions

If we take a look at one of the HttpUnhandledExceptions we find that it has an inner exception. It is even nice enough to let us know how to find out more about it.

0:000> !pe 10544f64
Exception object: 10544f64
Exception type: System.Web.HttpUnhandledException
Message:
InnerException: System.NullReferenceException, use !PrintException 10544df8 to see more
StackTrace (generated):
    SP       IP       Function
    1E3BE1D8 6614FDB2 System_Web_ni!System.Web.UI.Page.HandleError(System.Exception)+0x3e6
    1E3BE220 6615681A System_Web_ni!System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)+0x1b3a
    1E3BF190 66154A8A System_Web_ni!System.Web.UI.Page.ProcessRequest(Boolean, Boolean)+0xd6
    1E3BF1C8 66154967 System_Web_ni!System.Web.UI.Page.ProcessRequest()+0x57
    1E3BF200 66154887 System_Web_ni!System.Web.UI.Page.ProcessRequestWithNoAssert(System.Web.HttpContext)+0x13
    1E3BF208 6615481A System_Web_ni!System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)+0x32
    1E3BF21C 1C741EAE App_Web__ekpvebx!ASP.sys_pages_application_application_aspx.ProcessRequest(System.Web.HttpContext)+0x1e
    1E3BF228 65FF27D4 System_Web_ni!System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()+0x130
    1E3BF25C 65FC15B5 System_Web_ni!System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)+0x41

This means that the System.NullReferenceException mentioned lead to the HttpUnhandledException we're currently investigating. So if we want to find the root cause we'll need to investigate the inner exception as well.

Extra credit

If you've looked at my "Advanced commands"-post a while back you saw some examples of the .foreach command. This is a great command to use, for example if you want to see the callstack for all System.ArgumentNullExceptions. Instead of manually iterating through all the exceptions we can now dump them all at once, check their callstacks, etc.

0:000> .foreach(myVariable {!dumpheap -type System.ArgumentNullException -short}){!pe myVariable;.echo *************}
Exception object: 03c6fbb8
Exception type: System.ArgumentNullException
Message: Value cannot be null.
InnerException:
StackTrace (generated):
    SP       IP       Function
    1F15E3E0 795FC73C mscorlib_ni!System.Guid..ctor(System.String)+0x2a14bc
    1F15E43C 1D0AB97F Foo_1cfd0000!Pages.PersonHomePage.Page_Load(System.Object, System.EventArgs)+0x77

StackTraceString:
HResult: 80004003
The current thread is unmanaged
*************
Exception object: 08378e24
Exception type: System.ArgumentNullException
Message: Value cannot be null.
InnerException:
StackTrace (generated):
    SP       IP       Function
    1CE6EC60 795FC73C mscorlib_ni!System.Guid..ctor(System.String)+0x2a14bc
    1CE6ECBC 1D0AB97F Foo_1cfd0000!Pages.PersonHomePage.Page_Load(System.Object, System.EventArgs)+0x77

StackTraceString:
HResult: 80004003
The current thread is unmanaged
*************
Exception object: 084c0b30
Exception type: System.ArgumentNullException
Message: Value cannot be null.
InnerException:
StackTrace (generated):
    SP       IP       Function
    1E3BEEE0 795FC73C mscorlib_ni!System.Guid..ctor(System.String)+0x2a14bc
    1E3BEF3C 1D0AB97F Foo_1cfd0000!Pages.PersonHomePage.Page_Load(System.Object, System.EventArgs)+0x77

StackTraceString:
HResult: 80004003
The current thread is unmanaged
*************
Exception object: 08522f84
Exception type: System.ArgumentNullException
Message: Value cannot be null.
InnerException:
StackTrace (generated):
    SP       IP       Function
    1E3BEEE0 795FC73C mscorlib_ni!System.Guid..ctor(System.String)+0x2a14bc
    1E3BEF3C 1D0AB97F Foo_1cfd0000!Pages.PersonHomePage.Page_Load(System.Object, System.EventArgs)+0x77

StackTraceString:
HResult: 80004003
The current thread is unmanaged
*************
Exception object: 0c036bf8
Exception type: System.ArgumentNullException
Message: Value cannot be null.
InnerException:
StackTrace (generated):
    SP       IP       Function
    1DC7EB60 795FC73C mscorlib_ni!System.Guid..ctor(System.String)+0x2a14bc
    1DC7EBBC 1D0AB97F Foo_1cfd0000!Pages.PersonHomePage.Page_Load(System.Object, System.EventArgs)+0x77

StackTraceString:
HResult: 80004003
The current thread is unmanaged
*************
Exception object: 105d7f60
Exception type: System.ArgumentNullException
Message: Value cannot be null.
InnerException:
StackTrace (generated):
    SP       IP       Function
    1F39F360 795FC73C mscorlib_ni!System.Guid..ctor(System.String)+0x2a14bc
    1F39F3BC 1D0AB97F Foo_1cfd0000!Pages.PersonHomePage.Page_Load(System.Object, System.EventArgs)+0x77

StackTraceString:
HResult: 80004003
The current thread is unmanaged
*************
Exception object: 106206fc
Exception type: System.ArgumentNullException
Message: Value cannot be null.
InnerException:
StackTrace (generated):
    SP       IP       Function
    1F39F360 795FC73C mscorlib_ni!System.Guid..ctor(System.String)+0x2a14bc
    1F39F3BC 1D0AB97F Foo_1cfd0000!Pages.PersonHomePage.Page_Load(System.Object, System.EventArgs)+0x77

StackTraceString:
HResult: 80004003
The current thread is unmanaged
*************
Exception object: 1077a864
Exception type: System.ArgumentNullException
Message: Value cannot be null.
InnerException:
StackTrace (generated):
    SP       IP       Function
    1E3BEEE0 795FC73C mscorlib_ni!System.Guid..ctor(System.String)+0x2a14bc
    1E3BEF3C 1D0AB97F Foo_1cfd0000!Pages.PersonHomePage.Page_Load(System.Object, System.EventArgs)+0x77

StackTraceString:
HResult: 80004003
The current thread is unmanaged
*************
Unknown option: ------------------------------
*************

Well this post was even longer than usual.

I hope you found it of value, and I'll gladly listen to any comments, feedback or wishes on future topics you might have.

/ Johan

Implicit casting in DataRepeater not working in certain scenarios after SP1

JohanS — Wed, 30 Jan 2008 15:21:34 GMT

Problem

Using the following VB-syntax in a DataRepeater will no longer work after applying .NET Framework 2.0 Service Pack 1 in a medium-trust environment.

<asp:Repeater ID="Repeater1" runat="server">
        <ItemTemplate>
               <%#Container.DataItem("data")%><br />
        </ItemTemplate>
</asp:Repeater>

Resolution

Use Eval-method, as recommended in all Microsoft documentation. Using explicit casting would work as well, but you'd have to re-write every page if you decide to change your data layer.

Repro

Create an ASP.NET page with a repeater using a simple OleDb database as it's DataSource.

If you run this application in full trust, you should encounter no problems.

Now, set the trust level to medium. (You'll have to modify the settings and give it access to OleDbConnections.) When you run it you'll get a Security Exception:

Security Exception
Description: The application attempted to perform an operation not allowed by the security policy. To grant this application the required permission please contact your system administrator or change the application's trust level in the configuration file.

Exception Details: System.Security.SecurityException: Request failed.

Source Error:

An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.

Stack Trace:

[SecurityException: Request failed.]
   System.Security.CodeAccessSecurityEngine.ThrowSecurityException(Assembly asm, PermissionSet granted, PermissionSet refused, RuntimeMethodHandle rmh, SecurityAction action, Object demand, IPermission permThatFailed) +150
   System.Security.CodeAccessSecurityEngine.ThrowSecurityException(Object assemblyOrString, PermissionSet granted, PermissionSet refused, RuntimeMethodHandle rmh, SecurityAction action, Object demand, IPermission permThatFailed) +100
   System.Security.CodeAccessSecurityEngine.CheckSetHelper(PermissionSet grants, PermissionSet refused, PermissionSet demands, RuntimeMethodHandle rmh, Object assemblyOrString, SecurityAction action, Boolean throwException) +281
   System.Security.PermissionSetTriple.CheckSetDemand(PermissionSet demandSet, PermissionSet& alteredDemandset, RuntimeMethodHandle rmh) +67
   System.Security.PermissionListSet.CheckSetDemand(PermissionSet pset, RuntimeMethodHandle rmh) +145
   System.Security.PermissionListSet.DemandFlagsOrGrantSet(Int32 flags, PermissionSet grantSet) +43
   System.Threading.CompressedStack.DemandFlagsOrGrantSet(Int32 flags, PermissionSet grantSet) +41
   System.Security.CodeAccessSecurityEngine.ReflectionTargetDemandHelper(Int32 permission, PermissionSet targetGrant, CompressedStack securityContext) +139
   System.Security.CodeAccessSecurityEngine.ReflectionTargetDemandHelper(Int32 permission, PermissionSet targetGrant) +51

Cause and resolution

Visual Basic has always been strong on implicit casting. For example, this code is completely valid:

Dim myString As String = 10

So, in the sample at the top, Container.DataItem is merely an Object. VB then casts this to a System.Data.Common.DbDataRecord object and we're able to retrieve the "data"-property. This is no longer working after Service Pack 1.

C, in all its variations wouldn't be caught dead accepting a syntax like String = Integer, so the code at the top would never work. Instead you'd have to explicitly cast the object to the appropriate type:

<asp:Repeater ID="Repeater1" runat="server">
        <ItemTemplate>
               <%#((System.Data.Common.DbDataRecord)Container.DataItem)["data"]%><br />
        </ItemTemplate>
</asp:Repeater>

If you change the VB code so that it uses the VB equivalent: CType you will resolve the problem.

This has never been the recommended approach

So, is the resolution to use explicit casting? Well, not really. If you look at the C# example above you might notice a potential problem. If we change our data layer we would have to go through every single page and change the explicit casting. This is one of the reasons why we've always recommended using the Databinder.Eval method for VB and C# alike, so if you've followed the design guidelines your code should look like this:

<asp:Repeater ID="Repeater1" runat="server">
        <ItemTemplate>
               <%#DataBinder.Eval(Container.DataItem, "data")%><br />
        </ItemTemplate>
</asp:Repeater>

Actually, as of Framework 2.0 it's enough to write Eval("data").

So, if you review the documentation on msdn for Framework 1.1, Framework 2.0, knowledge base articles, etc. you'll see that we consistently recommend using the Eval method.

/ Johan

Using WinDbg - Advanced commands

JohanS — Wed, 23 Jan 2008 17:46:00 GMT

Did you know you can build your own advanced commands using for each, if, etc? The complete list of control tokens are:

.if
.else
.elseif
.foreach
.for
.while
.do
.break
.continue
.catch
.leave
.printf
.block

Using these command tokes you can send quite advanced instructions to the debugger that not only will make your job a lot easier, but also impress your manager immensely. :)

.foreach

Let's begin with an easy example. Imagine you want to investigate all strings on the heap that are 6500 bytes or more. To list them you'd simply type !dumpheap -type System.String -min 6500. This will give you the following information:

0:000> !dumpheap -type System.String -min 6500
------------------------------
Heap 0
Address       MT     Size
790da154 790f9244     9280
0264c4d0 790f9244    32788
total 2 objects
------------------------------
Heap 1
Address       MT     Size
total 0 objects
------------------------------
Heap 2
Address       MT     Size
0b62e790 790f9244    11284
total 1 objects
------------------------------
Heap 3
Address       MT     Size
0e6839d0 790f9244    32788
0e717904 790f9244    32788
0fb2a320 790f9244     6828
total 3 objects
------------------------------
total 6 objects
Statistics:
      MT    Count    TotalSize Class Name
790f9244        6       125756 System.String
Total 6 objects

So far, so good. The problem is that in order to investigate each string you'd have to run !dumpobject (!do) on every address. This might be acceptable now that we're only dealing with 6 strings, but what if it were 25, or 100? I don't know if you're aware of this, but if you pass the -short argument to !dumpheap it will give you the minimum information (just the addresses of the objects in question):

0:000> !dumpheap -type System.String -min 6500 -short
790da154
0264c4d0
0b62e790
0e6839d0
0e717904
0fb2a320
------------------------------

Now, let's use this information in a .foreach-clause:

0:000> .foreach(myVariable {!dumpheap -type System.String -min 6500 -short}){!do myVariable;.echo *************}
Name: System.String
MethodTable: 790f9244
EEClass: 790f91a4
Size: 9280(0x2440) bytes
(C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
String:
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWGRkAiEPDxYCHwEFZFhYWFhYWFhYWFhYWFhYWFhYWFhYWF
hYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhkZAInDw8WCh8CBQFFHwMFCjIyLzExLzIwMDcfBAUBVh8FBQFFHwYFASpkZAIpD2QWBGYPZBYEAg
ETC...

Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
790fdb60 4000096        4         System.Int32 1 instance     4632 m_arrayLength
790fdb60 4000097        8         System.Int32 1 instance     4631 m_stringLength
790fad38 4000098        c          System.Char 1 instance       3c m_firstChar
790f9244 4000099       10        System.String 0   shared   static Empty
    >> Domain:Value 000d5eb8:790d57b4 000fb4c0:790d57b4 000ca848:790d57b4 1d8334d8:790d57b4 <<
79122994 400009a       14        System.Char[] 0   shared   static WhitespaceChars
    >> Domain:Value 000d5eb8:026203f0 000fb4c0:02624504 000ca848:026745f0 1d8334d8:026dcef4 <<
*************
Name: System.String
MethodTable: 790f9244
EEClass: 790f91a4
Size: 32786(0x8012) bytes
(C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
String:
g8PFgIfAQUHYWFhYWFhYWRkAgMPDxYCHwEFCWFhYWFhYWFhYWRkAgQPDxYCHwEFCjI4LzEyLzIwMDdkZAIvD2QWAmY
PZBYCZg9kFgICAw8PFgIfAQUFWFhYWFhkZAIxDw8WAh8JZ2QWBGYPZBYEAgEPZBYCZg9kFgQCAQ8WAh8IBQMxcHgWA
gIBDw8WAh8JaGRkAgMPZBYEAgEPDxYCHwEFCkFkZCBSZWNvcmRkZAIDDw8WAh8ABRZ+L0ltYWdlcy9UaXRsZS9OZXc
ETC...

Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
790fdb60 4000096        4         System.Int32 1 instance    16385 m_arrayLength
790fdb60 4000097        8         System.Int32 1 instance    10960 m_stringLength
790fad38 4000098        c          System.Char 1 instance       3c m_firstChar
790f9244 4000099       10        System.String 0   shared   static Empty
    >> Domain:Value 000d5eb8:790d57b4 000fb4c0:790d57b4 000ca848:790d57b4 1d8334d8:790d57b4 <<
79122994 400009a       14        System.Char[] 0   shared   static WhitespaceChars
    >> Domain:Value 000d5eb8:026203f0 000fb4c0:02624504 000ca848:026745f0 1d8334d8:026dcef4 <<
*************
Name: System.String
MethodTable: 790f9244
EEClass: 790f91a4
Size: 11282(0x2c12) bytes
(C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
String:
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWGRkAiEPDxYCHwEFZFhYWFhYWFhYWFhYWFhYWFhYWFhYWFa
YWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWF
hYWFhYWFhYWFhYWFhYWFhkZAInDw8WCh8CBQFFHwMFCjIyLzExLzIwMDcfBAUBVh8FBQFFHwYFASpkZAIpD2QWBGYPZ
ETC...

Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
790fdb60 4000096        4         System.Int32 1 instance     5633 m_arrayLength
790fdb60 4000097        8         System.Int32 1 instance     3092 m_stringLength
790fad38 4000098        c          System.Char 1 instance       5b m_firstChar
790f9244 4000099       10        System.String 0   shared   static Empty
    >> Domain:Value 000d5eb8:790d57b4 000fb4c0:790d57b4 000ca848:790d57b4 1d8334d8:790d57b4 <<
79122994 400009a       14        System.Char[] 0   shared   static WhitespaceChars
    >> Domain:Value 000d5eb8:026203f0 000fb4c0:02624504 000ca848:026745f0 1d8334d8:026dcef4 <<
*************
Name: System.String
MethodTable: 790f9244
EEClass: 790f91a4
Size: 32786(0x8012) bytes
(C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
String:
SRU5UIFBBR0U6IDMgb2YgMTVkZAILD2QWAmYPZBYcAgcPDxYCHwEFZFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWF
hYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYW
FhYWFhkZAINDw8WCh4BRQUBRR4CVFQFCjIyLzExLzIwMDceAlRWBQFWHgJURQUBRR4CVFIFASpkZAITDw8WCh8CBQFF
ETC...

Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
790fdb60 4000096        4         System.Int32 1 instance    16385 m_arrayLength
790fdb60 4000097        8         System.Int32 1 instance    10960 m_stringLength
790fad38 4000098        c          System.Char 1 instance       3c m_firstChar
790f9244 4000099       10        System.String 0   shared   static Empty
    >> Domain:Value 000d5eb8:790d57b4 000fb4c0:790d57b4 000ca848:790d57b4 1d8334d8:790d57b4 <<
79122994 400009a       14        System.Char[] 0   shared   static WhitespaceChars
    >> Domain:Value 000d5eb8:026203f0 000fb4c0:02624504 000ca848:026745f0 1d8334d8:026dcef4 <<
*************
Name: System.String
MethodTable: 790f9244
EEClass: 790f91a4
Size: 32786(0x8012) bytes
(C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
String:
0b25Ib21lBR9BcHBsaWNhdGlvbk5hdmlnYXRpb246aWJ0TG9nb3V0BTdfY3RsMDpQZXJzb25BcHBsaWNhdGlvbkRlZ3J
lZUxpc3Q6U3lzdGVtVGl0bGU6RWRpdEltYWdlBT5fY3RsMDpQZXJzb25BcHBsaWNhdGlvbkRlZ3JlZURpcGxvbWFMaXN
0OlN5c3RlbVRpdGxlOkVkaXRJbWFnZQVDX2N0bDA6UGVyc29uQXBwbGljYXRpb25PdGhlclF1YWxpZmljYXRpb25MaXN
ETC...

Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
790fdb60 4000096        4         System.Int32 1 instance    16385 m_arrayLength
790fdb60 4000097        8         System.Int32 1 instance    10960 m_stringLength
790fad38 4000098        c          System.Char 1 instance       3c m_firstChar
790f9244 4000099       10        System.String 0   shared   static Empty
    >> Domain:Value 000d5eb8:790d57b4 000fb4c0:790d57b4 000ca848:790d57b4 1d8334d8:790d57b4 <<
79122994 400009a       14        System.Char[] 0   shared   static WhitespaceChars
    >> Domain:Value 000d5eb8:026203f0 000fb4c0:02624504 000ca848:026745f0 1d8334d8:026dcef4 <<
*************
Name: System.String
MethodTable: 790f9244
EEClass: 790f91a4
Size: 6826(0x1aaa) bytes
(C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
String:
/wEPDwULLTEyMTQ5MDkyMjgPZBYCAgEPZBYGAgcPZBYEAgEPDxYCHghJbWFnZVVybAUrfi9pbWFnZXMvbmF2ZAILD2QW
aWdhdGlvbi9wYWdlcy9QYWdlMlByb2dyZXNzLmdpZmRkAgIPDxYCHgRUZXh0BRVDVVJSRU5UIFBBR0U6IDMgb2YgMTVk
ZAILD2QWAmYPZBYcAgcPDxYCHwEFZFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
ETC...

Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
790fdb60 4000096        4         System.Int32 1 instance     3405 m_arrayLength
790fdb60 4000097        8         System.Int32 1 instance     3404 m_stringLength
790fad38 4000098        c          System.Char 1 instance       2f m_firstChar
790f9244 4000099       10        System.String 0   shared   static Empty
    >> Domain:Value 000d5eb8:790d57b4 000fb4c0:790d57b4 000ca848:790d57b4 1d8334d8:790d57b4 <<
79122994 400009a       14        System.Char[] 0   shared   static WhitespaceChars
    >> Domain:Value 000d5eb8:026203f0 000fb4c0:02624504 000ca848:026745f0 1d8334d8:026dcef4 <<
*************
Unknown option: ------------------------------
*************

Let's analyze the exact syntax. Here's the command

.foreach(myVariable {!dumpheap -type System.String -min 6500 -short}){!do myVariable;.echo *************}

"myVariable" is, as the name implies, the name of the variable that I wish to use for the data generated by the first set of commands. The second set of commands is what I wish to execute for each token. First I run !do on the variable, and then I use the .echo-command to print a separator in order to make it a bit easier on the eyes.

There are additional parameters you can use. For example you can choose to skip every n number of variables, or specify a text file to be parsed and used as tokens instead. Take a look at the windbg documentation if you're interested.

.shell

I first saw this being used by my colleague Doug Stewart, who is a genius with these things.

0:000> .shell -i - -ci "!iisinfo.clientconns" FIND /c "Request active"
40
.shell: Process exited

What it does is, it runs !iisinfo.clientonns and uses the MS-DOS FIND-command to count the number of times the string "Request active" appears. Off course you could use it to search for certain strings from any type of output, like ".shell -i - -ci "!do 0b62e790" FIND /c /i "<table"" or whatever suits your needs.

Let's take a quick look at the syntax.

When it comes to ".shell" the -i option is mandatory. It specifies the file we want to use for input. In this scenario we don't want to use a file, so we add another hyphen, resulting in the following syntax: ".shell -i -"

We then add the -ci option, which states that we should treat the following commands as an input file instead. ".shell -i - -ci "!iisinfo.clientconns""

Finally we state what shell command we wish to run using this input. ".shell -i - -ci "!iisinfo.clientconns" FIND /c "Request active"".

Naturally we can use any complicated command we wish in the statement, so instead of !iisinfo.clientconns we could run one of our .foreach-loops instead.

/ Johan

Exceptions

JohanS — Fri, 18 Jan 2008 18:50:22 GMT

Introduction

This was originally intended to be a post on identifying and troubleshooting Exceptions using windbg. As I started writing I thought I should write a little something about exceptions in general, just to begin the post. That little something grew to be quite big and all of a sudden it was by all means worthy of a post of its own.

There are two scenarios that are exceptionally (no pun intended) common in my line of work:

Clients are reporting 2nd chance exceptions displayed on screen with the classic "Server Error"- page.
Performance is generally bad, and when we investigate it turns out that there are tons of exceptions being thrown every second.

What exactly is an exception?

When it comes to the .NET Framework an exception is a managed object that inherits from the Exception Class. It is usually created when something's gone wrong either in the application or the runtime. The exception contains detailed information about the error that created it such as a friendly error message, the stack trace leading to this exception being created, possible inner exceptions (exceptions that caused the exception) and more. In other words they contain a lot more information than a function simply returning a status code other than 0 if something went wrong. This makes it a lot easier for us to troubleshoot the application.

The downside is that this is resource-consuming. All the information available needs to be gathered and so throwing an exception is a lot more costly than returning an error code.

Exceptions must be handled

When an exception is thrown it must be handled one way or another or the application/web request will crash/fail. This is done by setting up a try/catch block.

   try {
     conn.Close();
   }
   catch(Exception ex) {
     // Something went wrong.
   }

If anything goes wrong within the try block, the catch block can now deal with it and act appropriately. It is still up to you to code a suitable reaction to the exception, but the interesting thing is that you need to deal with the problem. If the exception is unattended it will cause the application to crash (winforms) or return a "Server Error"-page (IIS).

This is really good, because this way you will know, one way or the other, that something has gone wrong. Even if you forget to add a try/catch block the exception will still be thrown. This will possibly cause the application to crash, but that's a much better option than continuing execution as if nothing happened.

Exceptions should be avoided

This is really important. To throw an exception is costly. Chris Brumme has written an excellent post on this subject and in it he writes about some of the things that the runtime does when throwing an exception. Such as:

Grab a stack trace by interpreting metadata emitted by the compiler to guide our stack unwind.
Run through a chain of handlers up the stack, calling each handler twice.
Compensate for mismatches between SEH, C++ and managed exceptions.
Allocate a managed Exception instance and run its constructor. Most likely, this involves looking up resources for the various error messages.
Probably take a trip through the OS kernel. Often take a hardware exception.
Notify any attached debuggers, profilers, vectored exception handlers and other interested parties.

Exceptions are there to tell you that something's gone wrong, but since they're quite costly to throw it would be much better if you anticipated them and tried to minimize them.

Why use the following code:

    void calculateAvg(Int32 X, Int32 Y)
    {
        try
        {
            Label1.Text = (X / Y).ToString();
        }
        catch (Exception e)
        {
            Label1.Text = "Division by Zero";
        }
    }

when you can just as well use this:

    void calculateAvg(Int32 X, Int32 Y)
    {
        if (Y != 0)
        {
            Label1.Text = (X / Y).ToString();
        }
        else
        {
            Label1.Text = "Division by Zero";
        }

    }

As you can see, the first piece of code will throw an exception if Y happens to be zero. And all to no use. Take a look again at all the things we go through when throwing an exception and consider if that is really necessary when all you needed to do was to verify that Y != 0.

How many exceptions is your application throwing per second?

In a perfect world you should be able to answer this without hesitation. Anyway, if you honestly have no idea then I suggest you start Performance Monitor, add a counter for .NET CLR Exceptions/# of Exceps Thrown and run a stress test.

I've come across applications that throw over 150.000 exceptions in less than an hour.

There may also be times when this is actually expected and acceptable. For example, the Response.Redirect command results in a System.Threading.ThreadAbortException. If your application relies heavily on Response.Redirect for some reason, then you needn't worry. If, however, you're not aware of what is causing these exceptions I'd urge you to look into it.

None of us would want to let our systems run with constant red or yellow warnings in the event logs. The least we'd do is check up on them and see what caused them.

Well, having written this little introduction I guess I should now get going on the "real" post. The one about how to find and troubleshoot those exceptions using Windbg...

Later! / Johan

Aspnet_state.exe crashing on Xp

JohanS — Fri, 21 Dec 2007 18:35:07 GMT

I ran across an interesting situation with a customer today.

He had a workstation running Windows Xp Professional. He'd previously been running Framework 1.1 and had been using the ASP.NET State Service. He had now upgraded to framework 2.0. and upon shutdown he now received the following error:

aspnet_state.exe. Application Error. The instruction at "0x6a2a2fec" referenced memory at "0x0000000c". The memory could not be "read".

To make a long story short it turned out this was due to an access violation upon shutdown. We did the following and resolved the situation:

Start Regedit.
Navigate to the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ASP.NET_2.0.50727 key.
Right-click "Names" and select Permissions on the context menu.
Click Add and enter NETWORK SERVICE, click "Check Names" and click Ok.
With NETWORK SERVICE highlighted in the "Groups or User Names" list click "Advanced".
On the "Advanced Security Settings for Names" dialogue highlight NETWORK SERVICE and click "Edit".
In the "Permission entry for Names" dialogue check the "Name" box is showing "NETWORK SERVICE" and put check marks against "Query Value", "Set Value", "Create Subkey", "Enumerate Subkeys", "Notify" and "Read Control".
Click Ok on all the dialogues and close Regedit.

As I assisted him it turned out my customer had searched the web for a solution and had found quite a few people experiencing the same problem. I guess most of you are working with Windows 2003 / Vista, so this information might be a bit obsolete. But I thought I should share the knowledge anyway.

Happy Holidays! / Johan

Lot's of objects on the Large Object Heap on a 64-bit server

JohanS — Fri, 14 Dec 2007 17:31:22 GMT

My colleague Tess showed me a dump today which I thought was really interesting. The scenario was as follows:

- We have a web application that's been running successfully on a 32-bit server for quite some time. We've now moved to a 64-bit server. The code remains unchanged but we're seeing a lot more objects on the Large Object Heap (LOH). How's that possible?

Well looking closer at the LOH revealed a lot of Object Arrays. These arrays were around the 150 KB-range. An Object Array is nothing but a collection of addresses, each being one DWORD long (32-bit). When we ran the application in a 64-bit environment the addresses were now two DWORDS long. Since the addresses had doubled in size the Object Arrays had also doubled in size. In the 32-bit environment they were around 75 KB, but now they were 150 KB. As I've mentioned before, the limit for when an object goes on the Large Object Heap is 85 KB, so all those arrays now ended up on the LOH.

Since the LOH also isn't compacted this lead to a lot of fragmentation as well. All in all the memory usage of the exact same application saw a significant increase when being executed in the 64-bit environment.

Tess has actually blogged about this in the past. I definitively recommend taking a look at that article.

/ Johan

Getting started with windbg - part II

JohanS — Mon, 26 Nov 2007 16:49:50 GMT

This is a continuation of my previous post with the imaginative name Getting started with windbg - part I. I'll be assuming that you've read it, so if you haven't I suggest you check it out first. We're still working with the same sample dump, so I'll pretty much pick up right where we left off.

More useful commands

Last time we used some nice commands from the sos-extension to look at the running callstacks, the requests, the CPU load of the threads, etc. We also dug deeper into the callstack to see what distinguished name we used for a SearchRequest. We'll keep using these commands, but also learn a few new ones.

!dumpstackobjects (!dso)

Now, imagine that we're looking at a specific thread and would like to see all managed objects referenced to by the current stack. Is there a way for us to do this? - There most certainly is. It's called dumpstackobjects, or dso for short. If we run it on the current thread we'll see a listing of objects that are referenced by the callstack. The whole output would look something like this (trimmed down to size):

0:050> !dso
OS Thread Id: 0x1e8c (50)
ESP/REG Object   Name
17a9e534 0741f860 System.RuntimeType
17a9e6b8 271fdfe0 System.DirectoryServices.Protocols.LdapConnection
17a9e6bc 27246f20 System.DirectoryServices.Protocols.SEC_WINNT_AUTH_IDENTITY_EX
17a9e740 271fdfe0 System.DirectoryServices.Protocols.LdapConnection
17a9e744 27246f20 System.DirectoryServices.Protocols.SEC_WINNT_AUTH_IDENTITY_EX
17a9e764 27246f20 System.DirectoryServices.Protocols.SEC_WINNT_AUTH_IDENTITY_EX
17a9e768 271fdfe0 System.DirectoryServices.Protocols.LdapConnection
17a9e780 271fdfe0 System.DirectoryServices.Protocols.LdapConnection
17a9e784 27246e38 System.DirectoryServices.Protocols.SearchRequest
17a9e794 271fdf14 System.DirectoryServices.Protocols.LdapDirectoryIdentifier
17a9e7a8 27246ef8 System.Collections.ArrayList
17a9e7bc 27246ef8 System.Collections.ArrayList
17a9e7c8 271fdfe0 System.DirectoryServices.Protocols.LdapConnection
17a9e8a4 27246e38 System.DirectoryServices.Protocols.SearchRequest
17a9e8d0 27246ed8 System.Object[]    (System.Object[])
17a9e8e0 073ff6b8 System.String    cn
17a9e8e4 271fdfe0 System.DirectoryServices.Protocols.LdapConnection
17a9e8f4 27246d00 System.String    CN=Dummy,CN=Accounts,CN=useradm,DC=Dummy,DC=net
17a9e8f8 271fdfe0 System.DirectoryServices.Protocols.LdapConnection
17a9e8fc 27246e38 System.DirectoryServices.Protocols.SearchRequest
17a9e910 03380310 System.String
17a9e914 27246e24 System.Object[]    (System.String[])
17a9e918 272399a8 System.String    CN=OID-Dummy-ABC123,CN=Dummy,CN=Accounts,CN=useradm,DC=Dummy,DC=net
17a9e91c 27246ddc System.String    (CN=OID-Dummy-ABC123)
...etc...

This is extremely useful when we want to see all the objects referenced by this thread alone. If you want to analyze one of the objects, simply copy the the address from the Object-column and use !dumpobject:

0:050> !do 271fdfe0
Name: System.DirectoryServices.Protocols.LdapConnection
MethodTable: 14a2040c
EEClass: 149daf08
Size: 56(0x38) bytes
(C:\WINDOWS\assembly\GAC_MSIL\System.DirectoryServices.Protocols\2.0.0.0__b03f5f7f11d50a3a\System.DirectoryServices.Protocols.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
14a2078c 40000c3        4 ...NetworkCredential 0 instance 00000000 directoryCredential
14a2144c 40000c4        8 ...ificateCollection 0 instance 271fe018 certificatesCollection
1202af88 40000c5       10      System.TimeSpan 1 instance 271fdff0 connectionTimeOut
1466fe50 40000c6        c ...rectoryIdentifier 0 instance 271fdf14 directoryIdentifier
14a2034c 4000236       24         System.Int32 0 instance        2 connectionAuthType
14a223a4 4000237       18 ...dapSessionOptions 0 instance 271fe2d8 options
0fb896d8 4000238       28        System.IntPtr 0 instance 564180944 ldapHandle
120261c8 4000239       2c       System.Boolean 0 instance        0 disposed
120261c8 400023a       2d       System.Boolean 0 instance        0 bounded
120261c8 400023b       2e       System.Boolean 0 instance        0 needRebind
14a22084 400023e       1c ...pResponseCallback 0 instance 271fe03c fd
120261c8 4000243       2f       System.Boolean 0 instance        0 setFQDNDone
120261c8 4000244       30       System.Boolean 0 instance        1 automaticBind
120261c8 4000245       31       System.Boolean 0 instance        1 needDispose
120261c8 4000246       32       System.Boolean 0 instance        1 connected
14a2267c 4000247       20 ...s.QUERYCLIENTCERT 0 instance 271fe394 clientCertificateRoutine
0fd314bc 400023c       20 ...ections.Hashtable 0   shared   static handleTable
    >> Domain:Value 0019daf0:NotInit 11b42540:073fe504 <<
02c36ca0 400023d       24        System.Object 0   shared   static objectLock
    >> Domain:Value 0019daf0:NotInit 11b42540:073fe53c <<
0fd314bc 400023f       28 ...ections.Hashtable 0   shared   static asyncResultTable
    >> Domain:Value 0019daf0:NotInit 11b42540:073fe610 <<
14a21864 4000240       2c ...lResultsProcessor 0   shared   static partialResultsProcessor
    >> Domain:Value 0019daf0:NotInit 11b42540:073fe678 <<
12305e94 4000241       30 ....ManualResetEvent 0   shared   static waitHandle
    >> Domain:Value 0019daf0:NotInit 11b42540:073fe64c <<
14a21954 4000242       34 ...lResultsRetriever 0   shared   static retriever
    >> Domain:Value 0019daf0:NotInit 11b42540:073fe6a8 <<

!dumparray (!da)

As you might have noticed we have a couple of object arrays on the stack. Look for the System.Object[]-type in the listing above and you'll find them. If you execute !dumpobject on an array you'll only get information about the array itself, not it's contents. To get information about what's in the array we need to use the !dumparray-command, or !da for short.

0:050> !do 27239b98
Name: System.Object[]
MethodTable: 02c3896c
EEClass: 02c388ec
Size: 24(0x18) bytes
Array: Rank 1, Number of elements 2, Type CLASS
Element Type: System.String
Fields:
None

0:050> !da 27239b98
Name: System.String[]
MethodTable: 02c3896c
EEClass: 02c388ec
Size: 24(0x18) bytes
Array: Rank 1, Number of elements 2, Type CLASS
Element Methodtable: 02c39310
[0] 272399a8
[1] 27239a44

As you can see the !dumparray-command gives us a bit more information about the object. We can see that it contains System.String data and we get the addresses of the two items in the array. Since they're System.String-objects we can simply use !dumpobject to view their contents.

!objsize

If you look at the listing above you'll see that the size of the object is listed as 24 bytes. To paraphrase Obi Wan Kenobi: "This is true, from a certain point of view." The 24 bytes are the size of the System.Object[]-object itself. Not it's contents. As you can see when we execute !dumparray, the array only contains two references to two strings. These strings are individual objects and could be 32 MB for all we know, so the 24 bytes are not the total size of the array, but it would still be correct to claim that the System.Object[] is only 24 bytes large.

To get the total size of the object we use the !objsize command:

0:050> !objsize 27239b98
sizeof(27239b98) = 348 ( 0x15c) bytes (System.Object[])

!objsize will iterate through all the child objects referenced by the object, as well as all the grandchildren and so on. Apparently the total size of the array and it's child objects is 348 bytes.

If there are a lot of child objects it may take some time for !objsize to calculate the total size of the object. You should also be aware that !objsize may not always be as smart as you'd like. If, for example, you have a custom button control that references it's parent aspx-page you'd get the total size of the aspx-page and all it's child controls. In other words: If !objsize claims that the object in question is ridiculously large you might want to manually check what the object references using !dumpobject.

!dumpheap

This is another command that I use frequently. It is, however a command that you will want to use with at least one argument. Dumpheap with no arguments will dump all objects on the heap, so I usually begin by using the -stat argument, which in itself will write a lot of info on the screen, but it will at least be summarized. Here' you'll find the trimmed down output of !dumpheap -stat:

0:050> !dumpheap -stat
------------------------------
Heap 0
total 2754508 objects
------------------------------
Heap 1
total 2761329 objects
------------------------------
total 5515837 objects
Statistics:
      MT    Count    TotalSize Class Name
16e0a6d8        1           12 System.Collections.Generic.ObjectEqualityComparer`1[[System.Data.ProviderBase.DbConnectionInternal, System.Data]]
16d9cd9c        1           12 System.Xml.Serialization.Configuration.DateTimeSerializationSection+DateTimeSerializationMode
16d9bf30        1           12 System.Diagnostics.OrdinalCaseInsensitiveComparer
16d9112c        1           12 System.Xml.Serialization.NameTable
16d7f664        1           12 System.Xml.Serialization.TempAssemblyCache
163ea85c        1           12 System.Data.Res
1501e4c4        1           12 System.Collections.Generic.ObjectEqualityComparer`1[[System.Web.UI.Control, System.Web]]
14efb138        1           12 System.Net.TimeoutValidator
14ef9964        1           12 System.Net.Cache.HttpRequestCacheLevel
14ef77a8        1           12 Microsoft.Win32.WinInetCache
14ef68e4        1           12 System.Net.WebRequest+WebProxyWrapper
14ef658c        1           12 System.Net.Configuration.ProxyElement+BypassOnLocalValues
14ef63d8        1           12 System.Net.Configuration.ProxyElement+AutoDetectValues
14ef5b68        1           12 System.Net.CaseInsensitiveAscii
14ef5610        1           12 System.Net.HeaderInfoTable
14ef4718        1           12 System.Net.HttpRequestCreator
14db6710        1           12 System.Web.Configuration.MachineKeyValidationConverter
14db3140        1           12 System.Collections.Generic.ObjectEqualityComparer`1[[System.Runtime.Serialization.MemberHolder, mscorlib]]
14b3f4d8        1           12 System.Web.UI.SupportsEventValidationAttribute
...etc...
14a276a8    19578       704808 System.DirectoryServices.Interop.AdsValueHelper
14a2ea24     9196       735680 System.Web.UI.WebControls.Label
14a2e51c    16862       741928 System.Web.UI.WebControls.Style
125778ec    48015       768240 System.Collections.Specialized.NameObjectCollectionBase+NameObjectEntry
120261c8    65842       790104 System.Boolean
14a2ee7c     9198       809424 System.Web.UI.WebControls.Table
14b311c4     9647       810348 System.Web.UI.WebControls.Image
13a2b7dc    34913       837912 System.Web.HttpServerVarsCollectionEntry
14b303a4    10605       848400 System.Web.UI.WebControls.HyperLink
14d8e3d4    77748       932976 Microsoft.Web.UI.WebControls.BaseChildNodeCollection+ActionType
14db90ac    81372       976464 System.Web.UI.WebControls.FontInfo
14b30694    28648      1031328 System.Web.UI.WebControls.TableRow+CellControlCollection
14d8fdbc    38912      1089536 Microsoft.Web.UI.WebControls.TreeNodeCollection
14b3d0bc    86592      1385472 System.Web.UI.Pair
1466c5c4    39305      1414980 System.Web.UI.ControlCollection
14d8e48c    77748      1865952 Microsoft.Web.UI.WebControls.BaseChildNodeCollection+Action
1545061c    38874      2176944 Microsoft.Web.UI.WebControls.TreeNode
14b30eec    52668      2317392 System.Web.UI.WebControls.TableItemStyle
14a2f804    28515      2395260 System.Web.UI.WebControls.TableRow
14a2be6c    40894      2453640 System.Web.UI.LiteralControl
0fd3da00   228792      2745504 System.Int32
14b3e3ac   244793      2937516 System.Web.UI.IndexedString
14a2de94   198580      3177280 System.Web.UI.StateBag
1466c454    80512      3542528 System.Web.UI.Control+OccasionalFields
12302c2c   205849      4116980 System.Collections.Specialized.HybridDictionary
14b30024    52934      4446456 System.Web.UI.WebControls.TableCell
12302f2c   178294      4992232 System.Collections.Specialized.ListDictionary
14a2e284   412762      6604192 System.Web.UI.StateItem
14d8ce64   117078      7024680 Microsoft.Web.UI.WebControls.CssCollection
0fd314bc   132065      7395640 System.Collections.Hashtable
1230319c   422580      8451600 System.Collections.Specialized.ListDictionary+DictionaryNode
1202a58c   380438      9130512 System.Collections.ArrayList
0fd32050   133000     22582944 System.Collections.Hashtable+bucket[]
02c3896c   649842     23275900 System.Object[]
0fd3c12c     3471     36385536 System.Byte[]
001fee20      338     65409920      Free
02c39310   683083    161821000 System.String
Total 5515837 objects
Fragmented blocks larger than 0.5 MB:
    Addr     Size      Followed by
2adf31cc    2.0MB         2aff85d8 System.String
2b006a2c   20.3MB         2c4530d8 System.Net.SocketAddress

As you can see we now get a listing sorted by size of all object-types on the heap. You'll usually find strings down at the bottom since that's what is commonly used the most.

Other useful arguments are -type and -mt (which stands for MethodTable). Using them you're able to see all objects of a specific type. For example. If we want to look at all HttpRequestCreators on the heap (there are one) you'll simply copy it's MethodTable which you'll find in the listing above (14ef4718) and use !dumpheap -mt

0:050> !dumpheap -mt 14ef4718
------------------------------
Heap 0
Address       MT     Size
0342ccf8 14ef4718       12
total 1 objects
------------------------------
Heap 1
Address       MT     Size
total 0 objects
------------------------------
total 1 objects
Statistics:
      MT    Count    TotalSize Class Name
14ef4718        1           12 System.Net.HttpRequestCreator

This gives us the address of the object and if we'd like to inspect it closer we simply use !dumpobject on that address.

!dumpheap -type works pretty much the same way, except this time you filter the results by class name. !dumpheap -type performs a substring match, so if you write !dumpheap -type System.Web you'll get every object who's class name contains System.Web, which would be a lot.

Other useful arguments are -min and -max which accept a number representing the minimum/maximum number of bytes the object size should be. This can be really useful when troubleshooting string-abuse, etc. Also !dumpheap -stat -min 85000 would list all objects on the large object heap.

Putting the tools to use

I'd now like to use the commands we've covered in a bit more practical scenario. The dump we've been looking at is from a previous case of mine. The application in question was running on a Web garden with two workerprocesses. Session State was handled by a SQL Server. The customer was experiencing performance issues and the problem description was hazy at best. Anyway I had tons of dumps to work with, so I simply poked around to see what I could find. One thing I did pretty early on was to look at caching. According to the customer they weren't using the cache at all, but I usually find it best to double-check this type of thing.

Determining the size of the cache

To find out how much data was kept in the cache I first needed to find the System.Web.Caching.Cache class. So I ran !dumpheap -stat -type System.Web.Caching.Cache. Note that I also used the -stat argument. Otherwise I would have gotten a very long list of System.Web.Caching.CacheKeys and System.Web.Caching.CacheEntrys as well. Anyway, here's the result:

0:050> !dumpheap -type System.Web.Caching.Cache -stat
------------------------------
Heap 0
total 665 objects
------------------------------
Heap 1
total 1084 objects
------------------------------
total 1749 objects
Statistics:
      MT    Count    TotalSize Class Name
123056f8        1           12 System.Web.Caching.CacheKeyComparer
1230494c        1           12 System.Web.Caching.Cache
1230500c        1           24 System.Web.Caching.CacheMultiple
1230514c        1           32 System.Web.Caching.CacheMemoryStats
123053b4        1           36 System.Web.Caching.CacheMemoryTotalMemoryPressure
123059bc        2           40 System.Web.Caching.CacheUsage
12304bdc        1           48 System.Web.Caching.CacheCommon
123054f4        1           52 System.Web.Caching.CacheMemoryPrivateBytesPressure
12305874        2           64 System.Web.Caching.CacheExpires
12304e64        2          200 System.Web.Caching.CacheSingle
1255b594       85         1360 System.Web.Caching.CacheDependency+DepFileInfo
123046c4       40         1440 System.Web.Caching.CacheDependency
123042ec       47         1504 System.Web.Caching.CacheItemRemovedCallback
123063fc      832        16640 System.Web.Caching.CacheKey
12306820      732        52704 System.Web.Caching.CacheEntry
Total 1749 objects

Okay, so now I had the MethodTable for the System.Web.Caching.Cache object. Therefore I could now get the address to the object itself. I did this by asking !dumpheap to list all the objects on the heap with that MethodTable. I knew that there was just going to be one hit:

0:050> !dumpheap -mt 1230494c
------------------------------
Heap 0
Address       MT     Size
03392d20 1230494c       12
total 1 objects
------------------------------
Heap 1
Address       MT     Size
total 0 objects
------------------------------
total 1 objects
Statistics:
      MT    Count    TotalSize Class Name
1230494c        1           12 System.Web.Caching.Cache
Total 1 objects

So now I ran !objsize on this object to see how big it was. This took a little time to calculate, since the cache is quite complex and there are a lot of children to iterate through.

0:050> !objsize 03392d20
sizeof(03392d20) = 266640828 ( 0xfe49dbc) bytes (System.Web.Caching.Cache)

So the cache is 266 MB in size. That's quite a lot considering the fact that the customer claimed that they weren't using the cache at all!

What is being cached?

To sample what the customer was caching I then took a look at a few of the CacheEntrys. I already had the MethodTable for the System.Web.Caching.CacheEntry from when I ran !dumpheap -type System.Web.Caching.Cache -stat (above), so I could use that to retrieve all CacheEntrys.

0:050> !dumpheap -mt 12306820
------------------------------
Heap 0
Address       MT     Size
033950bc 12306820       72
033a20d8 12306820       72
033ac79c 12306820       72
033da21c 12306820       72
033f04c4 12306820       72
03428ec8 12306820       72
0344dab4 12306820       72
03815d00 12306820       72
038265d8 12306820       72
....etc...
03af7010 12306820       72
03b291bc 12306820       72
03b2c674 12306820       72
03b6dca0 12306820       72
03b797dc 12306820       72
03b85318 12306820       72
03ba9150 12306820       72
03c258cc 12306820       72
03de43c8 12306820       72
03e160f8 12306820       72
total 382 objects
------------------------------
total 732 objects

Another valid command that would have given me the exact same output would off course have been !dumpheap -type System.Web.Caching.CacheEntry.

Okay, so now I had a long list of CacheEntrys. To sample the contents I just picked an address and examined it by using !dumpobject

0:050> !do 03b2c674
Name: System.Web.Caching.CacheEntry
MethodTable: 12306820
EEClass: 122f6470
Size: 72(0x48) bytes
(C:\WINDOWS\assembly\GAC_32\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
02c39310 4001327        4        System.String 0 instance 03b2c600 _key
0fb8f1f8 4001328        c          System.Byte 0 instance        2 _bits
0fd3da00 4001329        8         System.Int32 0 instance -1314181915 _hashCode
02c36ca0 4001330       10        System.Object 0 instance 03b2c644 _value
120219d0 4001331       1c      System.DateTime 1 instance 03b2c690 _utcCreated
120219d0 4001332       24      System.DateTime 1 instance 03b2c698 _utcExpires
1202af88 4001333       2c      System.TimeSpan 1 instance 03b2c6a0 _slidingExpiration
0fb8f1f8 4001334        d          System.Byte 0 instance        7 _expiresBucket
123062d8 4001335       34 ...g.ExpiresEntryRef 1 instance 03b2c6a8 _expiresEntryRef
0fb8f1f8 4001336        e          System.Byte 0 instance 4294967295 _usageBucket
12306738 4001337       38 ...ing.UsageEntryRef 1 instance 03b2c6ac _usageEntryRef
120219d0 4001338       3c      System.DateTime 1 instance 03b2c6b0 _utcLastUpdate
123046c4 4001339       14 ...g.CacheDependency 0 instance 00000000 _dependency
02c36ca0 400133a       18        System.Object 0 instance 033d8344 _onRemovedTargets
120219d0 400132d      1bc      System.DateTime 1   shared   static NoAbsoluteExpiration
    >> Domain:Value 0019daf0:NotInit 11b42540:03395104 <<
1202af88 400132e      1c0      System.TimeSpan 1   shared   static NoSlidingExpiration
    >> Domain:Value 0019daf0:NotInit 11b42540:03395114 <<
1202af88 400132f      1c4      System.TimeSpan 1   shared   static OneYear
    >> Domain:Value 0019daf0:NotInit 11b42540:03395124 <<

This dumped the CacheEntry and it's properties. I figured the most interesting piece of information would be the _value, so I simply copied the address of that property (look in the Value column) and used !dumpobject again.

0:000> !do 03e160c8
Name: System.Web.SessionState.InProcSessionState
MethodTable: 14dbad5c
EEClass: 14e43af8
Size: 48(0x30) bytes
(C:\WINDOWS\assembly\GAC_32\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
1466c9d8 4001d89        4 ...ateItemCollection 0 instance 1a7f5438 _sessionItems
1292672c 4001d8a        8 ...ObjectsCollection 0 instance 00000000 _staticObjects
0fd3da00 4001d8b        c         System.Int32 0 instance       20 _timeout
120261c8 4001d8c       18       System.Boolean 0 instance        0 _locked
120219d0 4001d8d       1c      System.DateTime 1 instance 03e160e4 _utcLockDate
0fd3da00 4001d8e       10         System.Int32 0 instance        1 _lockCookie
1202bf60 4001d8f       24 ...ReadWriteSpinLock 1 instance 03e160ec _spinLock
0fd3da00 4001d90       14         System.Int32 0 instance        0 _flags

Here I found something interesting. The value stored was in fact an InProcSessionState object, which -in case you didn't know this before- is stored in the Cache. This meant that the claim that the application was using SQL Server Session state was incorrect.

As it turned out the customer had temporarily switched to In-process for a brief test, but forgotten to switch back again. Had the application been live they would have spotted this in no time, but since they were stress testing they weren't really paying attention to what the server was returning as long as it returned something. Unknowingly running session state in-process compromised the stress-test in a number of ways. For example:

Memory usage for the worker process was a lot higher than expected
The load on the network was not as high as it would be under normal circumstances
The response times were no longer accountable
The SQL-server was not stressed to the extent it should have been, so any potential bottlenecks there went unnoticed

This was in no way the final solution for their performance issue. There were a lot more things we had to deal with, but I think it's a really nice example of how to use only three commands (!dumpheap, !objsize & !dumpobject) to dig up some really useful information.

Later! / Johan