Speaking of which... : Performance

Make sure you have an Internet connection when using signed assemblies

JohanS — Thu, 12 Nov 2009 11:08:22 GMT

A customer called in the other day and told me that his web application took a long time to start. My initial thought was (off course) that it was a matter of the classic slow-to start web services I’ve written about earlier.

(What to do about the slow startup of web services)

As I learned more about his problem I found that this was more than just a simple matter of pre-compilation and keeping your w3wp.exe alive.

The Cause

The customer was using authenticode signed assemblies in his web application. When that application calls the assembly for the first time it will want to go on-line and check that the certificate is still valid. So, what if this is an internal server with no Internet access? Well, then we have a problem. The process will spend quite some time trying to check the certificate revocation lists. Eventually it will give up, but for my customer this meant that it hung for at least 20 seconds upon startup.

The Resolution

To resolve this I’d chose one of the following three options:

Grant Internet access to the server

This is will obviously resolve the problem, but may not always be possible.

Review the need for Authenticode signing

If you own the assemblies then you might want to consider not using Authenticode signing at all. A Strong Named Assembly might be sufficient. For more information on Strong Named Assemblies and how to create them, please see the following article on MSDN: How to: Sign an Assembly with a Strong Name

Disable Signature Verification

If all else fails you can disable Signature Verification by adding the following to your machine.config or application.exe.config:

Identifying the problem using windbg

Okay, so if we suspect that this is the problem, how can we verify this?

I got a memory dump from my customer, so I’ll use that to demonstrate.

First of all you want to make sure the dump is taken as the application pool hangs upon the first request. Getting a dump right after the problem has occurred will usually do us no good at all. We need to get the dump as the problem is actually ocurring. Having made sure the dump is fine I open up the dump in windbg and load sos.dll. I then use the !aspxpages command to see the pending requests currently on the server. !aspxpages dumps all the HttpContexts found on the heap so we get a list of all currently executing requests, as well as recently finished ones that haven’t been garbage collected yet.

0:000> .load C:\Windows\Microsoft.NET\Framework\v2.0.50727\sos.dll
0:000> !aspxpages
Going to dump the HttpContexts found in the heap.
Loading the heap objects into our cache.
HttpContext    Timeout Completed     Running ThreadId ReturnCode   Verb RequestPath+QueryString
0x01b41ad4        0            no        11 Sec     XXX        200       /App/
0x01b75f48      110 Sec        no        11 Sec     XXX        401   GET /App/Default.aspx
0x01b7eb40    19200 Sec        no        11 Sec      19        200   GET /App/Default.aspx
Total 3 HttpContext objects

So, the request on thread 19 is the one we want. As we can see in the ThreadId column it is the only active request. Let’s jump to that thread and investigate what it is doing.

0:000> ~19s
eax=01510000 ebx=00000000 ecx=03e2d348 edx=776d9a94 esi=03e2d07c edi=00000000
eip=776d9a94 esp=03e2d034 ebp=03e2d0a4 iopl=0         nv up ei ng nz ac pe cy
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000             efl=00000297
ntdll!KiFastSystemCallRet:
776d9a94 c3              ret
0:019> !clrstack
OS Thread Id: 0x11cc (19)
ESP       EIP
03e2eb8c 776d9a94 [PrestubMethodFrame: 03e2eb8c] App.BasePage.Page_Load(System.Object, System.EventArgs)
03e2eba0 6aada7ff System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr, System.Object, System.Object, System.EventArgs)
03e2ebb0 6c392594 System.Web.Util.CalliEventHandlerDelegateProxy.Callback(System.Object, System.EventArgs)
03e2ebc4 6c38ba84 System.Web.UI.Control.OnLoad(System.EventArgs)
03e2ebd8 6c38bac3 System.Web.UI.Control.LoadRecursive()
03e2ebf0 6c387b74 System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)
03e2ed48 6c3877a4 System.Web.UI.Page.ProcessRequest(Boolean, Boolean)
03e2ed80 6c3876d1 System.Web.UI.Page.ProcessRequest()
03e2edb8 6c387666 System.Web.UI.Page.ProcessRequestWithNoAssert(System.Web.HttpContext)
03e2edc4 6c387642 System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)
03e2edd8 012b02a6 ASP.text.ProcessRequest(System.Web.HttpContext)
03e2ede8 6c38db16 System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
03e2ee1c 6c36132c System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)
03e2ee5c 6c95531f System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)
03e2ee60 6c94b704 [InlinedCallFrame: 03e2ee60]
03e2ef00 6c93613d System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)
03e2ef70 6ca0a7a2 System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
03e2ef74 6ca0a58f [InlinedCallFrame: 03e2ef74]
03e2f4c8 01092314 [NDirectMethodFrameStandalone: 03e2f4c8] System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr, System.Web.RequestNotificationStatus ByRef)
03e2f4d8 6ca0a839 System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
03e2f55c 6ca0a58f System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
03e2f65c 01092314 [ContextTransitionFrame: 03e2f65c]

So we’re in the Page_Load event of the page in question, but apart from that !clrstack doesn’t tell us much. Let’s take a look at the native callstack instead using the kb-command.

0:019> kb20
ChildEBP RetAddr Args to Child
03e2d030 776d9254 76adc244 000006d8 00000000 ntdll!KiFastSystemCallRet
03e2d034 76adc244 000006d8 00000000 03e2d07c ntdll!ZwWaitForSingleObject+0xc
03e2d0a4 74256247 000006d8 00003a98 00000000 kernel32!WaitForSingleObjectEx+0xbe
03e2d0d8 74251f7c 0141f2e4 00000002 00202005 cryptnet!CryptRetrieveObjectByUrlWithTimeout+0x1a5
03e2d1c8 742527a1 0141f2e4 00000002 00202005 cryptnet!CryptRetrieveObjectByUrlW+0xcc
03e2d268 742591be 00000000 0141f2e4 00000002 cryptnet!RetrieveObjectByUrlValidForSubject+0xa0
03e2d2c8 74252556 00000000 00000000 00000001 cryptnet!RetrieveTimeValidObjectByUrl+0x15c
03e2d374 74253fc1 00000082 0141f2d8 00000000 cryptnet!CTVOAgent::GetTimeValidObjectByUrl+0x178
03e2d45c 74253d98 00000003 018aa590 00000002 cryptnet!CTVOAgent::GetTimeValidObject+0x4a8
03e2d48c 74253428 00000003 018aa590 018c8220 cryptnet!FreshestCrlFromCrlGetTimeValidObject+0x2d
03e2d4d0 74259283 00000003 018aa590 018c8220 cryptnet!CryptGetTimeValidObject+0x58
03e2d54c 74253b1c 00000003 018aa590 018aa590 cryptnet!GetTimeValidCrl+0x2cb
03e2d590 74253992 018aa590 018c8220 03e2d5c8 cryptnet!GetBaseCrl+0x34
03e2d63c 757f77d5 00000001 00000001 00000001 cryptnet!MicrosoftCertDllVerifyRevocation+0x163
03e2d6c8 757f7641 00000001 00000001 00000001 crypt32!I_CryptRemainingMilliseconds+0x2aa
03e2d74c 757f7a26 00000001 00000001 00000001 crypt32!CertVerifyRevocation+0xd4
03e2d7f8 757f7838 01899590 00000000 00000000 crypt32!CChainPathObject::CalculateRevocationStatus+0x2d0
03e2d83c 75806c44 01899590 018c3018 01870fc0 crypt32!CChainPathObject::CalculateAdditionalStatus+0x152
03e2d904 758069a5 01899590 018aa590 01870fc0 crypt32!CCertChainEngine::CreateChainContextFromPathGraph+0x23e
03e2d93c 7580be14 018aa590 018aff94 01870fc0 crypt32!CCertChainEngine::GetChainContext+0x46
03e2d974 749172ba 01421338 018aa590 018aff94 crypt32!CertGetCertificateChain+0x72
03e2d9dc 749170fc 40000001 00000000 03e2da10 wintrust!_WalkChain+0x1ae
03e2da18 7491363b 00000000 03e2dbdc 03e2db74 wintrust!WintrustCertificateTrust+0xb9
03e2db30 7491346c 00000000 03e2dbe4 00000000 wintrust!_VerifyTrust+0x253
03e2db54 64025b1b 00000000 03e2dbe4 03e2db74 wintrust!WinVerifyTrust+0x50
03e2dbf8 716b7f4d 014443f8 00000000 00000003 mscorsec!GetPublisher+0xe4
03e2dc50 7145ed3b d60caabc 0188c4a8 00603c48 mscorwks!PEFile::CheckSecurity+0xcb
03e2dc78 7145ec84 d60caa64 00000000 0188c4a8 mscorwks!PEAssembly::DoLoadSignatureChecks+0x3a
03e2dca0 7145f0ca 01879f00 00000000 0188c4a8 mscorwks!PEAssembly::PEAssembly+0x109
03e2df3c 7145f1c5 0188c4a8 00000000 00000000 mscorwks!PEAssembly::DoOpen+0x103
03e2dfd0 71459062 0188c4a8 00000000 00000000 mscorwks!PEAssembly::Open+0x79
03e2e134 71456ace 03e2e15c 00000001 00000000 mscorwks!AppDomain::BindAssemblySpec+0x247

Here we see that we’re currently trying to check the certificate revocation list. This is done on a separate thread so this thread is currently doing nothing but waiting. Our next step would be to find this thread, but first we can find out the name of the .dll that we’re trying to verify. Look at the first argument to GetPublisher.

0:019> du 014443f8
014443f8 "C:\Windows\Microsoft.NET\Framewo"
01444438 "rk\v2.0.50727\Temporary ASP.NET "
01444478 "Files\scstt\57ae16cd\2f8637d5\as"
014444b8 "sembly\dl3\078f3d05\00fba208_c03"
014444f8 "6ca01\Microsoft.Practices.Enterp"
01444538 "riseLibrary.Logging.DLL"

So it’s a Microsoft .dll that is part of the Enterprise Library. Good to know. Now let’s find the thread that is attempting to retrieve the revocation list by running ~*kb and scanning through the threads until we find this one:

0:021> kb2000
ChildEBP RetAddr Args to Child
0427d7f8 776d9254 75502283 0000070c 00000001 ntdll!KiFastSystemCallRet
0427d7fc 75502283 0000070c 00000001 0427d824 ntdll!ZwWaitForSingleObject+0xc
0427d83c 75501fc8 0000070c 00000714 00000001 mswsock!SockWaitForSingleObject+0x19f
0427d928 77801693 00000001 00000000 0427d9d0 mswsock!WSPSelect+0x38c
0427d9a8 72a733e5 00000001 00000000 0427d9d0 ws2_32!select+0x494
0427e1ec 72a730fd 00000714 0427e210 72a72e88 winhttp!ICSocket::Connect_Start+0x3b0
0427e1f8 72a72e88 01828250 72a737bc 00000000 winhttp!CFsm_SocketConnect::RunSM+0x42
0427e210 72a7382b 018b0c60 00000000 00000000 winhttp!CFsm::Run+0x20
0427e234 72a7467d 01828250 018ccf30 0427e258 winhttp!DoFsm+0x2a
0427e244 72a74646 0000ea60 00000005 00000020 winhttp!ICSocket::Connect+0x32
0427e258 72a742b5 0000ea60 00000005 0000ea60 winhttp!ICSocket::Connect+0x13
0427e2a4 72a74038 01434458 0427e2c8 72a72e88 winhttp!HTTP_REQUEST_HANDLE_OBJECT::OpenConnection_Fsm+0x471
0427e2b0 72a72e88 01434458 00000000 00000000 winhttp!CFsm_OpenConnection::RunSM+0x37
0427e2c8 72a7382b 018b0c60 00000000 00000000 winhttp!CFsm::Run+0x20
0427e2ec 72a74625 01434458 018ccf30 0427e324 winhttp!DoFsm+0x2a
0427e2fc 72a745bc 00000000 00000000 018b0c60 winhttp!HTTP_REQUEST_HANDLE_OBJECT::OpenConnection+0x2f
0427e324 72a74526 0141f638 0427e348 72a72e88 winhttp!HTTP_REQUEST_HANDLE_OBJECT::MakeConnection_Fsm+0x9b
0427e330 72a72e88 0141f638 00000000 00000000 winhttp!CFsm_MakeConnection::RunSM+0x37
0427e348 72a7382b 018b0c60 00000000 00000000 winhttp!CFsm::Run+0x20
0427e36c 72a743c1 0141f638 018b0c60 014343c0 winhttp!DoFsm+0x2a
0427e3ac 72a74339 014343c0 0427e3d0 72a72e88 winhttp!HTTP_REQUEST_HANDLE_OBJECT::SendRequest_Fsm+0x93
0427e3b8 72a72e88 014343c0 018ccf30 00000000 winhttp!CFsm_SendRequest::RunSM+0x37
0427e3d0 72a7382b 018b0c60 00000000 00000000 winhttp!CFsm::Run+0x20
0427e3f4 72a73db0 014343c0 018b0c60 01888820 winhttp!DoFsm+0x2a
0427e414 72a76f71 00000000 0427e438 72a72e88 winhttp!HTTP_REQUEST_HANDLE_OBJECT::HttpSendRequest_Start+0x2af
0427e420 72a72e88 01888820 00000001 00000000 winhttp!CFsm_HttpSendRequest::RunSM+0x4c
0427e438 72a73c3e 018b0c60 00000000 00000000 winhttp!CFsm::Run+0x20
0427e480 72a73b8b 01888820 018ccf30 00000000 winhttp!StartFsmChain+0xcf
0427e4c4 72a739d8 018ccf30 00000000 00000000 winhttp!HttpWrapSendRequest+0x18c
0427e544 742599b4 018ccf30 00000000 00000000 winhttp!WinHttpSendRequest+0x19b
0427f5e4 74259741 005f7bc0 018ccf30 01898e60 cryptnet!InetSendAuthenticatedRequestAndReceiveResponse+0x56f
0427f6ec 7425921d 005f7bc0 01898e60 00202005 cryptnet!InetSendReceiveUrlRequest+0x2c8
0427f724 742523e9 01898e60 00000002 00202005 cryptnet!CInetSynchronousRetriever::RetrieveObjectByUrl+0x5f
0427f75c 742521a8 01898e60 00000002 00202005 cryptnet!InetRetrieveEncodedObject+0x64
0427f7b8 74256350 01898e60 00000002 00202005 cryptnet!CObjectRetrievalManager::RetrieveObjectByUrl+0xbb
0427f810 76ad4911 00000000 0427f85c 776be4b6 cryptnet!CryptRetrieveObjectByUrlWithTimeoutThreadProc+0x67
0427f81c 776be4b6 01898db0 4e080acf 00000000 kernel32!BaseThreadInitThunk+0xe
0427f85c 776be489 742562f7 01898db0 ffffffff ntdll!__RtlUserThreadStart+0x23
0427f874 00000000 742562f7 01898db0 00000000 ntdll!_RtlUserThreadStart+0x1b
0:021> du 01898e60
01898e60 "http://crl.microsoft.com/pki/crl"
01898ea0 "/products/CodeSignPCA2.crl"

This is the thread. To finally verify that it is in fact attempting to download the Certificate Revocation List we can check the value of the first argument to cryptnet!CInetSynchronousRetriever, InetRetrieveEncodedObject and CObjectRetrievalManager::RetrieveObjectByUrl or the second argument to cryptnet!InetSendReceiveUrlRequest as I did above.

Conclusion

In this case the customer chose to disable Signature Verification completely rather than granting Internet access to the server.

If you found the windbg-portion of this post hard to understand I can recommend a closer look at my Debugging School (In the menu to the left). Begin with Getting started with windbg - Part I and you should be set to go in no time.

/ Johan

Investigating Locks

JohanS — Fri, 09 Oct 2009 16:49:56 GMT

Consider the following scenario:

You have an ASP.NET application which intermittently responds sluggishly. As the problem occurs memory usage is about average, as is CPU usage, but still certain pages respond slower and slower. The machine acts just as if it is under heavy load, but judging from the CPU it isn't. In fact CPU usage might even decrease rather than increase. When trying to access non .NET content such as plain text/html or images you usually find that response times are just as fast as ever.

When you find that your application has hung in a low CPU state it is usually because of one of the following reasons (in order of likelihood based on my experience) :

External resources responding slowly. E.g. a database query or AD lookup is taking a long time to complete and so the application is simply waiting for another server to respond.
One of the threads of the application has entered a critical section and now several other threads are waiting to enter the same critical section.
A classic deadlock. Thread A waits for thread B which waits for thread C which waits for thread A.

Quite often you'll see a mixture of items 1 and 2. A thread enters a critical section, calls a remote database, or similar, and all other threads attempting to enter the same critical section will have to nicely stay in line until the struggling database has responded. Still, it is quite possible that the critical section is waiting for a local event to finish, but then you'd usually see a higher CPU load on the machine. (Though not necessarily in the w3wp.exe process.)

What are Critical Sections?

In case you've forgotten or simply not worked that much with critical sections it might be necessary to refresh your memory on this.

A Critical Section is a segment of code that you do not want to be executed more than one at a time. In order to ensure this you use the lock keyword to ensure that a certain block of code is able to complete with no interruption from any other threads. To illustrate this I've created the following code sample:

public class FileIO
{
    private static object myLock = new object();

	public FileIO()
	{
	}

    public string performLockOperation()
    {
        string strResult = "The Lock Operation started on " + System.DateTime.Now.ToLongTimeString();
        lock (myLock)
        {
            // Let the thread sleep for 10 seconds
            // This way we simulate a File IO operation that takes some time to complete.
            System.Threading.Thread.Sleep(10000);
        }
        strResult += " and ended on " + System.DateTime.Now.ToLongTimeString();
        return strResult;
    }

    public string performNoLockOperation()
    {
        string strResult = "The No Lock Operation started on " + System.DateTime.Now.ToLongTimeString();
        System.Threading.Thread.Sleep(10000);
        strResult += " and ended on " + System.DateTime.Now.ToLongTimeString();
        return strResult;
    }

}

Let's take a quick look at the parts of this class that I've written, called FileIO:

myLock

The class has a static variable called myLock that I will be using to identify the lock I'm about to create. This is why the variable is static. Otherwise I'd be creating new locks for each instance of the FileIO class leaving the lock operation completely redundant. Now that I use a static variable the lock spans all instances of the FileIO class.

constructor

The constructor is empty. Move along. Nothing to see here.

performLockOperation

This is the most interesting part of the class. It is the function we'll be using to actually reproduce the potential problems. The function returns a string which will contain information about when the call to the function was made and when it completed. We store this data in a string called strResult which we will eventually be returning as the result of the function call.

Having declared this variable and put the initial time of the function call in it we then create our lock. This is simply done with the following syntax:

lock (object)
{
}

Any code within this block will first make sure that there is not already a lock taken on this object. If there is, then they will pause execution and wait for the lock to become available before continuing execution.

Inside our lock we then simulate a slow File IO operation by letting the thread sleep for 10 seconds.

After that we append the current time to the result and return it.

performNoLockOperation

This function is identical to the previous one, except this time we do not have a critical section. This function is included just for comparison.

The potential problem

Now, let's get down to business. If we have a web page that makes a call to the performLockOperation-function and that page is requested at the same time by four different clients. What would be the response times?

The answer is 10, 20, 30 & 40 seconds.

Here's why:

The thread handling the first request enters the critical section, effectively locking it for everyone else. When it is complete 10 seconds have passed due to the Sleep-call.
One of the other threads enters the critical section once the first request is finished. It has already waited for 10 seconds, and the thread will now sleep for an additional 10 seconds.
As the second thread finishes one of the remaining two enters the critical section. It has now waited for 20 seconds and will sleep for an additional 10.
The final request is able to enter the critical section after having waited for 30 seconds. Once it is complete 40 seconds have passed.

If we'd been using the performNoLockOperation instead then all requests would have responded after 10 seconds.

Now, I know that 10 seconds is an exceptionally long time to let the thread sleep, but this was all for illustration purposes. Imagine instead that the sleeping time was 1 second and the number of requests was 200...

Troubleshooting this issue

Okay, so what would it look like when this problem occurs? Let's have a look. If you haven't looked at my posts on getting started with Windbg, then I suggest reading them before continuing. (Part I & Part II)

First of all I fire up WinDbg, then I attach to the w3wp.exe. I then use "~*e!clrstack" to look at the managed threads of the w3wp.exe application. Here is a snippet of what I get:

OS Thread Id: 0xce0 (24)
Child-SP         RetAddr          Call Site
000000000447e0f0 000007ff001e13d2 FileIO.performLockOperation()
000000000447e170 000007feee323ec9 _Default.Page_Load(System.Object, System.EventArgs)
000000000447e1b0 000007fee7956aea System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr, System.Object, System.Object, System.EventArgs)
000000000447e1e0 000007fee794d2c4 System.Web.Util.CalliEventHandlerDelegateProxy.Callback(System.Object, System.EventArgs)
000000000447e210 000007fee794d322 System.Web.UI.Control.OnLoad(System.EventArgs)
000000000447e250 000007fee79498ac System.Web.UI.Control.LoadRecursive()
000000000447e2a0 000007fee7948db0 System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)
000000000447e370 000007fee7948cdb System.Web.UI.Page.ProcessRequest(Boolean, Boolean)
000000000447e3d0 000007fee7948c70 System.Web.UI.Page.ProcessRequest()
000000000447e430 000007ff001e0c09 System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)
000000000447e490 000007fee7950117 ASP.default_aspx.ProcessRequest(System.Web.HttpContext)
000000000447e4c0 000007fee791449b System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
000000000447e570 000007fee7ffbd41 System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)
000000000447e610 000007fee7fed132 System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)
000000000447e7a0 000007fee7fcf599 System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)
000000000447e7f0 000007fee80f5344 System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)
000000000447e910 000007fee80f65bb System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
000000000447ea80 000007fee80f4994 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
000000000447eae0 000007fef72701ea DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)
000000000447f310 000007fee80f5424 DomainNeutralILStubClass.IL_STUB(IntPtr, System.Web.RequestNotificationStatus ByRef)
000000000447f3f0 000007fee80f65bb System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
000000000447f560 000007fee80f4994 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
000000000447f5c0 000007fef727043b DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)
OS Thread Id: 0x15ec (25)
Unable to walk the managed stack. The current thread is likely not a 
managed thread. You can run !threads to get a list of managed threads in
the process
OS Thread Id: 0x1958 (26)
Child-SP         RetAddr          Call Site
0000000004c1dd10 000007ff001e13d2 FileIO.performLockOperation()
0000000004c1dd90 000007feee323ec9 _Default.Page_Load(System.Object, System.EventArgs)
0000000004c1ddd0 000007fee7956aea System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr, System.Object, System.Object, System.EventArgs)
0000000004c1de00 000007fee794d2c4 System.Web.Util.CalliEventHandlerDelegateProxy.Callback(System.Object, System.EventArgs)
0000000004c1de30 000007fee794d322 System.Web.UI.Control.OnLoad(System.EventArgs)
0000000004c1de70 000007fee79498ac System.Web.UI.Control.LoadRecursive()
0000000004c1dec0 000007fee7948db0 System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)
0000000004c1df90 000007fee7948cdb System.Web.UI.Page.ProcessRequest(Boolean, Boolean)
0000000004c1dff0 000007fee7948c70 System.Web.UI.Page.ProcessRequest()
0000000004c1e050 000007ff001e0c09 System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)
0000000004c1e0b0 000007fee7950117 ASP.default_aspx.ProcessRequest(System.Web.HttpContext)
0000000004c1e0e0 000007fee791449b System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
0000000004c1e190 000007fee7ffbd41 System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)
0000000004c1e230 000007fee7fed132 System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)
0000000004c1e3c0 000007fee7fcf599 System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)
0000000004c1e410 000007fee80f5344 System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)
0000000004c1e530 000007fee80f65bb System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
0000000004c1e6a0 000007fee80f4994 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
0000000004c1e700 000007fef72701ea DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)
0000000004c1ef30 000007fee80f5424 DomainNeutralILStubClass.IL_STUB(IntPtr, System.Web.RequestNotificationStatus ByRef)
0000000004c1f010 000007fee80f65bb System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
0000000004c1f180 000007fee80f4994 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
0000000004c1f1e0 000007fef727043b DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)
OS Thread Id: 0x1e28 (27)
Child-SP         RetAddr          Call Site
000000000492dcf0 000007ff001e13d2 FileIO.performLockOperation()
000000000492dd70 000007feee323ec9 _Default.Page_Load(System.Object, System.EventArgs)
000000000492ddb0 000007fee7956aea System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr, System.Object, System.Object, System.EventArgs)
000000000492dde0 000007fee794d2c4 System.Web.Util.CalliEventHandlerDelegateProxy.Callback(System.Object, System.EventArgs)
000000000492de10 000007fee794d322 System.Web.UI.Control.OnLoad(System.EventArgs)
000000000492de50 000007fee79498ac System.Web.UI.Control.LoadRecursive()
000000000492dea0 000007fee7948db0 System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)
000000000492df70 000007fee7948cdb System.Web.UI.Page.ProcessRequest(Boolean, Boolean)
000000000492dfd0 000007fee7948c70 System.Web.UI.Page.ProcessRequest()
000000000492e030 000007ff001e0c09 System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)
000000000492e090 000007fee7950117 ASP.default_aspx.ProcessRequest(System.Web.HttpContext)
000000000492e0c0 000007fee791449b System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
000000000492e170 000007fee7ffbd41 System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)
000000000492e210 000007fee7fed132 System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)
000000000492e3a0 000007fee7fcf599 System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)
000000000492e3f0 000007fee80f5344 System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)
000000000492e510 000007fee80f65bb System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
000000000492e680 000007fee80f4994 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
000000000492e6e0 000007fef72701ea DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)
000000000492ef10 000007fee80f5424 DomainNeutralILStubClass.IL_STUB(IntPtr, System.Web.RequestNotificationStatus ByRef)
000000000492eff0 000007fee80f65bb System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
000000000492f160 000007fee80f4994 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
000000000492f1c0 000007fef727043b DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)
OS Thread Id: 0x1c38 (28)
Child-SP         RetAddr          Call Site
0000000004d0e0b0 000007ff001e13d2 FileIO.performLockOperation()
0000000004d0e130 000007feee323ec9 _Default.Page_Load(System.Object, System.EventArgs)
0000000004d0e170 000007fee7956aea System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr, System.Object, System.Object, System.EventArgs)
0000000004d0e1a0 000007fee794d2c4 System.Web.Util.CalliEventHandlerDelegateProxy.Callback(System.Object, System.EventArgs)
0000000004d0e1d0 000007fee794d322 System.Web.UI.Control.OnLoad(System.EventArgs)
0000000004d0e210 000007fee79498ac System.Web.UI.Control.LoadRecursive()
0000000004d0e260 000007fee7948db0 System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)
0000000004d0e330 000007fee7948cdb System.Web.UI.Page.ProcessRequest(Boolean, Boolean)
0000000004d0e390 000007fee7948c70 System.Web.UI.Page.ProcessRequest()
0000000004d0e3f0 000007ff001e0c09 System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)
0000000004d0e450 000007fee7950117 ASP.default_aspx.ProcessRequest(System.Web.HttpContext)
0000000004d0e480 000007fee791449b System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
0000000004d0e530 000007fee7ffbd41 System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)
0000000004d0e5d0 000007fee7fed132 System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)
0000000004d0e760 000007fee7fcf599 System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)
0000000004d0e7b0 000007fee80f5344 System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)
0000000004d0e8d0 000007fee80f65bb System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
0000000004d0ea40 000007fee80f4994 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
0000000004d0eaa0 000007fef72701ea DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)
0000000004d0f2d0 000007fee80f5424 DomainNeutralILStubClass.IL_STUB(IntPtr, System.Web.RequestNotificationStatus ByRef)
0000000004d0f3b0 000007fee80f65bb System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
0000000004d0f520 000007fee80f4994 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
0000000004d0f580 000007fef727043b DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32

So, what does this tell us?

Well, obviously we have four threads that call my function FileIO.performLockOperation() from Page_Load.

Please note that we don't have any additional callstack, and since the performLockOperation-function was named by me it could just as well have been named "foo". In order to find out that these threads are actually waiting for a lock we need to look at the native callstack using the kb-command. Let's take a look at thread 24:

0:029> ~24kb
ntdll!NtDelayExecution+0xa
kernel32!SleepEx+0x84
mscorwks!EESleepEx+0x2d
mscorwks!Thread::UserSleep+0x71
mscorwks!ThreadNative::Sleep+0xf9
App_Code_tvh3usvo!FileIO.performLockOperation()+0x8a
App_Web_uywrbugj!_Default.Page_Load(System.Object, System.EventArgs)+0x32
System_Web_RegularExpressions_ni!System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr, System.Object, System.Object, System.EventArgs)+0x19
System_Web_ni!System.Web.Util.CalliEventHandlerDelegateProxy.Callback(System.Object, System.EventArgs)+0x2a
System_Web_ni!System.Web.UI.Control.OnLoad(System.EventArgs)+0x84
System_Web_ni!System.Web.UI.Control.LoadRecursive()+0x42
System_Web_ni!System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)+0x97c
System_Web_ni!System.Web.UI.Page.ProcessRequest(Boolean, Boolean)+0xa0
System_Web_ni!System.Web.UI.Page.ProcessRequest()+0x5b
System_Web_ni!System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)+0xf0
App_Web_uywrbugj!ASP.default_aspx.ProcessRequest(System.Web.HttpContext)+0x9
System_Web_ni!System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()+0x257
System_Web_ni!System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)+0xab
System_Web_ni!System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)+0x501
System_Web_ni!System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)+0x72

Apparently we entered FileIO.performLockOperation, and then called sleep, (which is what we're doing in this repro in order to simulate a slow IO operation.) So, what do the other threads look like? Here's thread 26:

0:029> ~26kb
ntdll!ZwWaitForMultipleObjects+0xa
kernel32!WaitForMultipleObjectsEx+0x10b
mscorwks!WaitForMultipleObjectsEx_SO_TOLERANT+0xc1
mscorwks!Thread::DoAppropriateAptStateWait+0x41
mscorwks!Thread::DoAppropriateWaitWorker+0x191
mscorwks!Thread::DoAppropriateWait+0x5c
mscorwks!CLREvent::WaitEx+0xbe
mscorwks!AwareLock::EnterEpilog+0xc9
mscorwks!AwareLock::Enter+0x72
mscorwks!AwareLock::Contention+0x1fb
mscorwks!JITutil_MonContention+0xdf
App_Code_tvh3usvo!FileIO.performLockOperation()+0x7f
App_Web_uywrbugj!_Default.Page_Load(System.Object, System.EventArgs)+0x32
System_Web_RegularExpressions_ni!System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr, System.Object, System.Object, System.EventArgs)+0x19
System_Web_ni!System.Web.Util.CalliEventHandlerDelegateProxy.Callback(System.Object, System.EventArgs)+0x2a
System_Web_ni!System.Web.UI.Control.OnLoad(System.EventArgs)+0x84
System_Web_ni!System.Web.UI.Control.LoadRecursive()+0x42
System_Web_ni!System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)+0x97c
System_Web_ni!System.Web.UI.Page.ProcessRequest(Boolean, Boolean)+0xa0
System_Web_ni!System.Web.UI.Page.ProcessRequest()+0x5b

This is one of the threads waiting for the critical section. It then found that it was unable to get a lock and so began waiting for the critical section on thread 24 to end.

Resolving the problem

This is the tricky part. There is no quick-fix for this. We can use the debugger to identify the bottleneck, but after that we'll need to review our design in order to get this working smoothly. A couple of generic things to try would be the following:

Review what you're doing inside the critical section. Is it really necessary to perform all of the operations you're doing, or would it be possible to do some/all of them outside the scope of the critical section?
Identify the reason for the lock taking so much time. Can it be resolved by load-balancing or clustering the resource we're waiting for?
If we're waiting for another server, make sure the physical infrastructure is okay. Are network response times within the expected parameters or do you need to fix your wiring, router, etc?
Are all calls to the critical section taking the same amount of time, or are there certain calls that take an exceptional amount of time? Perhaps most calls to the function are just fine, but intermittently a call comes along that tries to get everything from the AD, database, etc. In order to investigate this I'd use the commands outlined in Getting started with windbg -part II. Identify the thread owning the lock and investigate the callstack and stack objects using !clrstack and !dso. Use !dso and !do to query the stack objects and try to find the SQL query or AD query that is causing the thread to take such a long time to finish. Is the query significantly different from the ones waiting to execute, then this is most likely our culprit.

Post Script

A quick thought on locks and their limits:

Locks do not scale well. A lock can only be made within a specific process, so if you're using a web farm, or in any other way more than one w3wp.exe, then you will not be able to lock all processes. In other words. Server 1 can get a lock, and so can Server 2. Both will happily execute their code independent of each other. For products such as SQL server you'd deal with this in your SQL query, but if you're performing other operations, such as File IO you might have to work around this by using the same approach as Ms Word. Create a lock file in a shared space, and before you enter your critical section you simply check if the file exists:

You are entering your critical section. Look for lock file in shared location
If file exists, then wait
If/when file doesn't/no longer exist, create file and enter critical section
Make sure you have solid exception handling within the critical section so that you under no circumstances leave the critical section without removing the lock file.
When done, remove lock file

/ Johan

Why doesn’t the GC kick in when the worker process is inactive?

JohanS — Tue, 13 May 2008 16:39:48 GMT

I got the following question in my Getting started with windbg – post and I thought it might be worth posting the replies in a separate article:

Hi Johan,

about those threads with an ID of XXXX, should they go away after certain amount of idle time like 2 minutes? I am trouble shooting an application while the application should sit idle (because of no stimulation), however it is still using quite some CPU time and and threadpool threads, where it should not. So I created an adplus dump when it is sitting idle, and I found that there are many threads with ID XXXX, and I created a dump file again after 15 minutes, again, it still have the exact same XXXX threads. All those threads are completion port threads, and I wonder why they did not get recycled?

Also, when doing a '~*e !clrstack', most of the worker threads and completion port threads are showing "Failed to start stack walk: 80004005". Is there a way to show the stack for those threads, because those are the threads I am interested in.

Thank you!

The GC will be triggered when you’re allocating memory. If your application is inactive, then so is the Garbage Collector. Even if you’re making a few, random allocations they still may not be enough to trigger a GC, so this is not at all unexpected.

The “Failed to start stack walk: 80004005”-error is displayed when the thread did contain a managed stack, but no longer does.

/ Johan

What to do about the slow startup of web services

JohanS — Wed, 02 Apr 2008 18:05:16 GMT

Due to the architecture of web services and web applications they can be quite slow to start. For example on my Windows 2003-box the initial localhost-call to a simple “Hello World!”-web service takes approximately 8 seconds, while the next request is more or less immediate.

Why is this?

This isn’t news, really. One of the first things we learned in the early beta stages of ASP.NET was that the first request would take a little extra time due to the Just In Time Compilation (JIT). This was a major change from the classic ASP architecture where everything was interpreted rather than compiled. Upon the first request the .aspx / .asmx file will be compiled (JIT’ed) into Microsoft Immediate Language (MSIL) and the resulting .dll will be moved to it’s proper location.

This compilation will occur every time the application pool starts causing a significantly longer response time on the first request compared to the following requests.

Run-time performance over startup performance

It all comes down to prioritizing run-time performance over a quick startup. Which, in my humble opinion, is a sound choice. There are, however, situations where you may feel that this is to your disadvantage. The other day I got a question from a customer who had a web service that was called quite infrequently and to him this meant that with every other request response times would be horrendous, simply because the application got recompiled.

For each case I get I usually do a quick search on the problem description. I don’t really expect to find the solution this way, since I can only assume that the customer has tried this as well. Instead it will likely present me with a list of troubleshooting steps that the customer has already tried as well as possible solutions that didn’t fit. (Off course I still need to verify this with the customer).

This time I didn’t really need to do this since I was pretty sure what the cause was. Still, I was curious to see how common this problem was. I honestly didn’t think this was a very common scenario, but my quick search on the web revealed the contrary. The few hits that I quickly browsed through suggested writing an application that would ping the web service in order to keep it alive. I must admit I find this quite inventive, but I really don’t think it’s the best approach. :)

What to do

So, what is to be done about this? Well, I’ve already mentioned that there are better solutions than pinging the web service. As I told my customer there are basically two good options:

Precompile the assemblies
Change the settings for automatic recycling on the worker process

Precompiling the assemblies

There is a nice little tool called Native Image Generator. (ngen.exe) It will compile the assembly into native images and installs them into the native image cache. The syntax is pretty straight-forward: ngen.exe install [assembly] The problem is that once you’ve changed the assembly you’d have to precompile it again making updates a bit tedious.

There is also a service-version of the ngen called the Native Image Service. More information about ngen can be found at http://msdn2.microsoft.com/sv-se/magazine/cc163808(en-us).aspx

Changing the recycling settings

The reason why my customer was experiencing this problem was because his application pool would time out and recycle itself due to the default settings in the IIS manager. By default the worker process will recycle after 20 minutes of inactivity, so if no one has pinged your application in that time the next request will cause the application to recompile, resulting in a longer response time. By tweaking this setting in the IIS manager to something more suitable to the current rate of requests, like 40 minutes or maybe even 60 he should also be able to find a nice balance. Actually, considering the low load it would probably be a good idea to turn this setting off completely. Here’s how:

Open up IIS Manager
Locate the Application Pool in question
Right-click it and select “Properties”
Go to the “Performance”-tab
Change, or disable the “Idle timeout” setting

/ Johan

Monitoring Application Pool and Application Restarts

JohanS — Mon, 18 Feb 2008 17:12:00 GMT

This morning I found the following in my inbox:

I had set my web servers running on IIS 6 to recycle if they hit 700 MB (Maximum used Memory).
Can I run a report to know how many times per day or week the W3WP got recycled.
Any suggestions please.

Thanks in anticipation

The quick answer to your question is to use performance monitor:

Start performance monitor
Add a new counter
Performance object: "ASP.NET v[n]"
Counter: "Worker Process Restarts"

So, there's the answer. But while we're on the subject I'd like to mention application restarts as well.

Applications vs. Application pools

Application restarts may lead you think that your entire application pool recycled, while in fact it was only one of the applications hosted by the application pool that restarted. If this is getting confusing, then please remember that there's a difference between application pools and application. An application pool will consist of one or several worker processes and may host one or several applications.

There are two easy ways of monitoring application restarts:

Using performance monitor again
Have ASP.NET write an event to the event log upon shutdown & startup. (Requires ASP.NET 2.0 or later)

Performance monitor

This is probably the easiest, though you won't get too much additional information other than the fact that an application restart occurred. Here's what you do:

Start performance monitor
Add a new counter
Performance object: "ASP.NET v[n]"
Counter: "Application Restarts"

Event log

By altering the root web.config you can get an event for every application shutdown and start up. This is a good way to get more detailed information on why the application shut down.

Open up the root web.config, (located in the %WinDir%\Microsoft.NET\Framework\v2.0.50727\CONFIG directory,) locate the healthMonitoring.rules subkey and add the following:

Now when the application exited for an application-specific reason you'll get an event like this:

Event code: 1002 
Event message: Application is shutting down. Reason: Configuration changed. 
Event time: 2/14/2008 10:00:41 AM 
Event time (UTC): 2/14/2008 9:00:41 AM 
Event ID: a1314c10a0c84222ae2d870d85308304 
Event sequence: 18 
Event occurrence: 1 
Event detail code: 50004 
 
Application information: 
    Application domain: /LM/w3svc/1/ROOT/Test-1-128474532435626182 
    Trust level: Full 
    Application Virtual Path: /Test
    Application Path: c:\inetpub\wwwroot\Test\ 
    Machine name: JOHAN

As you can see the application shut down because the configuration changed. Note that you won't get an event if you manually kill the entire application pool in IIS manager, or similar. One thing that you will get, however, is the following event each and every time the application starts up again:

Event code: 1001 
Event message: Application is starting. 
Event time: 2/14/2008 10:00:47 AM 
Event time (UTC): 2/14/2008 9:00:47 AM 
Event ID: 1f41fd3b17764330ac61804094b0abf0 
Event sequence: 1 
Event occurrence: 1 
Event detail code: 0 
 
Application information: 
    Application domain: /LM/w3svc/1/ROOT/Test-1-128474532435626182 
    Trust level: Full 
    Application Virtual Path: /Test
    Application Path: c:\inetpub\wwwroot\Test\ 
    Machine name: JOHAN

So even if the application shut down for a reason that didn't generate an event, (IISReset, idle server, etc.) you'll at least see that for some reason it had to start up again.

/ Johan

Exceptions

JohanS — Fri, 18 Jan 2008 18:50:22 GMT

Introduction

This was originally intended to be a post on identifying and troubleshooting Exceptions using windbg. As I started writing I thought I should write a little something about exceptions in general, just to begin the post. That little something grew to be quite big and all of a sudden it was by all means worthy of a post of its own.

There are two scenarios that are exceptionally (no pun intended) common in my line of work:

Clients are reporting 2nd chance exceptions displayed on screen with the classic "Server Error"- page.
Performance is generally bad, and when we investigate it turns out that there are tons of exceptions being thrown every second.

What exactly is an exception?

When it comes to the .NET Framework an exception is a managed object that inherits from the Exception Class. It is usually created when something's gone wrong either in the application or the runtime. The exception contains detailed information about the error that created it such as a friendly error message, the stack trace leading to this exception being created, possible inner exceptions (exceptions that caused the exception) and more. In other words they contain a lot more information than a function simply returning a status code other than 0 if something went wrong. This makes it a lot easier for us to troubleshoot the application.

The downside is that this is resource-consuming. All the information available needs to be gathered and so throwing an exception is a lot more costly than returning an error code.

Exceptions must be handled

When an exception is thrown it must be handled one way or another or the application/web request will crash/fail. This is done by setting up a try/catch block.

   try {
     conn.Close();
   }
   catch(Exception ex) {
     // Something went wrong.
   }

If anything goes wrong within the try block, the catch block can now deal with it and act appropriately. It is still up to you to code a suitable reaction to the exception, but the interesting thing is that you need to deal with the problem. If the exception is unattended it will cause the application to crash (winforms) or return a "Server Error"-page (IIS).

This is really good, because this way you will know, one way or the other, that something has gone wrong. Even if you forget to add a try/catch block the exception will still be thrown. This will possibly cause the application to crash, but that's a much better option than continuing execution as if nothing happened.

Exceptions should be avoided

This is really important. To throw an exception is costly. Chris Brumme has written an excellent post on this subject and in it he writes about some of the things that the runtime does when throwing an exception. Such as:

Grab a stack trace by interpreting metadata emitted by the compiler to guide our stack unwind.
Run through a chain of handlers up the stack, calling each handler twice.
Compensate for mismatches between SEH, C++ and managed exceptions.
Allocate a managed Exception instance and run its constructor. Most likely, this involves looking up resources for the various error messages.
Probably take a trip through the OS kernel. Often take a hardware exception.
Notify any attached debuggers, profilers, vectored exception handlers and other interested parties.

Exceptions are there to tell you that something's gone wrong, but since they're quite costly to throw it would be much better if you anticipated them and tried to minimize them.

Why use the following code:

    void calculateAvg(Int32 X, Int32 Y)
    {
        try
        {
            Label1.Text = (X / Y).ToString();
        }
        catch (Exception e)
        {
            Label1.Text = "Division by Zero";
        }
    }

when you can just as well use this:

    void calculateAvg(Int32 X, Int32 Y)
    {
        if (Y != 0)
        {
            Label1.Text = (X / Y).ToString();
        }
        else
        {
            Label1.Text = "Division by Zero";
        }

    }

As you can see, the first piece of code will throw an exception if Y happens to be zero. And all to no use. Take a look again at all the things we go through when throwing an exception and consider if that is really necessary when all you needed to do was to verify that Y != 0.

How many exceptions is your application throwing per second?

In a perfect world you should be able to answer this without hesitation. Anyway, if you honestly have no idea then I suggest you start Performance Monitor, add a counter for .NET CLR Exceptions/# of Exceps Thrown and run a stress test.

I've come across applications that throw over 150.000 exceptions in less than an hour.

There may also be times when this is actually expected and acceptable. For example, the Response.Redirect command results in a System.Threading.ThreadAbortException. If your application relies heavily on Response.Redirect for some reason, then you needn't worry. If, however, you're not aware of what is causing these exceptions I'd urge you to look into it.

None of us would want to let our systems run with constant red or yellow warnings in the event logs. The least we'd do is check up on them and see what caused them.

Well, having written this little introduction I guess I should now get going on the "real" post. The one about how to find and troubleshoot those exceptions using Windbg...

Later! / Johan

The KB article every one should know about

JohanS — Wed, 23 May 2007 18:23:46 GMT

Have you read http://support.microsoft.com/?kbid=307340? If not, I suggest you do so. If you need convincing or simply want to know why this problem occurs I suggest you keep reading.

The problem described in the article above can cause your application to spike in CPU time, Memory usage or both. This will lead to your application hanging or possibly even crashing. The cause of the problem is one of the most basic operations, and is something that we do very frequently.

The problem

The problem in a nutshell is that string concatenation is bad. I don't mean bad as in "You shouldn't eat that extra piece of chocolate". I mean it as in "You really shouldn't put your head in wet concrete". If you've been reading my previous post on memory management you might be able to figure out why string concatenation could be such a potential hazard. Consider the following scenario:

You're getting ready to go grocery shopping. You take a sheet of paper and a pen and begin writing down what you need to buy. The first item that crosses your mind is milk, so you write it down. Now you remember that you're out of bread as well, so you throw away the first sheet of paper, pick out a new one and write down milk and bread. You then throw away this paper too and write down milk, bread and apples on a third piece of paper. You throw away the third piece of paper and repeat until finally the list is done. If you have 20 items on your list you now have 19 discarded papers lying on the floor waiting for the "garbage collector".

This is exactly how regular string concatenation works in .NET. Every time you concatenate a string the framework needs to allocate a new memory segment, large enough to hold the result of the concatenation and store the string in this new segment. The old segment is flagged as ready for garbage collection and as we all know garbage collection costs CPU and pending garbage collection costs memory.

A real example

So if you thought that 19 sheets of paper were a waste consider the following piece of code:

private String buildTable()
{
   String sReturn;
   int i, j;
   sReturn = "<table>"
   for (i = 0; i < 35; i++) // Rows
   {
      sReturn += "<tr>"
      for (j = 0; j < 15; j++) // Columns
      {
         sReturn += "<td>"
         sReturn += i.ToString() + " : " + j.ToString();
         sReturn += "</td>"
      }
      sReturn += "</tr>"
   }
   sReturn += "</table>"
   return sReturn;
}

All in all we have 2697 concatenations which means we have 2696 unnecessary allocations. The allocations will be gradually bigger and bigger, so the last ones will be almost as big as the final table. Needless to say this will consume a lot of unnecessary memory. Also, if this table was built for a "real" application it would probably contain a lot more data for each cell, not to mention custom styles and classes. Imagine what would happen when the string goes past the magic 85000 bytes barrier that defines what goes on the large object heap and what doesn't. In a larger scale this would have a severe impact on performance.

This is a really weird behavior, how can it be?!

It's not that odd really. If we extend the grocery shopping analogy a little we can see why there's a logical reason for this: You know that you want to write something down. You have a limited amount of paper, so what do you do? You cut out a piece of paper just large enough for you to write what you need. When you find that you need to add another item to the list you'll have to discard the old piece and cut out a new one. Not very practical, agreed, but you're trying to minimize your paper usage.

As the clever beings that we are we normally make a rough estimate of how big a piece of paper we need and then start writing. The framework, however, has no way of estimating how big a string is going to be. It doesn't know if you're going to concatenate it zero or ten thousand time, so that's something you'll have to estimate yourself.

So what should I do instead?

One word: StringBuilder

The StringBuilder class is made especially for this type of operations. When using the StringBuilder class our sample above would look something like this

private String buildTableSB()
{
   StringBuilder sReturn = new StringBuilder();
   sReturn.Capacity = 8000; // This is optional, but may be very efficient
   int i, j;
   sReturn.Append("<table>");
   for (i = 0; i < 35; i++) // Rows
   {
      sReturn.Append("<tr>");
      for (j = 0; j < 15; j++) // Columns
      {
         sReturn.Append("<td>");
         sReturn.Append(i.ToString());
         sReturn.Append(" : ");
         sReturn.Append(j.ToString());
         sReturn.Append("</td>");
      }
      sReturn.Append("</tr>");
   }
   sReturn.Append("</table>");
   return sReturn.ToString;
}

The difference in performance for a larger operation is enormous. I highly recommend trying out these samples. Copy the functions to a winforms/webforms application, increase the number of rows to 500 and run both versions. The StringBuilder version will finish within milliseconds, and the "classic" version will not... For additional tidbits, take a look at the GC counter in performance monitor and look at how many GCs you have to make to complete the operation.

Bye for now / Johan

Common reasons why your application pool may unexpectedly recycle

JohanS — Wed, 16 May 2007 11:12:27 GMT

If your application crashes, hangs and deadlocks it will cause/require the application pool to recycle in order to be resolved, but sometimes your application pool inexplicably recycles for no obvious reason. This is usually a configuration issue or due to the fact that you're performing file system operations in the application directory.

For the sake of elimination I thought I'd list the most common reasons.

Application pool settings

If you check the properties for the application pool you'll see a number of settings for recycling the application pool. In IIS6 they are:

Recycle worker processes (in minutes)
Recycle worker process (in requests)
Recycle worker processes at the following times
Maximum virtual memory
Maximum used memory

These settings should be pretty self explanatory, but if you want to read more, please take a look at this MSDN article

The processModel element of machine.config

If you're running IIS5 or the IIS5 isolation mode you'll have to look at the processModel element. The Properties you should pay the closest attention to are:

memoryLimit
requestLimit
timeout

memoryLimit

The default value of memoryLimit is 60. This value is only of interest if you have fairly little memory on a 32 bit machine. 60 stands for 60% of total system memory. So if you have 1 GB of memory the worker process will automatically restart once it reaches a memory usage of 600 MB. If you have 8 GB, on the other hand, the process would theoretically restart when it reaches 4,8 GB, but since it is a 32 bit process it will never grow that big. See my post on 32 bit processes for more information why.

requestLimit

This setting is "infinite" by default, but if it is set to 5000 for example, then ASP.NET will launch a new worker process once it's served 5000 requests.

timeout

The default timeout is "infinite", but here you can set the lifetime of the worker process. Once the timeout is reached ASP.NET will launch a new worker process, so setting this to "00:05:00" would recycle the application every five minutes.

Other properties

There are other properties within the processModel element that will cause your application pool to recycle, like responseDeadlockInterval. But these other settings usually depend on something going wrong or being out of the ordinary to trigger. If you have a deadlock then that's your main concern. Changing the responseDeadlockInterval setting wouldn't do much to resolve the situation. You'd need to deal with the deadlock itself.

Editing and updating

ASP.NET 2.0 depends on File Change Notifications (FCN) to see if the application has been updated. Depending on the change the application pool will recycle. If you or your application is adding and removing directories to the application folder, then you will be restarting your application pool every time, so be careful with those temporary files.

Altering the following files will also trigger an immediate restart of the application pool:

web.config
machine.config
global.asax
Anything in the bin directory or it's sub-directories

Updating the .aspx files, etc. causing a recompile will eventually trigger a restart of the application pool as well. There is a property of the compilation element under system.web that is called numRecompilesBeforeAppRestart. The default value is 20. This means that after 20 recompiles the application pool will recycle.

A workaround to the sub-directory issue

If your application really depends on adding and removing sub-directories you can use linkd to create a directory junction. Here's how:

Create a directory you'd like to exclude from the FCN, E.g. c:\inetpub\wwwroot\WebApp\MyDir
Create a separate folder somewhere outside the wwwroot. E.g. c:\MyExcludedDir
use linkd to link the two: linkd c:\inetpub\wwwroot\WebApp\MyDir c:\MyExcludedDir
Any changes made in the c:\inetpub\wwwroot\WebApp\MyDir will actually occur in c:\MyExcludedDir so they will go unnoticed by the FCN.

Is recycling the application pool really that bad?

You really shouldn't have to recycle the application pool, but if you're dealing with a memory leak in your application and need to buy time to fix it, then by all means recycling the application pool could be a good idea.

What about session state?

Well, if you're running in-process session state, then obviously it's going to be reset each and every time the application pool is recycled. If you need to brush up on your state server options, then I recommend taking a look at this entry.

Later! / Johan

Finalizers and weak references

JohanS — Thu, 26 Apr 2007 17:25:57 GMT

What do finalizers and weak references have in common? Well more than you might think actually.

Finalizers

Finalizers are clean-up code that will be run at the end of an objects life-cycle. You should only release native resources in the finalizer. When you use your object you should not the finalizer cleaning up after you. You're much better off performance-wise by making your object Disposable and making sure that you're actually calling the Dispose method. The finalizers should only be the final fail-safe in case the Dispose call is forgotten.

Having a finalizer means your object is always promoted at least one generation by the Garbage Collector (GC). Here's why:

You have an object with a finalizer and the object is no longer referenced
The GC comes to collect the object. It sees that the object has a finalizer. The GC therefore adds the object to the finalizer queue and promotes it one generation on the heap
The finalizer is run, once the finalizer thread is available. (If this takes too long the object might actually be promoted a second time.)
The object is finally garbage collected

Since you only have one finalizer thread and rely heavily on finalizers you may be creating an unintentional bottleneck.

So shouldn't I use finalizers?

Well sure you should. But as I've said before, only regard them as a fail-safe. Make your objects disposable and add a finalizer just in case somebody less skilled than yourself forgets to call the Dispose method. :-)

Weak References

A weak reference is different from a strong reference since this reference is disregarded when the GC comes to collect the object, so if there are nothing but weak references remaining for an object when the heap is GC' d the object will be collected. Weak references are generally useful in two ways:

1. Poor mans cache

If you have a weak reference to an object, say a dataset that takes some time putting together, then you can still work with that object as long as it hasn't been garbage collected. You would work with the reference as you would with any cached item. First you check if the object still exists, if it does, then you get a strong reference to the object and work with it. If not, then you create a new dataset and and work with that one.

2. Custom finalizer

Consider the following scenario. (No, you're not having Déjà Vu. This is the same scenario I described in the finalizers section above.):

You have an object with a finalizer and the object is no longer referenced
The GC comes to collect the object. It sees that the object has a finalizer. The GC therefore adds the object to the finalizer queue and promotes it one generation on the heap
The finalizer is run, once the finalizer thread is available. (If this takes too long the object might actually be promoted a second time.)
The object is finally garbage collected

Here's an alternative way of doing the same thing:

You don't use a finalizer. Instead you have an array of weak references to objects that you occasionally check
When you find that one of the weak references' target is null that means the GC has collected that object
The object has been garbage collected, so now you run your own clean-up code.

Okay, so the second alternative has two obvious benefits compared to the first:

Your object won't be automatically promoted to generation 1 or generation 2.
You won't be depending on the finalizer thread being available to run your clean-up code.

The ultimate option would off course be to have your object being Disposable, make sure you call the Dispose method and use alternatives one or two as back-ups.

Long Weak References

Long weak references will survive the finalizer, so even if your object has been finalized you can still reference it. This may be useful in certain situations, but it can be risky if you're not 100% sure what the finalizer does. Let's say your object performs some kind of file I/O and all file references are closed by the finalizer. If you reference this object using a Long Weak Reference you will have an instance of the object with all I/O functionality disabled. Obviously this could cause all sorts of exceptions if you don't know what you're doing.

Myths and Facts

I've previously gotten some positive feedback on the Myth / Fact format, so I thought I'd round this topic off by adding a summary using that format. Here goes:

Myth:
You can pretty much do anything in the finalizer.

Fact:
The only thing you should do in the finalizer is to release your native resources. The reason why your doing this is because you can never be sure that the person using your object is smart enough to call the Dispose method. If everybody did do that, then we wouldn't need the finalizer at all

Myth:
If you have a finalizer you don't need to make your component disposable

Fact:
You should always call the Dispose method of any disposable object. It doesn't matter if all it seems to do is set GC.SuppressFinalize(Me). That is no guarantee that the next version of the object won't do all kinds of stuff in the Dispose method. The rule of thumb is: If it is disposable, dispose it.

Myth:
Having a finalizer has no impact on performance. When the Garbage Collector (GC) comes to collect it the GC will simply call the Finalize method first and then collect it.

Fact:
Having a finalizer is a guarantee that your object will be promoted one generation before it is collected by the GC. When the GC comes across an object with a finalizer that is ready to be collected it puts the object in the finalizer queue, but the object is then moved to the next generation. For more information on this, please see my previous post on the managed heap, the GC and its generations. You also have only one finalizer thread, so this has bottleneck warning written all over it.

Myth:
Long Weak References are available as long as the object is in the finalizer queue, but they're released once the Finalize method is called.

Fact:
Nope. They are available when they're in the finalizer queue, but they're also available after the finalizer has been called. You really should be aware of this or you might get the strangest results.

Jedi skills

If you want to know more about finalizers, weak references, the GC and other goodies I highly recommend checking out Maoni's WebLog. What better place to learn about this than from one of the developers?

Over and out

/ Johan

Memory management in the .NET Framework

JohanS — Fri, 20 Apr 2007 16:07:29 GMT

This is a subject that has been covered before and I have no intention of writing the ultimate post on the subject. Still I think this is something that every good developer should know.

Why do I need to know this?

My colleagues and I are quite often asked about the necessity of knowing how the Garbage Collector (GC) works. After all, it isn't really necessary to know what's going on behind the scenes, right..? A developer shouldn't have to bother about how the framework works..? Or should he..?

Well you may be a pretty good bus driver even if you know nothing about engines. Basic stuff like not stepping on the break and the gas at the same time is probably enough. If you want to do F1 racing, on the other hand, you will want to know as much as you can about your car. So, if you're content doing small scale web applications or not so efficient winforms applications, then you probably don't have to bother. If you want to do front-line work, then you've got to get your hands dirty.

The different generations

In order to be effective the managed heap is trying to figure out which objects it can disregard and only check up on occasionally. Some objects may span the entire lifecycle of the application and others may be very short lived, so if an object stays alive for a longer period of time the GC will check up on it less frequently. This is accomplished by dividing the managed heap into three generations. Generations 0, 1 and 2. An object will begin in generation 0 and if it is in use for a long period of time it will travel through generations 0 to 2. A possible metaphor for this scenario would the following:

You're asked to call a customer. You're pretty good with numbers so you keep the phone number in your head. (Generation 0)
There is no reply and so you need to remember the phone number a little longer. Meanwhile you're given additional numbers to call.
You know from past experience that your limit for keeping numbers in your head is 10, so once you reach that critical point you decide to write down the numbers you still need on post-it notes (Generation 1)
As the day goes on you keep transferring numbers from memory to post-its.
Your desktop has now reached maximum capacity. It is cluttered with post-it notes, so you sort through them, determining which notes you still need. You throw away the ones that are obsolete and write down the remaining to your address book. (Generation 2)

Like I stated above; when an object is created it is put in Generation 0. The initial size limit of Generation 0 is determined by the size of the processor cache. This is dynamically changed depending on the allocation rate of the application. Once Generation 0 reaches its limit it will go through all the items in Generation 0, tag the obsolete objects for collection and remove them. This is called the Mark and Sweep phase. Everything that survives this sweep will be compacted and moved to Generation 1. The size limit of Generation 1 is also determined by the allocation rate of the application, and once it is reached a Generation 1 collection will occur. This will first mark and sweep all items in Generation 1, moving the surviving items to Generation 2, and then mark and sweep the items in Generation 0.

The healthy ratio between GC's in the different generations is approximately 100 - 10 - 1, so for 100 Generation 0 GC's you normally have 10 GC's of Generation 1 and 1 of Generation 2. A normal Generation 0 GC usually takes a few milliseconds. Performing a GC of Generation 1 rarely takes more than 30 milliseconds, but a Generation 2 collection can take quite some time depending on the application.

The Large Object Heap

To top it off we also have what is called "The Large Object Heap" (LOH). This is where anything larger than 85.000 bytes will be stored. The LOH is collected each time a new segment needs to be reserved (see below), but it is not compacted like generations 0, 1 and 2. When you perform a collection on the LOH you will also perform a GC on the other generations.

Now, if the limit is at 85.000 bytes, won't most of my objects end up on the LOH? - Well not necessarily. For example: A dataset may contain lots and lots of data, but the dataset object only contains references to other objects. The data in the columns are each stored in individual strings and as long as they're not larger than 85 KB you're clear.

Memory segments

The managed heap will reserve memory in segments. The size of the segments depend on the configuration. If <gcServer enabled="true"/> you will reserve memory in 64 MB segments, otherwise you'll be doing it in 32 MB segments. LOH are reserved in 16 MB segments. Only Generation 2 and the LOH will span several segments.

What happens during GC?

Let's say we're performing a full GC, including the LOH. This is what happens:

The objects on the LOH are Marked. Each item in the LOH is checked for references. If none are found it's ready to be collected.
The LOH is Swept. All marked objects are released from memory.
The LOH is not compacted.
Generation 2 is Marked.
Generation 2 is Swept.
Generation 2 is compacted. (Imagine removing a few books from your bookcase, and then pushing the remaining books together freeing up continuous space at the end.)
Generation 1 is Marked.
Generation 1 is Swept.
Everything that survived the sweep is compacted.
The pointer for where Generation 2 ends is updated. Everything that survived the sweep is now Generation 2.
Generation 0 is Marked.
Generation 0 is Swept.
Everything that survived the sweep is compacted.
The pointer for where Generation 1 ends is updated. Everything that survived the sweep is now Generation 1.

A few quick tips

This topic could off course be a lot bigger, but here are some quick suggestions.

Try to stay out of Generation 1

Well off course Generation 1 is better than Generation 2, but you should aim for keeping only a select few objects in Generation 2. Those should be variables that are defined at the beginning of the application lifecycle and released at the end. In an ideal world all other variables should be of the hit-and-run variety and never leave Generation 0.

Don't call GC.Collect()

This has been said before. It will be said again, and again, and again. You should almost never call GC.Collect() manually. And by almost I mean once in a lifetime, not once per application, and certainly not once per function call. I occasionally call GC.Collect() for testing purposes just to see if memory has been released. Normally you would never call it. The GC is self-balancing and by calling GC.Collect() you are disrupting that balance. Think of it as tampering with the eco-system, pouring sugar in the gasoline or whatever metaphor you prefer. :-)

Avoid large objects

If you can stay below the 85.000 byte limit, then do so. If not, consider reusing the object. When it comes to large objects it's better to use one for a long time than to use many for short periods of time.

Don't use finalizers

When your object has a finalizer the finalize method will be called when the object is no longer alive. So far so good. Unfortunately your object will be passed into the next generation, since it's not yet ready to be collected. This means that all objects with a finalizer will at least end up in Generation 1. Most likely in Generation 2.

Well, as I said: There is a lot more to cover on this. There are books to be written and songs to be sung, but you've got to draw the line somewhere. I will most certainly cover more of this in future posts.

/ Johan

Why adding more memory won't resolve OutOfMemoryExceptions

JohanS — Thu, 19 Apr 2007 12:43:59 GMT

Quite often I am met with the incorrect assumption that out of memory exceptions can be resolved by adding more memory. I can understand why you'd think that, but actually it won't matter at all. Additional RAM may increase performance, but an additional 8GB of memory won't increase the available amount of memory for the .NET application.

Here's what you need to know:

If you have a 32 bit machine then you have 2 GB of memory available.

That's the way it is. Never mind if you have 512 MB or 8 GB of RAM. A 32-bit system can address 4 GB of virtual memory. 2 GB of those are reserved for the operating system, leaving 2 GB for each user mode process. If there is no RAM available we will use the page file and save infrequently used memory to disk instead. So additional RAM will cause fewer page faults, but the amount of memory available for the process is still only 2 GB.

(Actually you can tune this in boot.ini. There is a switch that will allow you to steal 1 GB from the OS and increase the limit for the user process from 2 GB to 3 GB. It still requires some additional tweaking though. I suggest looking at http://support.microsoft.com/kb/820108/en-us for more information on this.)

Okay, so now that we know this it's time for the next vital piece of information:

Your process will most likely be out of memory when it reaches ~800 MB

- What!? How is that possible!? Didn't I just say that we have 2 GB available? I did, but there is another thing you really need to know about memory allocations:

Memory allocations need to be continuous.

If you want to allocate a 100 MB string, then you need 100 MB of continuous space. Memory allocations don't work the way the file system does. If we were talking about files we could save part of the file in one place, another part in another place, etc. but memory != file system. Open up the disk defragmenter and take a look at your hard drive. You may have 100 GB available on your 250 GB hard drive, but how much of that is continuous? The available memory for your process will eventually become just as fragmented as your hard drive. And as your memory becomes more and more fragmented it will be harder and harder to squeeze in those large allocations.

Let's say you and your party of four walk into a restaurant. The restaurant is packed with four seat tables so finding a seat shouldn't be a problem. Unfortunately there is 1-3 persons sitting at each table. So even if only half the seats are taken you are still unable to get your own table. The usher gets an OutOfSeatsException as he tries to seat you. :-)

The rule of thumb is that you'll begin seeing OutOfMemoryExceptions at around 800 MB. Off course this isn't set in stone. It depends entirely on the average size of the objects you allocate and how fragmented your memory becomes. You might make it up to 1.2 GB or you might see the exceptions already at 550 MB.

So what can be done?

Well if you're in the planning stage of a WebService that will load 4 GB images, process them somehow and then return them to the client you'll want to either:

Work with streams and make sure you never load the full image into memory
Set aside the money to invest in a 64 bit system
Rethink

I'm currently writing a bigger post on how the managed heap and garbage collector work, so expect more to come on the subject.

/ Johan

I've upgraded and now my application doesn't work anymore

JohanS — Tue, 23 Jan 2007 19:09:47 GMT

Scenario:

A quite common scenario when working in the support industry is a call along theese lines:

"My application worked just fine, but now that I've upgraded to IE7, IIS6, Vista, etc. it doesn't work any more. This has got to be a bug! This new version of the software obviously isn't any good, so when are you going to fix it?"

Is it a bug?

Well, possibly. But most likely the bug doesn't lie within IE, IIS or the operating system. Instead you should look at your code to make sure you did everything following the recommended guidelines. Chances are that you didn't do things the right way originally, and for some reasons the previous version of the software was more forgiving.

Example

A little while ago I had the following scenario on my hands:

A customer had just upgraded their webservers from Windows 2000 to Windows 2003. After the upgrade certain requests just "vanished" into thin air. The response never reached the clients. We managed to track down the problem to the following lines of code:

this.Page.Response.ClearContent();
this.Page.Response.Write(TextToWrite);
this.Page.Response.Flush();
this.Page.Response.Close();

Okay, so you probably see what is strange here. Why are they calling Response.Flush() and Response.Close()?

If we remove theese two lines and replace them with Response.End() then everything works fine:

this.Page.Response.ClearContent();
this.Page.Response.Write(TextToWrite);
this.Page.Response.End();

Okay, so this is the proper way to do it. Response.End() will call actually call Response.Flush() and then gracefully end execution of the page, while Response.Close() will simply "cut the cord". But how come it worked in IIS5 and not IIS6? Does this mean that IIS6 is a bad product? - Not at all!

One of the things that changed with IIS6 is that it now processes responses asynchronously. This means that in IIS5 all execution will be paused until the page has been sent, while in IIS6 the response will be put in a send-buffer, allowing IIS to immediately continue execution. This is one of the reasons why IIS6 is both faster and more secure than IIS5. The thread executing the page does not have to take the connection speed of the client into consideration. It can execute the page and move on to the next. In IIS5 all execution on the thread would be stopped until the client had downloaded every last bit. This made the server more vulnerable to Denial of Service (DOS) -attacks, and something as trivial as a bunch of clients with poor modem connections could impair the performance of the server.

In brief, here’s what happened with the old code:

IIS5:

Response.Flush sends data to client
Thread waits until data has been sent
Response.Close closes client connection

IIS6:

Response.Flush puts the data in a send buffer and immediately moves to the next line of code
Response.Close closes the client connection before the data has been sent

Here’s what happens with the new code:

IIS5:

Response.End is called

The data is sent to the client
IIS gracefully ends all further execution of the page

IIS6:

Response.End is called

The data is transferred to the send buffer
IIS gracefully ends all further execution of the page

Summary:

The old code was incorrect, but worked anyway due to the synchronous design of IIS5. As IIS6 switched to an asynchronous response model this stopped working. I can sympathize with anyone that feels that this is a bug/mistake, but in reality it isn't. In fact it is a very concious choice made to further improve performance and reliability.

/ Johan

SessionState Performance

JohanS — Mon, 20 Nov 2006 11:49:00 GMT

Problem:

Unable to make the session state request to the session state server.
Please ensure that the ASP.NET State service is started and that the client and server ports are the same.
If the server is on a remote machine, please ensure that it accepts remote requests by checking the value of HKLM\SYSTEM\CurrentControlSet\Services\aspnet_state\Parameters\AllowRemoteConnection.

Cause:

Excessive use of SessionState

Resolution:

Minor tweaks can be made, but be aware. You are pushing SessionState to it's absolute limits.

Scenario:

Judging from the number of calls we get it seems quite common for people to store all sorts of things in SessionState and to rely quite heavily on it. I would personally say that the rule of thumb is to avoid SessionState as much as possible.

The different ways of storing SessionState

You're probably already familiar with the three different ways of storing SessionState, but I'll recap them quickly, just to refresh your memory:

InProc
In process. Works fine if you have only one web server. Great performance, but if the application pool recycles, then your SessionData is lost forever.
State Server
The simple out-of process solution. By using the ASP.NET State Server you can share data between web servers. The ASP.NET State Server can be run either on one of the web servers or on a separate server.
SQL Server
Using SQL server isn't that different from using the State Server. SQL Server may be a tiny bit slower performance-wise, but the main benefit is that you can set up a fail-over cluster.

Common myths

There are some common misunderstandings regarding SessionState and it's uses. The ones I come across most frequently would probably be the following:

Myth:

"SessionState is really efficient. If you store all datasets, etc. in a SessionVariable it's like having the data in RAM, rather than on disk"

Fact:

When storing session data out of process it is serialized, stored and deserialized. This is a CPU- and memory consuming process and if you store large objects in SessionState you will use a lot of resources to handle the data. For small chunks of data I'd recommend cookies, hidden controls (ViewState) or even query strings. For larger objects... Well, do you really need them for every single request?

Myth:

"The Server will only retrieve SessionState when I actually read or write to it."

Fact:

Sorry, but that's incorrect as well. The data will be deserialized and serialized with each request unless you have clearly specified otherwise.

Myth:

"It doesn't matter what type of data I store in SessionState."

Fact:

It does matter, and it matters a lot. For example; a 40KB string and a 40KB multi-dimensional array are not equivalent. The data must be serialized and deserialized, and serializing a string is a lot less CPU-consuming than serializing a complex array.

What is serialization anyway?

Serialization means transforming an object to a stream of bytes. This stream of bytes can then be sent over the web, saved to disk, etc. and be recreated on demand. When serializing the worker process takes your complex object and transform it to a simple byte stream that is easily dealt with. This byte stream in itself is pretty useless until you deserialize it. You can not access its properties until you've "unpacked it". Think of it as a .cab archive or instant coffee. :)

Increasing performance

If you are aware from the start that SessionState isn't the most effective way to store data, then you should be fine. If you're stuck with the ungrateful task of cleaning up someone else's mess, then you have a little more work to do.

The planning phase

Like I said, you should always try to limit your use of SessionState. It's quite common to store data "for later use", but unless you absolutely need it right now, then why store it in Session State? And if you absolutely need it right now, then why not use ViewState, cookies or some other means of temporary storage like a database? Here are some samples:

If you have a huge dataset that was really taxing to get, then save it to a temporary table or take it as a hint to rethink your database design.
Form data should be kept in ViewState or possibly in cookies.
Try really, really hard to keep Objects out of SessionState. Especially STA COM objects. If you really have to save an STA COM object, then remember to set the AspCompat attribute of the @ Page directive.
Disable Session State on all pages that don't require it, and set it to read-only on pages that don't require write-access. This is done with <@ Page EnableSessionState="..."> (Framework 2.0)
Read the article Improving .NET Application Performance and Scalability. Especially Chapter 6 which deals with ASP.NET.

Existing applications

There are a few things worth tryng if you're running an application and suddenly find yourself with a StateServer on it's knees begging for mercy. Please note that most of this is to be considered respiratory only. A few more users and you'll probably be facing the same scenario again. So my personal suggestion would probably be to quickly get the server up and running within acceptable parameters and then immediately begin working on a redesign. Anyway. here are my quick tips for a quick fix:

Disable Session State on all pages that don't use it, or set it to read-only by using the EnableSessionState attribute in the @ Page directive (Framework 2.0)
If you've hit the ceiling and are running the in-process state server, then there might be some extra performance to be gained by setting up an out-of process State Server on a separate machine.
If you're using the out of process StateServer, then you might want to increase the timeout according to the following article:
State server logs Event ID 1072 or Event ID 1076
You might also want to increase the available number of ports on the StateServer by following the guidelines in this article: (I know it doesn't mention StateServer, but the solution is the same.)
PRB: "WSAEADDRESSINUSE" Error Message When You Try to Connect Through an Anonymous Port After You Increase the IMAP Connection Limit
There is also a possibility that this fix will help you out:
FIX: You may receive an error message when you use the ASP.NET State Server service to store ASP.NET session state in the .NET Framework 1.1

More than one StateServer:

If all else fails you can actually have more than one StateServer, provided that you have a webfarm. If you set up your load balancer to use sticky sessions you could actually run each web server with its own StateServer. This would mean that each web server would have its own settings for SessionState. This gives you many options for load balancing. If you, for example, have 8 web servers you could divide the servers into pairs and let each pair share an out of process StateServer. This would give you a solution with 8 web server and 4 state servers. You could even have a separate fail-over cluster for each web server. But as much as this would please your hardware vendor you're probably compensating for poor planning by adding horsepower.

Well I guess that's all for now.

/ Johan

Are you getting OutOfMemoryExceptions when uploading large files?

JohanS — Wed, 15 Nov 2006 12:13:00 GMT

Problem:

Using the WebClient.Upload method for posting large files will eventually leave you stranded with OutOfMemoryExceptions.

Cause:

WebClient.Upload reads the entire file to memory by default.

Resolution:

Build your own uploader.

Scenario:

One of my customers was using WebClient.Upload in a Winforms application to transfer files to a webserver. The idea in itself was fine, but when they transferred a couple of large files they'd get OutOfMemoryExceptions. When uploading a 500 MB file the application would need approximately 520 MB of memory and if you uploaded a few large files after each other you quickly hit the roof. Running GC.Collect(); after each transfer didn't help. Judging from the number of hits on the internet for this scenario they weren't the only ones with this problem.

This is the code they were using:

WebClient oWeb = new WebClient();
oWeb.UploadFile("http://localhost/test.aspx", "c:\\bigfile.cab");

Okay, so why was this happening?
Well, first of all I wouldn't recommend running GC.Collect(); in any application. A lot has been written on this allready, but if you're interested in why I suggest you look at Rico Mariani's post on the subject. Anyway, for testing purposes we ran the following instead:

GC.Collect(3);

GC.WaitForPendingFinalizers();

GC.Collect(3);

And this cleared the memory. So why isn't this a valid solution? Well, like I said, I wouldn't recommend using GC.Collect(); in any application, and why read the entire file to memory when you can stream it? I looked up the UploadFile-method and it seems like it does read the entire file to a byte array before posting. This is great for smaller files, but in this particular scenario it wasn't too good. So what I did was to write my own uploader:

public static string MyUploader(string strFileToUpload, string strUrl)

{

string strFileFormName = "file";

Uri oUri = new Uri(strUrl);

string strBoundary = "----------" + DateTime.Now.Ticks.ToString("x");

// The trailing boundary string

byte[] boundaryBytes = Encoding.ASCII.GetBytes("\r\n--" + strBoundary + "\r\n");

// The post message header

StringBuilder sb = new StringBuilder();

sb.Append("--");

sb.Append(strBoundary);

sb.Append("\r\n");

sb.Append("Content-Disposition: form-data; name=\"");

sb.Append(strFileFormName);

sb.Append("\"; filename=\"");

sb.Append(Path.GetFileName(strFileToUpload));

sb.Append("\"");

sb.Append("\r\n");

sb.Append("Content-Type: ");

sb.Append("application/octet-stream");

sb.Append("\r\n");

string strPostHeader = sb.ToString();

byte[] postHeaderBytes = Encoding.UTF8.GetBytes(strPostHeader);

// The WebRequest

HttpWebRequest oWebrequest = (HttpWebRequest)WebRequest.Create(oUri);

oWebrequest.ContentType = "multipart/form-data; boundary=" + strBoundary;

oWebrequest.Method = "POST";

// This is important, otherwise the whole file will be read to memory anyway...

oWebrequest.AllowWriteStreamBuffering = false;

// Get a FileStream and set the final properties of the WebRequest

FileStream oFileStream = new FileStream(strFileToUpload, FileMode.Open, FileAccess.Read);

long length = postHeaderBytes.Length + oFileStream.Length + boundaryBytes.Length;

oWebrequest.ContentLength = length;

Stream oRequestStream = oWebrequest.GetRequestStream();

// Write the post header

oRequestStream.Write(postHeaderBytes, 0, postHeaderBytes.Length);

// Stream the file contents in small pieces (4096 bytes, max).

byte[] buffer = new Byte[checked((uint)Math.Min(4096, (int)oFileStream.Length))];

int bytesRead = 0;

while ((bytesRead = oFileStream.Read(buffer, 0, buffer.Length)) != 0)

oRequestStream.Write(buffer, 0, bytesRead);

oFileStream.Close();

// Add the trailing boundary

oRequestStream.Write(boundaryBytes, 0, boundaryBytes.Length);

WebResponse oWResponse = oWebrequest.GetResponse();

Stream s = oWResponse.GetResponseStream();

StreamReader sr = new StreamReader(s);

String sReturnString = sr.ReadToEnd();

// Clean up

oFileStream.Close();

oRequestStream.Close();

s.Close();

sr.Close();

return sReturnString;

}

One of the things worth noting is that you need to set oWebrequest.AllowWriteStreamBuffering = false; Otherwise you will read the entire file to memory anyway. This is because the default behavior of the WebRequest is to buffer the entire request in case it needs to re-send it due to authentication, connectivity problems, etc. Again, this is a default behavior that normally is a performance boost, but in this case is a performance killer.

So what was the end result?

During my first test runs the application needed as much memory as the file I was trying to upload, and then some. So in order to upload a 500 MB .cab-file the application needed at least 520 MB. The application using the custom uploader never went above 23 MB.

End of transmission

/ Johan