For my first technical post, I wanted to discuss about IIS6 & ASP.NET request processing, and their related queuing & threading mechanisms
I see quite every day some people who misunderstand the processing flow in their production applications, whereas it helps a lot to diagnose potential contentions.
For sure I’ve been here only using public symbols store (http://msdl.microsoft.com/download/symbols) and public information for the explanation bellow.


Quick reminds around IIS6 architecture

As all of you know, IIS6 has been completely re-designed compared to previous versions, and one of the biggest change is the usage of a kernel HTTP dedicated driver, http.sys
This allowed to move all the HTTP processing and part of the caching mechanisms to the kernel, that improves a lot performances, compared to the old IIS5 usermode & winsock architecture.

Take a look at
Web Application Infrastructure - Performance and scability (that following graph is coming from) for some detailed explanations about those architectural changes.
It shows that http.sys now handles a dedicated request queue for each application pool, and those worker processes pick them up.

IIS6 Kernel Queues 

Speaking about user mode side, we have still inetinfo.exe that is in charge of metabase management & other services like FTP and SMTP.
IIS Application Pools are handled by one (or more for webgardens) worker process named w3wp.exe, that will be the hosting your server code.
Following graph comes from
Version Differences in IIS Web Application Features article and details this new architecture:


IIS6 WPI mode

 

IIS Request processing & queues

That being reminded, so what are the queuing mechanisms in place, and how can I locate where my site may have a performance bottleneck?

Let’s follow an ASPX request processing, from kernel mode until the normal page lifecycle.

1.     Inside http.sys (aka kernel side)

As explained above, each client HTTP request reaches first our kernel driver http.sys.
I won’t go in details about what it does exactly, but here are the first steps, quoting IIS Request Processing:

“A request arrives at HTTP.sys:

1)     HTTP.sys determines if the request is valid. If the request is not valid, it sends a code for an invalid request back to the client.

2)     If the request is valid, HTTP.sys checks to see if the request is for static content (HTML) because static content can be served immediately.

3)     If the request is for dynamic content, HTTP.sys checks to see if the response is located in its kernel-mode cache.

4)     If the response is in the cache, HTTP.sys returns the response immediately.

5)     If the response is not cached, HTTP.sys determines the correct request queue, and places the request in that queue.

6)     If the queue has no worker processes assigned to it, HTTP.sys signals the WWW service to start one.”

 

So at that time the client request is sent into a kernel mode queue (per application pool), to be picked up by the corresponding IIS worker process.
That is the first queue that will be in use while processing our request, and you can configure its max size in IIS MMC - Application Pool - "Limit the kernel request queue to”.
If one of the kernel queue gets exhausted (exceed the configured limit) then http.sys will return an
HTTP 503 QueueFull error within httperr log files.

That means the user mode application pool doesn’t pick up requests quick enough, so more requests are sitting in kernel than the max allowed.
For sure you won’t be able from a user dump to check current utilization of a kernel specific application pool’s queue

2.     Inside W3 Thread Pool 

With IIS6, userland processing begins here, when w3wp.exe picks up requests to handle from this kernel queue.
Remember on IIS4 & IIS5, all requests were first going to inetinfo.exe, that was redirecting them either via COM to IWAM interface or to ASPNET async named pipe.

To pick up those requests, w3wp.exe just handles a dedicated thread pool bound to an I/O Completion Port.
That thread pool is implemented within w3tp.dll, and so is always used, regardless of the type of requests it’s getting.
Indeed W3 thread pool (w3tp.dll) only grabs completion packets, so it hasn’t to know now if that async I/O completion is a request or not.

The completion routine that will get now executed within w3dt.dll, that is the interface between IIS Worker Process & Http.sys.
Now those completion packets will be turned into native requests, that are what the useful iisinfo!clientconns command will show you (windbg extension brought with IIS Debug Diagnostic Tools).

 

0:005> !iisinfo.clientconns

UL_NATIVE_REQUEST listhead at: 0x5a36a064

 

====================================

Item# 6, UL_NATIVE_REQUEST = 0x00d5bbc0

 

Client IP/Port:        127.0.0.1:3334

Server IP/Port:        127.0.0.1:80

Host Header:           localhost:80

Request state:         NREQ_STATE_PROCESS

Requested URL:         GET /wicked/DummyPage.aspx HTTP/1.1

 

Starttime (0x0b537877) Uptime (0x0bc44db8)

Request active 7394625 ms (0 days: 02:03:14.625)

 

Once we have native requests, IIS has to decide what handler will execute it, depending on the corresponding scriptmap configuration (ISAPI extension, CGI handler, etc…).
In the case of an ASPX page, for sure our handler is the ISAPI extension aspnet_isapi.dll.

 

0:009> kL999

ChildEBP RetAddr 

00d4fe04 5a322991 aspnet_isapi!HttpExtensionProc

00d4fe24 5a3968ff w3isapi!ProcessIsapiRequest+0x214

00d4fe58 5a3967e0 w3core!W3_ISAPI_HANDLER::IsapiDoWork+0x3fd

00d4fe78 5a396764 w3core!W3_ISAPI_HANDLER::DoWork+0xb0

00d4fe98 5a3966f4 w3core!W3_HANDLER::MainDoWork+0x16e

00d4fea8 5a3966ae w3core!W3_CONTEXT::ExecuteCurrentHandler+0x53

00d4fec4 5a396648 w3core!W3_CONTEXT::ExecuteHandler+0x51

00d4feec 5a392264 w3core!W3_STATE_HANDLE_REQUEST::DoWork+0x9a

00d4ff10 5a3965ea w3core!W3_MAIN_CONTEXT::DoWork+0xa6

00d4ff2c 5a36169f w3core!W3_MAIN_CONTEXT::OnNewRequest+0x55

00d4ff38 5a361650 w3dt!UL_NATIVE_REQUEST::DoStateProcess+0x48

00d4ff48 5a3616ca w3dt!UL_NATIVE_REQUEST::DoWork+0x7f

00d4ff5c 5a3024de w3dt!OverlappedCompletionRoutine+0x1a

00d4ff8c 5a3026bc W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x73

00d4ffa0 5a301db9 W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x24

00d4ffb8 77e6608b W3TP!THREAD_MANAGER::ThreadManagerThread+0x39

00d4ffec 00000000 kernel32!BaseThreadStart+0x34

 

Maybe you noticed we have now a new module called webengine.dll in addition to aspnet_isapi.dll.
With ASP.NET 2.0, aspnet_isapi.dll became a very light ISAPI extension, that just calls into webengine.dll, in order to create AppDomain and to drive the request processing.
Webengine.dll gives all its power on IIS7, where it is loaded as a global module, and provides the managed code extensibility inside IIS7 Integrated Request Pipeline,  but that’s another story.

3.     Inside ASP.NET ISAPI handler

So we are now at the point that webengine.dll gets the .NET request to execute from the ECB it gets (the well known ISAPI “Extension Control Block”).

 

0:009> kp

ChildEBP RetAddr 

00d4fe04 5a322991 webengine!AspNetHttpExtensionProc

00d4fe24 5a3968ff w3isapi!ProcessIsapiRequest+0x214

00d4fe58 5a3967e0 w3core!W3_ISAPI_HANDLER::IsapiDoWork+0x3fd

00d4fe78 5a396764 w3core!W3_ISAPI_HANDLER::DoWork+0xb0

00d4fe98 5a3966f4 w3core!W3_HANDLER::MainDoWork+0x16e

00d4fea8 5a3966ae w3core!W3_CONTEXT::ExecuteCurrentHandler+0x53

00d4fec4 5a396648 w3core!W3_CONTEXT::ExecuteHandler+0x51

00d4feec 5a392264 w3core!W3_STATE_HANDLE_REQUEST::DoWork+0x9a

00d4ff10 5a3965ea w3core!W3_MAIN_CONTEXT::DoWork+0xa6

00d4ff2c 5a36169f w3core!W3_MAIN_CONTEXT::OnNewRequest+0x55

00d4ff38 5a361650 w3dt!UL_NATIVE_REQUEST::DoStateProcess+0x48

00d4ff48 5a3616ca w3dt!UL_NATIVE_REQUEST::DoWork+0x7f

00d4ff5c 5a3024de w3dt!OverlappedCompletionRoutine+0x1a

00d4ff8c 5a3026bc W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x73

00d4ffa0 5a301db9 W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x24

00d4ffb8 77e6608b W3TP!THREAD_MANAGER::ThreadManagerThread+0x39

00d4ffec 00000000 kernel32!BaseThreadStart+0x34

 

The first thing webengine.dll will do (same for aspnet_isapi on 1.1) will be to check if its application queue is maxed out, otherwise it will send back a HTTP 503 - Server Too Busy error.
So here is our 2nd queue inside IIS processing, called “Application Queue”, that will contain .NET requests, called HttpCompletion instances.
We can configure the size of this queue using appRequestQueueLimit attribute of httpRuntime element of your web/machine.config (the default is 5000 on 2.0 but was only 100 on 1.0/1.1).

 

To check from a dump file the current status of that application queue, you can compare g_RequestQueueLimit with HttpCompletion::s_RequestsCurrent (for .NET 1.1 they are hosted in aspnet_isapi.dll)
For example bellow only 36 requests are processed and application queue is configured to handle maximum of 5000 requests (default setting).

 

0:029> ?poi(webengine!HttpCompletion::s_RequestsCurrent)

Evaluate expression: 36 = 00000024

 

0:028> ?poi(webengine!g_RequestQueueLimit)

Evaluate expression: 5000 = 00001388

 

s_RequestsCurrent is only used to check the state of that application queue, after that if you want to know how much ongoing completion requests you have, check s_ActiveManagedRequestCount

 

0:029> ?poi(webengine!HttpCompletion::s_ActiveManagedRequestCount)

Evaluate expression: 36 = 00000024

 

Those HttpCompletion instances will be then processed by a separated thread, inside .NET thread pool…

4.     Inside .NET Thread pool

Once the request reaches at that point, it’s now being handled by .NET thread pool. Those thread pool settings were one of the first thing to check if you experience a performance issue under stress.
We use to recommend to read Contention, poor performance, and deadlocks when you make Web service requests from ASP.NET applications, that explains you how to configure settings of that thread pool like:

·         maxWorkerThreads

·         minWorkerThreads

·         maxIoThreads

·         minFreeThreads

·         minLocalRequestFreeThreads

·         maxconnection

·         executionTimeout

 

Seeing badly configured .NET thread pool was quite happening on a daily basis, .NET 2.0 includes now a great AutoConfig setting.
I highly recommend you to keep that autoconfig setting enabled if you don’t exactly know what you want to change and why.

From a userdump, you can easily check .NET threadpool usage using sos.dll, as lots of other blogs & sites already describe perfectly.

 

0:001> !threadpool

CPU utilization 12%

Worker Thread: Total: 2 Running: 1 Idle:1 MaxLimit: 200 MinLimit: 2

Work Request in Queue: 1

--------------------------------------

Number of Timers: 7

--------------------------------------

Completion Port Thread:Total: 2 Free: 2 MaxFree: 4 CurrentLimit: 2 MaxLimit: 200 MinLimit: 2

 

0:001> !threads

ThreadCount: 8

UnstartedThread: 0

BackgroundThread: 7

PendingThread: 0

DeadThread: 1

Hosted Runtime: no

                                      PreEmptive   GC Alloc           Lock

       ID OSID ThreadOBJ    State     GC       Context       Domain   Count APT Exception

  17    1  fb8 00115b88   1808220 Enabled  0217dac4:0217e630 000ddd40     0 Ukn (Threadpool Worker)

  21    2  974 00118708   b220 Enabled  00000000:00000000 000ddd40         0 MTA (Finalizer)

  22    3  dc8 00132ee8  80a220 Enabled  00000000:00000000 000ddd40     0 MTA (Threadpool Completion Port)

  23    4  e24 00135b30    1220 Enabled  00000000:00000000 000ddd40        0 Ukn

  24    5  f90 0014bf28   880b220 Enabled  00000000:00000000 000ddd40     0 MTA (Threadpool Completion Port)

  15    6  f6c 0014d508  880a220 Enabled  00000000:00000000 000ddd40     0 MTA (Threadpool Completion Port)

  25    7  c28 0e5e9640 180b220 Enabled  0217bd80:0217c630 000ddd40     0 MTA (Threadpool Worker)

XXXX    a   0 000cfdb0  1801820 Enabled  00000000:00000000 000ddd40     0 Ukn (Threadpool Worker)

 

I also recommend you to read “Production Debugging For .Net Framework Applications – Debugging Contention Problems” for detailed information regarding possible thread pool issues.

To understand correctly how .NET threading works, you might to know that .NET ThreadPoolMgr handles 5 kinds of threads:

·         Worker Thread

·         I/O Thread (aka Completion Port Thread)

·         Wait thread

·         Gate thread

·         Timer thread

 

 

a.     .NET Worker Threads

The number of .NET worker threads can be configured by maxWorkerThreads / minFreeThreads / minWorkerThreads (auto-tuned if AutoConfig setting is enabled on 2.0)
Basically a simple ASPX processing will be done in a synchronous manner by one of those worker thread (begins with mscorwks!ThreadpoolMgr::ExecuteWorkRequest()).

For example, you can see bellow a .NET worker thread that executes Page_Load() event inside my hello.aspx page:

 

0:028> kL999

ChildEBP RetAddr 

0619d204 698a1928 App_Web_zggy9p1v!ASP.hello_aspx.Page_Load(System.Object, System.EventArgs)+0x83

0feff2a4 6627f05f System_Web_RegularExpressions_ni!System.Web.Util.CalliHelper.EventArgFunctionCaller(....)+0x10

0feff2a4 6612bda4 System_Web_ni!System.Web.Util.CalliEventHandlerDelegateProxy.Callback(...)+0x23

0feff2a4 6612bdf0 System_Web_ni!System.Web.UI.Control.OnLoad(System.EventArgs)+0x64

0feff2a4 6613d416 System_Web_ni!System.Web.UI.Control.LoadRecursive()+0x30

0feff2a4 6613cd41 System_Web_ni!System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)+0x426

0feff2d4 6613cca7 System_Web_ni!System.Web.UI.Page.ProcessRequest(Boolean, Boolean)+0x4d

0feff310 6613cbc7 System_Web_ni!System.Web.UI.Page.ProcessRequest()+0x57

0feff32c 6613cb5a System_Web_ni!System.Web.UI.Page.ProcessRequestWithNoAssert(System.Web.HttpContext)+0x13

0feff32c 0e2d6395 System_Web_ni!System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)+0x32

0feff364 65fe90df App_Web_zggy9p1v!ASP. hello _aspx.ProcessRequest(System.Web.HttpContext)+0x5

0feff364 65fba191 System_Web_ni!System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()+0x9b

0feff3a0 65fba4bb System_Web_ni!System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)+0x41

0feff3ec 65fb924d System_Web_ni!System.Web.HttpApplication.ResumeSteps(System.Exception)+0x163

060d8e08 65fbe244 System_Web_ni!System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest(...)+0x91

0feff43c 65fbde92 System_Web_ni!System.Web.HttpRuntime.ProcessRequestInternal(System.Web.HttpWorkerRequest)+0x194

0feff470 65fbc567 System_Web_ni!System.Web.HttpRuntime.ProcessRequestNoDemand(System.Web.HttpWorkerRequest)+0x62

0feff470 79f1ef33 System_Web_ni!System.Web.Hosting.ISAPIRuntime.ProcessRequest(IntPtr, Int32)+0x57

0feff528 79f1ed6a mscorwks!COMToCLRWorkerBody+0x208

0feff584 79f3b836 mscorwks!COMToCLRWorkerDebuggerWrapper+0x37

0feff768 01cba295 mscorwks!COMToCLRWorker+0x4ba

0feff790 6a2bfe7f CLRStub[StubLinkStub]@1cba295

0feffaf0 6a2c0044 webengine!HttpCompletion::ProcessRequestInManagedCode+0x1a3

0feffafc 6a2d9475 webengine!HttpCompletion::ProcessCompletion+0x3e

0feffb10 7a110f08 webengine!CorThreadPoolWorkitemCallback+0x18

0feffb28 7a112328 mscorwks!ThreadpoolMgr::ExecuteWorkRequest+0x40

0feffb94 79ecb00b mscorwks!ThreadpoolMgr::WorkerThreadStart+0x1f2

0fefffb8 77e6608b mscorwks!Thread::intermediateThreadProc+0x49

0fefffec 00000000 kernel32!BaseThreadStart+0x34

 

 

At that point we’ve entered the normal page lifecycle done by our HttpHandler (page_init(), page_load(), page_prerender(), page_unload(), etc…)
So for a synchronous ASPX request, you can consider that your code is now being notified / executed until it finishes, and response goes to client.

b.    .NET I/O Threads (aka Completion Port Threads)

Like worker threads, the number of I/O threads is controlled by maxIoThreads / minIoThreads (auto-tuned if AutoConfig setting is enabled on 2.0).
That thread pool is bound to an I/O completion port mechanism, to handle asynchronous I/O completions that arrive either from kernel, or being reposted from usermode.

Before IIS6 WPI mode (when ASP.NET was still handled by a dedicated aspnet_wp.exe process), that I/O thread pool was being used very frequently.
At that old time, all requests were first going to inetinfo.exe process, and then forwarded to aspnet_wp.exe using an async named pipe.
Then, each request needed first to be picked up by that I/O thread, and then they got processed very frequently inside that I/O threads, or redirected to worker thread pool.

Nowadays with WPI process model, worker process directly picks up requests from kernel side http.sys by W3TP, and then directly handled by .NET worker thread pool.
However if you are using new .NET 2.0 async pages (Async="true") or ThreadPool.QueueUserWorkItem(), then the asynchronous part of the processing will be done inside such I/O thread.

As the following graph shows (taken the article mentioned above), an async page can be processed by 3 different threads: Worker Thread 1 -> I/O thread -> Worker Thread 2.
That means for asynchronous pages, really don’t store anything bounded to the executing thread

Synchronous vs. Asynchronous Page Processing 

Bellow is an example of a .NET I/O thread handling an Async ASP.NET 2.0 page, and currently executing EndAsyncOperation()

 

0:029> !tp

CPU utilization 50%

Worker Thread: Total: 2 Idle: 2 MaxLimit: 200 MinLimit: 2

Work Request in Queue: 0

--------------------------------------

--------------------------------------

Completion Port Thread: Total: 3 Free: 2 MaxFree: 4 CurrentLimit: 3 MaxLimit: 200 MinLimit: 2

 

0:029> !threads

ThreadCount: 9

UnstartedThread: 0

BackgroundThread: 9

PendingThread: 0

DeadThread: 0

Hosted Runtime: no

                                      PreEmptive   GC Alloc           Lock

       ID OSID ThreadOBJ    State     GC       Context       Domain   Count APT Exception

  21    2  974 00118708      b220 Enabled  02185514:02187488 000ddd40     0 MTA (Finalizer)

  22    3  dc8 00132ee8    80a220 Enabled  00000000:00000000 000ddd40     0 MTA (Threadpool Completion Port)

  23    4  e24 00135b30      1220 Enabled  00000000:00000000 000ddd40     0 Ukn

  25    7  c28 0e5e9640   180b220 Enabled  06281c70:06283828 000ddd40     0 MTA (Threadpool Worker)

  29    8  e14 000cfdb0   200b220 Enabled  0218b4c4:0218d488 0e565db8     1 MTA

   1    b  fbc 0ec4d008   880b220 Disabled 021d3448:021d3724 0e565db8     0 MTA (Threadpool Completion Port)

   4    c  264 0ec57840   880b220 Enabled  0627a544:0627b828 000ddd40     0 MTA (Threadpool Completion Port)

  13    d  e4c 0ec5a6c0   180b220 Enabled  021ce9c8:021cf724 000ddd40     0 MTA (Threadpool Worker)

  28    a  c10 0e5ea0d8   880b220 Enabled  00000000:00000000 000ddd40     0 MTA (Threadpool Completion Port)

 

0:001> kL999

ChildEBP RetAddr 

00a2ea38 66140a23 App_Web_bdcvaxlh!AsyncPage.EndAsyncOperation(System.IAsyncResult)+0x1d7

00a2ea64 7a5653d7 System_Web_ni!System.Web.UI.Page+PageAsyncInfo.OnAsyncHandlerCompletion(...)+0x5f

00a2ea98 7a565c72 System_ni!System.Net.LazyAsyncResult.Complete(IntPtr)+0x7f

00a2eab0 793685af System_ni!System.Net.ContextAwareResult.CompleteCallback(System.Object)+0x1a

00a2eab0 79e88f63 mscorlib_ni!System.Threading.ExecutionContext.runTryCode(System.Object)+0x43

00a2eac0 79e88ee4 mscorwks!CallDescrWorker+0x33

00a2eb40 79e88e31 mscorwks!CallDescrWorkerWithHandler+0xa3

00a2ec78 79e88d19 mscorwks!MethodDesc::CallDescr+0x19c

00a2ec90 79e88cf6 mscorwks!MethodDesc::CallTargetWorker+0x20

00a2eca4 79ef80d3 mscorwks!MethodDescCallSite::Call+0x18

00a2ee70 79ef7fde mscorwks!ExecuteCodeWithGuaranteedCleanupHelper+0xb2

00a2ef20 793684fb mscorwks!ReflectionInvocation::ExecuteCodeWithGuaranteedCleanup+0xf9

021ce844 793683ee mscorlib_ni!System.Threading.ExecutionContext.RunInternal(...)+0xa7

00000000 7a565c3b mscorlib_ni!System.Threading.ExecutionContext.Run(...)+0x92

00a2ef6c 7a5652eb System_ni!System.Net.ContextAwareResult.Complete(IntPtr)+0xa7

00a2efb8 7a57e5bd System_ni!System.Net.LazyAsyncResult.ProtectedInvokeCallback(...)+0x8b

00a2efb8 7a57e4a3 System_ni!System.Net.HttpWebRequest.ProcessResponse()+0xe1

00a2effc 7a57e2d9 System_ni!System.Net.HttpWebRequest.SetResponse(...)+0x19b

00000000 7a5aadb0 System_ni!System.Net.HttpWebRequest.SetAndOrProcessResponse(...)+0x1b1

00a2f054 7a5af3db System_ni!System.Net.ConnectionReturnResult.SetResponses(...)+0x70

00a2f094 7a5aed5f System_ni!System.Net.Connection.ReadComplete(...)+0x303

00a2f0d4 7a5aec88 System_ni!System.Net.Connection.ReadCallback(...)+0xc3

00a2f114 7a5653d7 System_ni!System.Net.Connection.ReadCallbackWrapper(...)+0x44

00a2f114 7a565bc3 System_ni!System.Net.LazyAsyncResult.Complete(...)+0x7f

00a2f12c 7a5652eb System_ni!System.Net.ContextAwareResult.Complete(...)+0x2f

00a2f174 7a60e232 System_ni!System.Net.LazyAsyncResult.ProtectedInvokeCallback(...)+0x8b

00a2f174 793d6ac4 System_ni!System.Net.Sockets.BaseOverlappedAsyncResult.CompletionPortCallback(...)+0x116

00a2f194 79e88f63 mscorlib_ni!System.Threading._IOCompletionCallback.PerformIOCompletionCallback(...)+0x68

00a2f1a8 79e88ee4 mscorwks!CallDescrWorker+0x33

00a2f228 79f20212 mscorwks!CallDescrWorkerWithHandler+0xa3

00a2f248 79f201bc mscorwks!DispatchCallBody+0x1e

00a2f2ac 79f2024b mscorwks!DispatchCallDebuggerWrapper+0x3d

00a2f2e0 7a07bebf mscorwks!DispatchCallNoEH+0x51

00a2f388 79ecb4a4 mscorwks!BindIoCompletionCallBack_Worker+0x123

00a2f398 79ecb442 mscorwks!Thread::UserResumeThread+0xfb

00a2f42c 79ecb364 mscorwks!Thread::DoADCallBack+0x355

00a2f468 79f3a0b3 mscorwks!Thread::DoADCallBack+0x541

00a2f474 7a0e17ed mscorwks!Thread::UserResumeThread+0xa6

00a2f524 7a0e27ef mscorwks!Thread::DoADCallBack+0xd9

00a2f53c 79ecb442 mscorwks!Thread::UserResumeThread+0xe1

00a2f5d0 79ecb364 mscorwks!Thread::DoADCallBack+0x355

00a2f60c 7a0e1b7e mscorwks!Thread::DoADCallBack+0x541

00a2f634 7a0e1bab mscorwks!Thread::DoADCallBack+0x575

00a2f648 7a07c031 mscorwks!ManagedThreadBase::ThreadPool+0x13

00a2f69c 7a07c063 mscorwks!BindIoCompletionCallbackStubEx+0x8c

00a2f6b0 79f2f3b0 mscorwks!BindIoCompletionCallbackStub+0x13

00a2f714 79ecb00b mscorwks!ThreadpoolMgr::CompletionPortThreadStart+0x406

00a2ffb8 77e6608b mscorwks!Thread::intermediateThreadProc+0x49

00a2ffec 00000000 kernel32!BaseThreadStart+0x34

 

Then depending on your application, that might be very important to have a correctly sized I/O thread pool, but once again please try first AutoConfig feature..

The 3 other kinds of threads are not directly related to a standard ASP.NET page processing, so I’ll discuss them more quickly.

c.     .NET Wait Threads

Wait threads are used to wait on a synchronization object, typically on a System.Threading.WaitHandle.
You can call ThreadPool.RegisterWaitForSingleObject() to register one of your delegate for a WaitHandle.

Bellow is what an idle wait thread looks like:

 

  32  Id: 230.ac0 Suspend: 0 Teb: 7ff4c000 Unfrozen

ChildEBP RetAddr  Args to Child             

1b88ff6c 7c573a4e 00000001 1b88ff84 00000000 NTDLL!ZwDelayExecution+0xb

1b88ff8c 7923558c ffffffff 00000001 198ba828 KERNEL32!SleepEx+0x32

1b88ffb4 7c57438b 00000000 198ba828 197f7498 mscorwks!ThreadpoolMgr::WaitThreadStart+0x45

1b88ffec 00000000 7923556a 1973b3d0 00000000 KERNEL32!BaseThreadStart+0x52

 

Maybe I’ll write later a post showing a practical sample about efficiently using wait threads in a multi threaded application.

d.    Gate Thread

The .NET gate thread is the one that is responsible for the worker threads & I/O threads creation or destruction, based on several criteria.

Bellow is what an idle gate thread looks like:

 

  16  Id: 930.aac Suspend: 1 Teb: 7ffaa000 Unfrozen

ChildEBP RetAddr 

01c4fe14 7c821364 ntdll!KiFastSystemCallRet

01c4fe18 77e41ea7 ntdll!NtDelayExecution+0xc

01c4fe80 79f2c9f3 kernel32!SleepEx+0x68

01c4feb4 79f2c9c3 mscorwks!EESleepEx+0xa3

01c4fee8 79f758ec mscorwks!__DangerousSwitchToThread+0x70

01c4fef4 79f2c895 mscorwks!__SwitchToThread+0xb

01c4ffb8 77e6608b mscorwks!ThreadpoolMgr::GateThreadStart+0xa1

01c4ffec 00000000 kernel32!BaseThreadStart+0x34

 

Check Marc Clifton very good article .NET's ThreadPool Class - Behind The Scenes, that shows a very comprehensive graph of how ShouldGrowWorkerThread()

e.     Timer Thread

Nothing new here, timer thread is there to handle .NET System.Threading.Timer delegates…

Just one thing to say around timers, if you are still using .NET 1.1 SP1 then please ensure to have installed following fix corresponding to Q900822
FIX: When a .NET Framework based application uses the System.Threading.Timer class, the timer event may not be signaled in the .NET Framework 1.1 SP1

 

Ok so I think that’s now time for my 1st technical post to end after those few lines.

Again understanding how your requests will get processed is a key point to pro-actively avoid contentions under stress.
I only described there the quite hidden part that IIS & ASP.NET are doing to process a request, lots of very good articles describe how you should write your multi threaded application.

HTH
Nico