I didn't post to this blog during many months, mainly because I moved from my Escalation Engineer position to an Application Development Consultant position (aka Technical Account Manager Dev).
I missed some free time for that kind of leisure, and as many other interesting blogs now spread over the web, I felt unuseful to post things that may not really help someone.
That's why I come back today, to post a debugger extension I wrote some months ago to help me to automate my debugging sessions.
I always felt a lack in windbg to easily filter other debugger extensions outputs and automate recursing commands with previous results.
Others like Doug & Tess are able to use both !foreach, .shell - -ci or even logparser to filter outputs, but I'm too bad/lazy for that and preferred having my own debugger syntax :)
After having shown this extension to some debugging class I held in France, I decided to share it publicly.
The binary version I'm posting here is the initial version of this extension, that I wanted to improve before posting, but as I never had time I'm posting it here as-is..
It contains in this version only 1 debugger command called !sosexec, with this awful syntax:
!sosexec /c “command1|filter1”->”command2 [tokenID1]|filter2”->”command3 [tokenID2]”->....
The parameter [tokenID1] is the column index for the token you’re looking for, starting from 0.
For example in the bellow output, 0 is the ‘MT’ token, 1 is the ‘Field’ token, and thus 6 is the tokenID for ‘Value’. Note those indexes are changing depending on sos version so take care to check them first.
0:021> !do 0acca994
Name: System.Web.HttpContext
MethodTable: 663a2750
EEClass: 663a26e0
Size: 168(0xa8) bytes
(C:\WINDOWS\assembly\GAC_32\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll)
Fields:
MT Field Offset Type VT Attr Value Name
663ad210 4000fc7 4 ...IHttpAsyncHandler 0 instance 08a308d0 _asyncAppHandler
663a206c 4000fc8 8 ...b.HttpApplication 0 instance 08a308d0 _appInstance
663a3a9c 4000fc9 c ....Web.IHttpHandler 0 instance 0accb490 _handler
663a2ba0 4000fca 10 ...m.Web.HttpRequest 0 instance 0accaa3c _request
A real world example for ASP.NET 2.0 application could be the one bellow to list the current ASPX pages easily (note1 663a2750 was my methodtable for HttpContext) (note2 that’s just a sample, Tess debugging script makes it lots better)
0:021> !sosexec /c "!dumpheap -mt 663a2750 -short"->"!do [0]|_request"->"!do [6]|_path"->"!do [6]|String:"
Executing !do 06c35cb0
Executing !do 06c35d58
Executing !do 06c34538
String: E:\Applications.Net\CRM Cog Integration Web\coghierarchy.aspx
Executing !do 0acca994
Executing !do 0acc91e4
String: E:\Applications.Net\CRM Cog Integration Web\cogcompanyinformation.aspx
Executing !do 0ca32e98
Executing !do 0ca32f40
Executing !do 0ca31310
String: E:\Applications.Net\CRM Cog Integration Web\cogcompanyinformation.aspx
Executing !do 0caa95fc
Executing !do 0caa96a4
Executing !do 0caa7e4c
String: E:\Applications.Net\CRM Cog Integration Web\cogcompanyinformation.aspx
This command will:
1. Execute !dumpheap -mt 663a2750 -short (I got MT from an earlier !dumpheap –stat)
3. Execute !do with the 1th token (ID 0) for each output lines of step 1
4. Filter this output for lines containing string “_request”
5. Execute !do again with the 7th token (ID 6) of each filtered line of step 4 (the ‘Value’ token)
6. Filter output with lines containing the string ‘_path’
7. Execute !do with again the 7th token (ID 6) of each filtered line of step 6
8. Filter output from step 7. to only show the string content, that starts with “String:”
Another example bellow will list SQL commands executed by current threads:
!sosexec /c "~*e !dso|System.Data.SqlClient.SqlCommand"->"!do [1]|cmdText"->"!do [5]"
Executing !do 0x2645ab0c
Executing !do 0x143f9908
String: sql_PrestataireLoadPatente
Executing !do 0x111bbc18
Executing !do 0x1c442bdc
String: sql_FactureLoad
Executing !do 0x14efddc0
Executing !do 0x143f5f44
String: sql_Vue_facture_impayee
The nice thing is that it works for both user (native & managed) and kernel debugging (live or postmortem).
Another example will dump all ASP templates compiled in an IIS process using the iisinfo.dll debugger extension brought by DebugDiag.
.logopen c:\temp\aspcode.txt
!sosexec /c "!iisinfo.templates -v|C:\"->"!templatecode [1]"
.logclose
You can also simply use it to filter your debugger output, for example listing all DLLs named ora* (don’t ask me why :-p)
0:029> !sosexec /c "lmnt|ora"
03f70000 041c7000 orageneric9 orageneric9.dll Mon May 06 02:23:24 2002 (3CD5CCFC)
60500000 60590000 oracommon9 oracommon9.dll Mon Apr 29 01:12:14 2002 (3CCC81CE)
60800000 60806000 oravsn9 oravsn9.dll Mon Apr 29 01:13:48 2002 (3CCC822C)
60810000 60816000 orawtc9 orawtc9.dll Mon Apr 29 01:13:50 2002 (3CCC822E)
61400000 6142c000 oranl9 oranl9.dll Sat Apr 27 04:30:17 2002 (3CCA0D39)
61480000 61534000 oran9 oran9.dll Sat Apr 27 04:31:08 2002 (3CCA0D6C)
I highly recommend to write and test those commands step by step, as a simple mistake can make this version of the extension loop infinitely executing wrong commands and thus looking for unknown symbols… J
When I'll have a better version, I'll post it with source to codeplex to also let other people improve it, as it exposes lightweight framework of C++ objects to allow to be reused in other debugger extensions, such as my only !sosexec command is only made of:
HRESULT CALLBACK
sosexec(PDEBUG_CLIENT4 Client, PCSTR args)
{
INIT_API();
// Read arguments command line
CArgs* oArgs = new CArgs(args);
vector<string> sCommands = oArgs->getCommands();
// Split args into commands/filters/tokens
vector<string> sFilters = oArgs->getFilters();
vector<int> iTokens = oArgs->getTokens();
// Recursively execute commands
CExecutor *oExec1 = new CExecutor(Client, sCommands,sFilters,iTokens);
oExec1->RecurseExec();
free(oExec1);
EXIT_API();
return S_OK;
}
On the feature list there is currently some more switches like /unique, /silent, ability to use multiple filters, and skip headers or footer lines.
As I’ll try to improve it and post source code for good, feel free to share ideas, feedbacks or sample commands that you find useful to share.
Enjoy!
BHO means “Browser Helper Object”, that’s an IE plugin that interacts with browser & user events.
Basically this is a COM component that implements IObjectWithSite and is registered under
“HKLM\Software\Microsoft\Windows\CurrentVersion\Explorer\Browser Helper Objects”
A typical BHO could be a pop-up blocker, customizing at client-side an HTML document.
Some of them are also doing some UI stuffs, like adding menu features or acting as a toolbar
Note however those toolbars & co. don’t always require to be a registered as BHOs.
For a step by step BHO go to Building Browser Helper Objects with Visual Studio 2005.
For a sample step by step toolbar, go to that tutorial .
That being said, we might get a great user experience from IE by reacting to its application...
That could make inside IE a kind of “rich intranet application”, interactions between client side scripts, and extensions.
We can imagine different kind of interaction with that schema (non exhaustive list…):
· Calling a method exposed by your browser extension from JavaScript
· Calling JavaScript method/event from your browser extension
· Modify the HTML / JS code from your browser extension
This post will for now only speak about calling a BHO method from your scripting code, maybe in future posts I’ll describe other possible things with some samples.
To get an IE extensions that changes some user UI (menu items, mouse gesture, etc...), we can easily implement in our BHO IDocHostUIHandler and attach it to current document using ICustomDoc::SetUIHandler()
That trick also allows to “extend” the HTML external object by implementing some custom window.external.foo() methods.
When making that window.external call, the webbrowser control will call into IDocHostUIHandler::GetExternal() to get the UI handler IDispatch interface to invoke
This means you just need to implement it to send back your own interface to make it callable.
The problem I wanted to discuss in that post is that there is no existing design for chaining UIHandlers… Indeed UIHandlers exist to allow extending user interface within a custom application that hosts a WebBrowser control, but not for Internet Explorer itself that already exposes a UI...
IOW from a BHO you might not be able to add your own menu items, but only to replace the all menu.
That’s basically the same for HTML external object, IE also relies on that external object to implement some UI features through IShellUIHelper & IShellUIHelper2 interfaces (like Favorites & find dialogs, or runonce and search providers with IE7).
That kind of problem you might have is described by Q330441 (PRB: ICustomDoc::SetUIHandler Causes Changes in Save As Dialog)
If you replace the UI Handler within IE by your BHO instance (in order to answer window.external() calls), you will break thereby several UI related features, that is a highly non desirable behavior.
In order to workaround that design issue, some people have implemented within their BHO a dispatch gateway towards builtin interfaces IShellUIHelper & IShellUIHelper2.
That was quite easy to do, just chaining within our BHO IDispatch methods GetIDsOfNames() & Invoke() to standard IDispatch shell interfaces and was working pretty well.
Unfortunately for some reasons I would need to discuss longer in a next post, that might not be working anymore as expected on IE7, getting instead an E_FAIL result.
The best way I could recommend to call your BHO from a script is to come back to a supported way...One of the easiest method is to create a simple ActiveX that is called from your script, and fires BHO methods. To do that, we basically only have to get the BHO instance from the ActiveX and to call its methods.
A good method is to rely on IE session to store a variant we’ll know about on each side (using IWebBrowser2::PutProperty() / IWebBrowser2::GetProperty()).
In the code bellow I register my BHO instance in session variable called “MyBHO_IDisp”. That allows my BHO to “register” for later use from ActiveX.
STDMETHODIMP CObjectBHO::SetSite(IUnknown* pUnkSite)
{
…
// Registers the IE session to allow ActiveX to call into it
hr = ReferenceMe(false);
if (FAILED(hr))
return S_FALSE;
…
}
HRESULT CObjectBHO::ReferenceMe(bool bRemove)
{
BSTR bstrThisKey = SysAllocString(L"MyBHO_IDisp");
VARIANT vThis;
HRESULT hr = S_FALSE;
if (!bRemove)
{
if (!m_spWebBrowserApp)
goto Cleanup;
// Save this to a variant that will be referenced in IE Session
VariantInit(&vThis);
vThis.vt = VT_DISPATCH;
vThis.pdispVal = static_cast<IDispatch*>(this);
// Add our this pointer to IE session by adding a named property
if (FAILED( m_spWebBrowserApp->PutProperty(bstrThisKey, vThis) ))
goto Cleanup;
}
else
{
VariantInit(&vThis);
vThis.vt = VT_NULL;
// Time to release, remove our reference
if (FAILED( m_spWebBrowserApp->PutProperty(bstrThisKey, vThis) ))
goto Cleanup;
}
hr = S_OK;
Cleanup:
VariantClear(&vThis);
SysFreeString(bstrThisKey);
return hr;
}
My ActiveX object now only has to implement IObjectWithSiteImpl::SetSite() to save the client site instance, and to grab the BHO instance from its session properties (GrabBHOInstance())
// CMyClass
class ATL_NO_VTABLE CMyClass :
public CComObjectRootEx<CComSingleThreadModel>,
public CComControl<CMyClass>,
public CComCoClass<CMyClass, &CLSID_MyClass>,
public IObjectWithSiteImpl <CMyClass>,
public IDispatchImpl<IMyClass, &IID_IMyClass, &LIBID_MySampleATLLib, /*wMajor =*/ 1, /*wMinor =*/ 0>
{
private:
IDispatch *m_pBHODisp;
CComPtr<IUnknown> _spUnkSite;
HRESULT(GrabBHOInstance)(void);
public:
CMyClass()
{
}
STDMETHODIMP CMyClass::SetSite(IUnknown* pUnkSite)
{
HRESULT hr = S_FALSE;
// Save our client site instance
if (pUnkSite)
{
_spUnkSite = pUnkSite;
hr = S_OK;
}
// Try to grab BHO
GrabBHOInstance();
return hr;
}
Here is the implementation for GrabBHOInstance, just getting property from IWebBrowser2::GetProperty()
HRESULT CMyClass::GrabBHOInstance()
{
IServiceProvider* pISP = NULL;
IWebBrowser2* pBrowser = NULL;
BSTR bstrBHOKey = SysAllocString(L"MyBHO_IDisp");
VARIANT vBHO;
HRESULT hr = S_FALSE;
VariantInit(&vBHO);
// Get the IWebBrowser2 interface
if (!_spUnkSite)
goto Cleanup;
if (FAILED(_spUnkSite->QueryInterface(IID_IServiceProvider, (void **)&pISP) ))
goto Cleanup;
if (FAILED( pISP->QueryService(IID_IWebBrowserApp,
IID_IWebBrowser2, (void **)&pBrowser)))
goto Cleanup;
// Get the BHO instance pointer
if (FAILED( pBrowser->GetProperty(bstrBHOKey, &vBHO) ))
goto Cleanup;
// Ensure it's valid and reference count it
if (vBHO.vt == VT_DISPATCH && vBHO.pdispVal != NULL)
{
m_pBHODisp = vBHO.pdispVal;
m_pBHODisp->AddRef();
}
hr = S_OK;
Cleanup:
VariantClear(&vBHO);
SysFreeString(bstrBHOKey);
if (pBrowser != NULL)
{
pBrowser->Release();
pBrowser = NULL;
}
if (pISP != NULL)
{
pISP->Release();
pISP = NULL;
}
return hr;
}
STDMETHODIMP CMyClass::get_IsBHOInstalled(VARIANT_BOOL* pVal)
{
// Check if our BHO instance could be get from the SetSite() call
if (m_pBHODisp)
*pVal = true;
else
*pVal = false;
return S_OK;
}
A simple ActiveX method now can call into our BHO
STDMETHODIMP CMyClass::SayHelloFromBHO(BSTR* pVal)
{
CComVariant varResult;
HRESULT hr = S_FALSE;
DISPPARAMS params = { NULL, NULL, 0, 0 };
// Ensure we have the BHO instance propertly got
if (!m_pBHODisp)
if (FAILED(GrabBHOInstance()))
return hr;
// Forward to BHO instance if valid
if (m_pBHODisp)
{
hr = m_pBHODisp->Invoke(DISP_MYBHOMETHOD, IID_NULL, LOCALE_USER_DEFAULT,
DISPATCH_METHOD, ¶ms, &varResult, NULL, NULL);
// Return the BSTR we got.
*pVal = SysAllocStringLen(varResult.bstrVal, SysStringLen(varResult.bstrVal));
}
return hr;
}
Bellow a sample of the Jscript needed to call your ActiveX, that will fire a method from your browser extension.
var myATL = new ActiveXObject("MySampleATL.MyClass");
if (myATL.IsBHOInstalled)
alert (myATL. SayHelloFromBHO());
else
alert ("BHO isn't installed now !");
window.external.AddFavorite("http://blogs.msdn.com/nicd", "Nico’s Blog")
That’s here a quick & modest sample of what can be done in order to integrate your BHO within your client web application.
Maybe in later posts I’ll discuss other possible things with some samples if some people are interested.
I’m now going to holidays for some days, so apologizes in advance if my 3rd post takes some times J
Nico
For my first technical post, I wanted to discuss about IIS6 & ASP.NET request processing, and their related queuing & threading mechanisms
I see quite every day some people who misunderstand the processing flow in their production applications, whereas it helps a lot to diagnose potential contentions.
For sure I’ve been here only using public symbols store (http://msdl.microsoft.com/download/symbols) and public information for the explanation bellow.
Quick reminds around IIS6 architecture
As all of you know, IIS6 has been completely re-designed compared to previous versions, and one of the biggest change is the usage of a kernel HTTP dedicated driver, http.sys
This allowed to move all the HTTP processing and part of the caching mechanisms to the kernel, that improves a lot performances, compared to the old IIS5 usermode & winsock architecture.
Take a look at Web Application Infrastructure - Performance and scability (that following graph is coming from) for some detailed explanations about those architectural changes.
It shows that http.sys now handles a dedicated request queue for each application pool, and those worker processes pick them up.
Speaking about user mode side, we have still inetinfo.exe that is in charge of metabase management & other services like FTP and SMTP.
IIS Application Pools are handled by one (or more for webgardens) worker process named w3wp.exe, that will be the hosting your server code.
Following graph comes from Version Differences in IIS Web Application Features article and details this new architecture:
.gif "IIS6 WPI mode")
IIS Request processing & queues
That being reminded, so what are the queuing mechanisms in place, and how can I locate where my site may have a performance bottleneck?
Let’s follow an ASPX request processing, from kernel mode until the normal page lifecycle.
1. Inside http.sys (aka kernel side)
As explained above, each client HTTP request reaches first our kernel driver http.sys.
I won’t go in details about what it does exactly, but here are the first steps, quoting IIS Request Processing:
“A request arrives at HTTP.sys:
1) HTTP.sys determines if the request is valid. If the request is not valid, it sends a code for an invalid request back to the client.
2) If the request is valid, HTTP.sys checks to see if the request is for static content (HTML) because static content can be served immediately.
3) If the request is for dynamic content, HTTP.sys checks to see if the response is located in its kernel-mode cache.
4) If the response is in the cache, HTTP.sys returns the response immediately.
5) If the response is not cached, HTTP.sys determines the correct request queue, and places the request in that queue.
6) If the queue has no worker processes assigned to it, HTTP.sys signals the WWW service to start one.”
So at that time the client request is sent into a kernel mode queue (per application pool), to be picked up by the corresponding IIS worker process.
That is the first queue that will be in use while processing our request, and you can configure its max size in IIS MMC - Application Pool - "Limit the kernel request queue to”.
If one of the kernel queue gets exhausted (exceed the configured limit) then http.sys will return an HTTP 503 QueueFull error within httperr log files.
That means the user mode application pool doesn’t pick up requests quick enough, so more requests are sitting in kernel than the max allowed.
For sure you won’t be able from a user dump to check current utilization of a kernel specific application pool’s queue
2. Inside W3 Thread Pool
With IIS6, userland processing begins here, when w3wp.exe picks up requests to handle from this kernel queue.
Remember on IIS4 & IIS5, all requests were first going to inetinfo.exe, that was redirecting them either via COM to IWAM interface or to ASPNET async named pipe.
To pick up those requests, w3wp.exe just handles a dedicated thread pool bound to an I/O Completion Port.
That thread pool is implemented within w3tp.dll, and so is always used, regardless of the type of requests it’s getting.
Indeed W3 thread pool (w3tp.dll) only grabs completion packets, so it hasn’t to know now if that async I/O completion is a request or not.
The completion routine that will get now executed within w3dt.dll, that is the interface between IIS Worker Process & Http.sys.
Now those completion packets will be turned into native requests, that are what the useful iisinfo!clientconns command will show you (windbg extension brought with IIS Debug Diagnostic Tools).
0:005> !iisinfo.clientconns
UL_NATIVE_REQUEST listhead at: 0x5a36a064
====================================
Item# 6, UL_NATIVE_REQUEST = 0x00d5bbc0
Client IP/Port: 127.0.0.1:3334
Server IP/Port: 127.0.0.1:80
Host Header: localhost:80
Request state: NREQ_STATE_PROCESS
Requested URL: GET /wicked/DummyPage.aspx HTTP/1.1
Starttime (0x0b537877) Uptime (0x0bc44db8)
Request active 7394625 ms (0 days: 02:03:14.625)
Once we have native requests, IIS has to decide what handler will execute it, depending on the corresponding scriptmap configuration (ISAPI extension, CGI handler, etc…).
In the case of an ASPX page, for sure our handler is the ISAPI extension aspnet_isapi.dll.
0:009> kL999
ChildEBP RetAddr
00d4fe04 5a322991 aspnet_isapi!HttpExtensionProc
00d4fe24 5a3968ff w3isapi!ProcessIsapiRequest+0x214
00d4fe58 5a3967e0 w3core!W3_ISAPI_HANDLER::IsapiDoWork+0x3fd
00d4fe78 5a396764 w3core!W3_ISAPI_HANDLER::DoWork+0xb0
00d4fe98 5a3966f4 w3core!W3_HANDLER::MainDoWork+0x16e
00d4fea8 5a3966ae w3core!W3_CONTEXT::ExecuteCurrentHandler+0x53
00d4fec4 5a396648 w3core!W3_CONTEXT::ExecuteHandler+0x51
00d4feec 5a392264 w3core!W3_STATE_HANDLE_REQUEST::DoWork+0x9a
00d4ff10 5a3965ea w3core!W3_MAIN_CONTEXT::DoWork+0xa6
00d4ff2c 5a36169f w3core!W3_MAIN_CONTEXT::OnNewRequest+0x55
00d4ff38 5a361650 w3dt!UL_NATIVE_REQUEST::DoStateProcess+0x48
00d4ff48 5a3616ca w3dt!UL_NATIVE_REQUEST::DoWork+0x7f
00d4ff5c 5a3024de w3dt!OverlappedCompletionRoutine+0x1a
00d4ff8c 5a3026bc W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x73
00d4ffa0 5a301db9 W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x24
00d4ffb8 77e6608b W3TP!THREAD_MANAGER::ThreadManagerThread+0x39
00d4ffec 00000000 kernel32!BaseThreadStart+0x34
Maybe you noticed we have now a new module called webengine.dll in addition to aspnet_isapi.dll.
With ASP.NET 2.0, aspnet_isapi.dll became a very light ISAPI extension, that just calls into webengine.dll, in order to create AppDomain and to drive the request processing.
Webengine.dll gives all its power on IIS7, where it is loaded as a global module, and provides the managed code extensibility inside IIS7 Integrated Request Pipeline, but that’s another story.
3. Inside ASP.NET ISAPI handler
So we are now at the point that webengine.dll gets the .NET request to execute from the ECB it gets (the well known ISAPI “Extension Control Block”).
0:009> kp
ChildEBP RetAddr
00d4fe04 5a322991 webengine!AspNetHttpExtensionProc
00d4fe24 5a3968ff w3isapi!ProcessIsapiRequest+0x214
00d4fe58 5a3967e0 w3core!W3_ISAPI_HANDLER::IsapiDoWork+0x3fd
00d4fe78 5a396764 w3core!W3_ISAPI_HANDLER::DoWork+0xb0
00d4fe98 5a3966f4 w3core!W3_HANDLER::MainDoWork+0x16e
00d4fea8 5a3966ae w3core!W3_CONTEXT::ExecuteCurrentHandler+0x53
00d4fec4 5a396648 w3core!W3_CONTEXT::ExecuteHandler+0x51
00d4feec 5a392264 w3core!W3_STATE_HANDLE_REQUEST::DoWork+0x9a
00d4ff10 5a3965ea w3core!W3_MAIN_CONTEXT::DoWork+0xa6
00d4ff2c 5a36169f w3core!W3_MAIN_CONTEXT::OnNewRequest+0x55
00d4ff38 5a361650 w3dt!UL_NATIVE_REQUEST::DoStateProcess+0x48
00d4ff48 5a3616ca w3dt!UL_NATIVE_REQUEST::DoWork+0x7f
00d4ff5c 5a3024de w3dt!OverlappedCompletionRoutine+0x1a
00d4ff8c 5a3026bc W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x73
00d4ffa0 5a301db9 W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x24
00d4ffb8 77e6608b W3TP!THREAD_MANAGER::ThreadManagerThread+0x39
00d4ffec 00000000 kernel32!BaseThreadStart+0x34
The first thing webengine.dll will do (same for aspnet_isapi on 1.1) will be to check if its application queue is maxed out, otherwise it will send back a HTTP 503 - Server Too Busy error.
So here is our 2nd queue inside IIS processing, called “Application Queue”, that will contain .NET requests, called HttpCompletion instances.
We can configure the size of this queue using appRequestQueueLimit attribute of httpRuntime element of your web/machine.config (the default is 5000 on 2.0 but was only 100 on 1.0/1.1).
To check from a dump file the current status of that application queue, you can compare g_RequestQueueLimit with HttpCompletion::s_RequestsCurrent (for .NET 1.1 they are hosted in aspnet_isapi.dll)
For example bellow only 36 requests are processed and application queue is configured to handle maximum of 5000 requests (default setting).
0:029> ?poi(webengine!HttpCompletion::s_RequestsCurrent)
Evaluate expression: 36 = 00000024
0:028> ?poi(webengine!g_RequestQueueLimit)
Evaluate expression: 5000 = 00001388
s_RequestsCurrent is only used to check the state of that application queue, after that if you want to know how much ongoing completion requests you have, check s_ActiveManagedRequestCount
0:029> ?poi(webengine!HttpCompletion::s_ActiveManagedRequestCount)
Evaluate expression: 36 = 00000024
Those HttpCompletion instances will be then processed by a separated thread, inside .NET thread pool…
4. Inside .NET Thread pool
Once the request reaches at that point, it’s now being handled by .NET thread pool. Those thread pool settings were one of the first thing to check if you experience a performance issue under stress.
We use to recommend to read Contention, poor performance, and deadlocks when you make Web service requests from ASP.NET applications, that explains you how to configure settings of that thread pool like:
· maxWorkerThreads
· minWorkerThreads
· maxIoThreads
· minFreeThreads
· minLocalRequestFreeThreads
· maxconnection
· executionTimeout
Seeing badly configured .NET thread pool was quite happening on a daily basis, .NET 2.0 includes now a great AutoConfig setting.
I highly recommend you to keep that autoconfig setting enabled if you don’t exactly know what you want to change and why.
From a userdump, you can easily check .NET threadpool usage using sos.dll, as lots of other blogs & sites already describe perfectly.
0:001> !threadpool
CPU utilization 12%
Worker Thread: Total: 2 Running: 1 Idle:1 MaxLimit: 200 MinLimit: 2
Work Request in Queue: 1
--------------------------------------
Number of Timers: 7
--------------------------------------
Completion Port Thread:Total: 2 Free: 2 MaxFree: 4 CurrentLimit: 2 MaxLimit: 200 MinLimit: 2
0:001> !threads
ThreadCount: 8
UnstartedThread: 0
BackgroundThread: 7
PendingThread: 0
DeadThread: 1
Hosted Runtime: no
PreEmptive GC Alloc Lock
ID OSID ThreadOBJ State GC Context Domain Count APT Exception
17 1 fb8 00115b88 1808220 Enabled 0217dac4:0217e630 000ddd40 0 Ukn (Threadpool Worker)
21 2 974 00118708 b220 Enabled 00000000:00000000 000ddd40 0 MTA (Finalizer)
22 3 dc8 00132ee8 80a220 Enabled 00000000:00000000 000ddd40 0 MTA (Threadpool Completion Port)
23 4 e24 00135b30 1220 Enabled 00000000:00000000 000ddd40 0 Ukn
24 5 f90 0014bf28 880b220 Enabled 00000000:00000000 000ddd40 0 MTA (Threadpool Completion Port)
15 6 f6c 0014d508 880a220 Enabled 00000000:00000000 000ddd40 0 MTA (Threadpool Completion Port)
25 7 c28 0e5e9640 180b220 Enabled 0217bd80:0217c630 000ddd40 0 MTA (Threadpool Worker)
XXXX a 0 000cfdb0 1801820 Enabled 00000000:00000000 000ddd40 0 Ukn (Threadpool Worker)
I also recommend you to read “Production Debugging For .Net Framework Applications – Debugging Contention Problems” for detailed information regarding possible thread pool issues.
To understand correctly how .NET threading works, you might to know that .NET ThreadPoolMgr handles 5 kinds of threads:
· Worker Thread
· I/O Thread (aka Completion Port Thread)
· Wait thread
· Gate thread
· Timer thread
a. .NET Worker Threads
The number of .NET worker threads can be configured by maxWorkerThreads / minFreeThreads / minWorkerThreads (auto-tuned if AutoConfig setting is enabled on 2.0)
Basically a simple ASPX processing will be done in a synchronous manner by one of those worker thread (begins with mscorwks!ThreadpoolMgr::ExecuteWorkRequest()).
For example, you can see bellow a .NET worker thread that executes Page_Load() event inside my hello.aspx page:
0:028> kL999
ChildEBP RetAddr
0619d204 698a1928 App_Web_zggy9p1v!ASP.hello_aspx.Page_Load(System.Object, System.EventArgs)+0x83
0feff2a4 6627f05f System_Web_RegularExpressions_ni!System.Web.Util.CalliHelper.EventArgFunctionCaller(....)+0x10
0feff2a4 6612bda4 System_Web_ni!System.Web.Util.CalliEventHandlerDelegateProxy.Callback(...)+0x23
0feff2a4 6612bdf0 System_Web_ni!System.Web.UI.Control.OnLoad(System.EventArgs)+0x64
0feff2a4 6613d416 System_Web_ni!System.Web.UI.Control.LoadRecursive()+0x30
0feff2a4 6613cd41 System_Web_ni!System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)+0x426
0feff2d4 6613cca7 System_Web_ni!System.Web.UI.Page.ProcessRequest(Boolean, Boolean)+0x4d
0feff310 6613cbc7 System_Web_ni!System.Web.UI.Page.ProcessRequest()+0x57
0feff32c 6613cb5a System_Web_ni!System.Web.UI.Page.ProcessRequestWithNoAssert(System.Web.HttpContext)+0x13
0feff32c 0e2d6395 System_Web_ni!System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)+0x32
0feff364 65fe90df App_Web_zggy9p1v!ASP. hello _aspx.ProcessRequest(System.Web.HttpContext)+0x5
0feff364 65fba191 System_Web_ni!System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()+0x9b
0feff3a0 65fba4bb System_Web_ni!System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)+0x41
0feff3ec 65fb924d System_Web_ni!System.Web.HttpApplication.ResumeSteps(System.Exception)+0x163
060d8e08 65fbe244 System_Web_ni!System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest(...)+0x91
0feff43c 65fbde92 System_Web_ni!System.Web.HttpRuntime.ProcessRequestInternal(System.Web.HttpWorkerRequest)+0x194
0feff470 65fbc567 System_Web_ni!System.Web.HttpRuntime.ProcessRequestNoDemand(System.Web.HttpWorkerRequest)+0x62
0feff470 79f1ef33 System_Web_ni!System.Web.Hosting.ISAPIRuntime.ProcessRequest(IntPtr, Int32)+0x57
0feff528 79f1ed6a mscorwks!COMToCLRWorkerBody+0x208
0feff584 79f3b836 mscorwks!COMToCLRWorkerDebuggerWrapper+0x37
0feff768 01cba295 mscorwks!COMToCLRWorker+0x4ba
0feff790 6a2bfe7f CLRStub[StubLinkStub]@1cba295
0feffaf0 6a2c0044 webengine!HttpCompletion::ProcessRequestInManagedCode+0x1a3
0feffafc 6a2d9475 webengine!HttpCompletion::ProcessCompletion+0x3e
0feffb10 7a110f08 webengine!CorThreadPoolWorkitemCallback+0x18
0feffb28 7a112328 mscorwks!ThreadpoolMgr::ExecuteWorkRequest+0x40
0feffb94 79ecb00b mscorwks!ThreadpoolMgr::WorkerThreadStart+0x1f2
0fefffb8 77e6608b mscorwks!Thread::intermediateThreadProc+0x49
0fefffec 00000000 kernel32!BaseThreadStart+0x34
At that point we’ve entered the normal page lifecycle done by our HttpHandler (page_init(), page_load(), page_prerender(), page_unload(), etc…)
So for a synchronous ASPX request, you can consider that your code is now being notified / executed until it finishes, and response goes to client.
b. .NET I/O Threads (aka Completion Port Threads)
Like worker threads, the number of I/O threads is controlled by maxIoThreads / minIoThreads (auto-tuned if AutoConfig setting is enabled on 2.0).
That thread pool is bound to an I/O completion port mechanism, to handle asynchronous I/O completions that arrive either from kernel, or being reposted from usermode.
Before IIS6 WPI mode (when ASP.NET was still handled by a dedicated aspnet_wp.exe process), that I/O thread pool was being used very frequently.
At that old time, all requests were first going to inetinfo.exe process, and then forwarded to aspnet_wp.exe using an async named pipe.
Then, each request needed first to be picked up by that I/O thread, and then they got processed very frequently inside that I/O threads, or redirected to worker thread pool.
Nowadays with WPI process model, worker process directly picks up requests from kernel side http.sys by W3TP, and then directly handled by .NET worker thread pool.
However if you are using new .NET 2.0 async pages (Async="true") or ThreadPool.QueueUserWorkItem(), then the asynchronous part of the processing will be done inside such I/O thread.
As the following graph shows (taken the article mentioned above), an async page can be processed by 3 different threads: Worker Thread 1 -> I/O thread -> Worker Thread 2.
That means for asynchronous pages, really don’t store anything bounded to the executing thread
Bellow is an example of a .NET I/O thread handling an Async ASP.NET 2.0 page, and currently executing EndAsyncOperation()
0:029> !tp
CPU utilization 50%
Worker Thread: Total: 2 Idle: 2 MaxLimit: 200 MinLimit: 2
Work Request in Queue: 0
--------------------------------------
--------------------------------------
Completion Port Thread: Total: 3 Free: 2 MaxFree: 4 CurrentLimit: 3 MaxLimit: 200 MinLimit: 2
0:029> !threads
ThreadCount: 9
UnstartedThread: 0
BackgroundThread: 9
PendingThread: 0
DeadThread: 0
Hosted Runtime: no
PreEmptive GC Alloc Lock
ID OSID ThreadOBJ State GC Context Domain Count APT Exception
21 2 974 00118708 b220 Enabled 02185514:02187488 000ddd40 0 MTA (Finalizer)
22 3 dc8 00132ee8 80a220 Enabled 00000000:00000000 000ddd40 0 MTA (Threadpool Completion Port)
23 4 e24 00135b30 1220 Enabled 00000000:00000000 000ddd40 0 Ukn
25 7 c28 0e5e9640 180b220 Enabled 06281c70:06283828 000ddd40 0 MTA (Threadpool Worker)
29 8 e14 000cfdb0 200b220 Enabled 0218b4c4:0218d488 0e565db8 1 MTA
1 b fbc 0ec4d008 880b220 Disabled 021d3448:021d3724 0e565db8 0 MTA (Threadpool Completion Port)
4 c 264 0ec57840 880b220 Enabled 0627a544:0627b828 000ddd40 0 MTA (Threadpool Completion Port)
13 d e4c 0ec5a6c0 180b220 Enabled 021ce9c8:021cf724 000ddd40 0 MTA (Threadpool Worker)
28 a c10 0e5ea0d8 880b220 Enabled 00000000:00000000 000ddd40 0 MTA (Threadpool Completion Port)
0:001> kL999
ChildEBP RetAddr
00a2ea38 66140a23 App_Web_bdcvaxlh!AsyncPage.EndAsyncOperation(System.IAsyncResult)+0x1d7
00a2ea64 7a5653d7 System_Web_ni!System.Web.UI.Page+PageAsyncInfo.OnAsyncHandlerCompletion(...)+0x5f
00a2ea98 7a565c72 System_ni!System.Net.LazyAsyncResult.Complete(IntPtr)+0x7f
00a2eab0 793685af System_ni!System.Net.ContextAwareResult.CompleteCallback(System.Object)+0x1a
00a2eab0 79e88f63 mscorlib_ni!System.Threading.ExecutionContext.runTryCode(System.Object)+0x43
00a2eac0 79e88ee4 mscorwks!CallDescrWorker+0x33
00a2eb40 79e88e31 mscorwks!CallDescrWorkerWithHandler+0xa3
00a2ec78 79e88d19 mscorwks!MethodDesc::CallDescr+0x19c
00a2ec90 79e88cf6 mscorwks!MethodDesc::CallTargetWorker+0x20
00a2eca4 79ef80d3 mscorwks!MethodDescCallSite::Call+0x18
00a2ee70 79ef7fde mscorwks!ExecuteCodeWithGuaranteedCleanupHelper+0xb2
00a2ef20 793684fb mscorwks!ReflectionInvocation::ExecuteCodeWithGuaranteedCleanup+0xf9
021ce844 793683ee mscorlib_ni!System.Threading.ExecutionContext.RunInternal(...)+0xa7
00000000 7a565c3b mscorlib_ni!System.Threading.ExecutionContext.Run(...)+0x92
00a2ef6c 7a5652eb System_ni!System.Net.ContextAwareResult.Complete(IntPtr)+0xa7
00a2efb8 7a57e5bd System_ni!System.Net.LazyAsyncResult.ProtectedInvokeCallback(...)+0x8b
00a2efb8 7a57e4a3 System_ni!System.Net.HttpWebRequest.ProcessResponse()+0xe1
00a2effc 7a57e2d9 System_ni!System.Net.HttpWebRequest.SetResponse(...)+0x19b
00000000 7a5aadb0 System_ni!System.Net.HttpWebRequest.SetAndOrProcessResponse(...)+0x1b1
00a2f054 7a5af3db System_ni!System.Net.ConnectionReturnResult.SetResponses(...)+0x70
00a2f094 7a5aed5f System_ni!System.Net.Connection.ReadComplete(...)+0x303
00a2f0d4 7a5aec88 System_ni!System.Net.Connection.ReadCallback(...)+0xc3
00a2f114 7a5653d7 System_ni!System.Net.Connection.ReadCallbackWrapper(...)+0x44
00a2f114 7a565bc3 System_ni!System.Net.LazyAsyncResult.Complete(...)+0x7f
00a2f12c 7a5652eb System_ni!System.Net.ContextAwareResult.Complete(...)+0x2f
00a2f174 7a60e232 System_ni!System.Net.LazyAsyncResult.ProtectedInvokeCallback(...)+0x8b
00a2f174 793d6ac4 System_ni!System.Net.Sockets.BaseOverlappedAsyncResult.CompletionPortCallback(...)+0x116
00a2f194 79e88f63 mscorlib_ni!System.Threading._IOCompletionCallback.PerformIOCompletionCallback(...)+0x68
00a2f1a8 79e88ee4 mscorwks!CallDescrWorker+0x33
00a2f228 79f20212 mscorwks!CallDescrWorkerWithHandler+0xa3
00a2f248 79f201bc mscorwks!DispatchCallBody+0x1e
00a2f2ac 79f2024b mscorwks!DispatchCallDebuggerWrapper+0x3d
00a2f2e0 7a07bebf mscorwks!DispatchCallNoEH+0x51
00a2f388 79ecb4a4 mscorwks!BindIoCompletionCallBack_Worker+0x123
00a2f398 79ecb442 mscorwks!Thread::UserResumeThread+0xfb
00a2f42c 79ecb364 mscorwks!Thread::DoADCallBack+0x355
00a2f468 79f3a0b3 mscorwks!Thread::DoADCallBack+0x541
00a2f474 7a0e17ed mscorwks!Thread::UserResumeThread+0xa6
00a2f524 7a0e27ef mscorwks!Thread::DoADCallBack+0xd9
00a2f53c 79ecb442 mscorwks!Thread::UserResumeThread+0xe1
00a2f5d0 79ecb364 mscorwks!Thread::DoADCallBack+0x355
00a2f60c 7a0e1b7e mscorwks!Thread::DoADCallBack+0x541
00a2f634 7a0e1bab mscorwks!Thread::DoADCallBack+0x575
00a2f648 7a07c031 mscorwks!ManagedThreadBase::ThreadPool+0x13
00a2f69c 7a07c063 mscorwks!BindIoCompletionCallbackStubEx+0x8c
00a2f6b0 79f2f3b0 mscorwks!BindIoCompletionCallbackStub+0x13
00a2f714 79ecb00b mscorwks!ThreadpoolMgr::CompletionPortThreadStart+0x406
00a2ffb8 77e6608b mscorwks!Thread::intermediateThreadProc+0x49
00a2ffec 00000000 kernel32!BaseThreadStart+0x34
Then depending on your application, that might be very important to have a correctly sized I/O thread pool, but once again please try first AutoConfig feature..
The 3 other kinds of threads are not directly related to a standard ASP.NET page processing, so I’ll discuss them more quickly.
c. .NET Wait Threads
Wait threads are used to wait on a synchronization object, typically on a System.Threading.WaitHandle.
You can call ThreadPool.RegisterWaitForSingleObject() to register one of your delegate for a WaitHandle.
Bellow is what an idle wait thread looks like:
32 Id: 230.ac0 Suspend: 0 Teb: 7ff4c000 Unfrozen
ChildEBP RetAddr Args to Child
1b88ff6c 7c573a4e 00000001 1b88ff84 00000000 NTDLL!ZwDelayExecution+0xb
1b88ff8c 7923558c ffffffff 00000001 198ba828 KERNEL32!SleepEx+0x32
1b88ffb4 7c57438b 00000000 198ba828 197f7498 mscorwks!ThreadpoolMgr::WaitThreadStart+0x45
1b88ffec 00000000 7923556a 1973b3d0 00000000 KERNEL32!BaseThreadStart+0x52
Maybe I’ll write later a post showing a practical sample about efficiently using wait threads in a multi threaded application.
d. Gate Thread
The .NET gate thread is the one that is responsible for the worker threads & I/O threads creation or destruction, based on several criteria.
Bellow is what an idle gate thread looks like:
16 Id: 930.aac Suspend: 1 Teb: 7ffaa000 Unfrozen
ChildEBP RetAddr
01c4fe14 7c821364 ntdll!KiFastSystemCallRet
01c4fe18 77e41ea7 ntdll!NtDelayExecution+0xc
01c4fe80 79f2c9f3 kernel32!SleepEx+0x68
01c4feb4 79f2c9c3 mscorwks!EESleepEx+0xa3
01c4fee8 79f758ec mscorwks!__DangerousSwitchToThread+0x70
01c4fef4 79f2c895 mscorwks!__SwitchToThread+0xb
01c4ffb8 77e6608b mscorwks!ThreadpoolMgr::GateThreadStart+0xa1
01c4ffec 00000000 kernel32!BaseThreadStart+0x34
Check Marc Clifton very good article .NET's ThreadPool Class - Behind The Scenes, that shows a very comprehensive graph of how ShouldGrowWorkerThread()
e. Timer Thread
Nothing new here, timer thread is there to handle .NET System.Threading.Timer delegates…
Just one thing to say around timers, if you are still using .NET 1.1 SP1 then please ensure to have installed following fix corresponding to Q900822
FIX: When a .NET Framework based application uses the System.Threading.Timer class, the timer event may not be signaled in the .NET Framework 1.1 SP1
Ok so I think that’s now time for my 1st technical post to end after those few lines.
Again understanding how your requests will get processed is a key point to pro-actively avoid contentions under stress.
I only described there the quite hidden part that IIS & ASP.NET are doing to process a request, lots of very good articles describe how you should write your multi threaded application.
HTH
Nico
