A couple of days ago I was helping a colleague of mine with some tests and I had the need to change the ThreadPool configuration; more precisely, I wanted to increase the minWorkerThreads value. Everything was working fine at first, because I was changing the value by code using the SetMinThreads method, but I also wanted to test it changing the values from my machine.config.
First thing, to be on the safe side I set autoConfig=”false” (see why here) and set my <processModel> element as follows:
<processModel autoConfig=”false” minWorkerThreads=”23” />
Why set it to 23? Simply because I did not want it to match any default.
I run my test page, attached Windbg to the process and had a look at !threadpool output: something was definitely wrong, as it was telling that minWorkerThreads was set to 2 (the default value is 1 * CPU number = 2 for my machine):
0:029> !threadpool CPU utilization 48% Worker Thread: Total: 2 Running: 0 Idle: 2 MaxLimit: 40 MinLimit: 2 Work Request in Queue: 0 -------------------------------------- Number of Timers: 8 -------------------------------------- Completion Port Thread:Total: 1 Free: 1 MaxFree: 4 CurrentLimit: 0 MaxLimit: 40 MinLimit: 2
Now I see from here and with some reasoning, it is already possible to understand why I was not seeing MinLimit=46 as I was expecting… Who has the eagle eye?
I was in a hurry (as usual) and I tried a few other things, one of them was to set autoConfig back to true, here is what I got:
0:030> !threadpool CPU utilization 7% Worker Thread: Total: 2 Running: 0 Idle: 2 MaxLimit: 200 MinLimit: 46 Work Request in Queue: 0 -------------------------------------- Number of Timers: 8 -------------------------------------- Completion Port Thread:Total: 1 Free: 1 MaxFree: 4 CurrentLimit: 0 MaxLimit: 200 MinLimit: 2
This time I’m getting it as I wanted! I’m sure someone already got it…
But before going to the solution, let’s just explain why this cannot be because of autoConfig as you may think
Here is how MSDN defines autoConfig:
Specifies whether to automatically configure the following settings to achieve optimal performance based on the machine configuration: The maxWorkerThreads attribute The maxIoThreads attribute The minFreeThreads attribute of the httpRuntime element. The minLocalRequestFreeThreads attribute of the httpRuntime element The maxConnection attribute of the <connectionManagement> Element (Network Settings) element.
Specifies whether to automatically configure the following settings to achieve optimal performance based on the machine configuration:
The values are set according to the KB article at http://support.microsoft.com/?id=821268. This attribute does not affect the .NET Framework client applications; only ASP.NET applications. The autoConfig attribute can be one of the following values. True: Indicates that ASP.NET automatically configures the attributes in the preceding list to achieve optimal performance based on the machine configuration. False: Indicates that ASP.NET should use the explicitly defined values for the attributes in the preceding list. The default in the Machine.config file is True, unless there is a previously existing configuration.
The values are set according to the KB article at http://support.microsoft.com/?id=821268. This attribute does not affect the .NET Framework client applications; only ASP.NET applications. The autoConfig attribute can be one of the following values.
The default in the Machine.config file is True, unless there is a previously existing configuration.
minWorkerThreads is not one of the values automatically configured by the runtime, so whether autoConfig is true or false does not make any difference.
Not quite…
Have another look at the !threadpool output above: with autoConfig=”true” the ThreadPool limit for maxWorkerThreads is 200 (100 * 2 CPUs), while with autoConfig=”false” maxWorkerThreads is set to 40 (default 20 * 2 CPUs).
As you can imagine, it is not possible to set minWorkerThreads higher than maxWorkerThreads… The runtime detects the inconsistency and simply set the “min” value to its default.
So I have 3 points to make:
Carlo
I got a couple of cases of this kind recently, one where the web pages were served by PerformancePoint and the other one was a completely custom web site, but both of them shared the same common root issue: browsing some specific pages, the CPU on client got constantly at 100% and the GUI was completely frozen
From the Internet Explorer dump we captured it was very clear that thread 2 was the one burning the CPU:
0:000> !runaway User Mode Time Thread Time 2:1654 0 days 0:07:55.954 0:1610 0 days 0:00:00.420 7:142c 0 days 0:00:00.070 9:1788 0 days 0:00:00.050 4:1388 0 days 0:00:00.010 17:8f0 0 days 0:00:00.000 16:11cc 0 days 0:00:00.000 15:1180 0 days 0:00:00.000 14:11ec 0 days 0:00:00.000 13:1778 0 days 0:00:00.000 12:f8 0 days 0:00:00.000 11:7dc 0 days 0:00:00.000 10:e0c 0 days 0:00:00.000 8:1784 0 days 0:00:00.000 6:1420 0 days 0:00:00.000 5:1478 0 days 0:00:00.000 3:1338 0 days 0:00:00.000 1:1634 0 days 0:00:00.000
And thread 2 has his stack:
0:002> kpL1000 ChildEBP RetAddr 01bcf0bc 43d0e4d7 mshtml!CFormatInfo::GetMatchedBy 01bcf170 43cf914f mshtml!CStyleSelector::Match+0x34d 01bcf694 43d12175 mshtml!CStyleSheetArray::Apply+0x1e1 01bcf6f4 43d17298 mshtml!CElement::ApplyDefaultFormat+0x490 01bcf708 43cf8958 mshtml!CAnchorElement::ApplyDefaultFormat+0x9f 01bcf748 43cf53e2 mshtml!CElement::ComputeFormatsVirtual+0xaa3 01bcf764 43d0cb08 mshtml!CElement::ComputeFormats+0x3a 01bcf9f4 43d3876b mshtml!CTreeNode::GetFancyFormatHelper+0x4b 01bcfb10 43e167ab mshtml!CElement::UpdateFormats+0x2da 01bcfb2c 43d3d908 mshtml!CElement::HandleMouseHoverForStyle+0x1a7 01bcfbac 43e3f53c mshtml!CDoc::PumpMessage+0xa3f 01bcfcf8 43d3c4a1 mshtml!CDoc::OnMouseMessage+0x4df 01bcfe1c 43d7e137 mshtml!CDoc::OnWindowMessage+0x748 01bcfe48 7e398734 mshtml!CServer::WndProc+0x78 01bcfe74 7e398816 user32!InternalCallWinProc+0x28 01bcfedc 7e3989cd user32!UserCallWinProcCheckWow+0x150 01bcff3c 7e398a10 user32!DispatchMessageWorker+0x306 01bcff4c 4367e6c3 user32!DispatchMessageW+0xf 01bcffb4 7c80b683 ieframe!CTabWindow::_TabWindowThreadProc+0x189 01bcffec 00000000 kernel32!BaseThreadStart+0x37
Something similar also happens if you have a .NET control inside a table using percentage width and height for the control, and using percentage width for the wrapping table cell:
ChildEBP RetAddr 018a84f8 77f17fda ntdll!KiFastSystemCallRet 018a854c 629c31f7 gdi32!NtGdiExtTextOutW+0xc 018a8588 629c322c lpk!InternalTextOut+0x1d4 018a85bc 77f31dc4 lpk!LpkExtTextOut+0x29 018a860c 436761bc gdi32!ExtTextOutW+0x26a 018a8890 43676038 mshtml!XHDC::ExtTextOutW+0x13e 018a88b8 43675ffa mshtml!ExtTextOutW+0x24 018a8e64 43675f06 mshtml!LSReExtTextOut+0x79 018a8f38 436763a5 mshtml!CLSRenderer::TextOutW+0x7fe 018a90d8 436b03e4 mshtml!CLSRenderer::BlastLineToScreen+0x4a7 018a9150 436aff59 mshtml!CLSRenderer::RenderLine+0x3f7 018a94b8 436afc4b mshtml!CDisplay::Render+0x4e0 018a94d0 436b004f mshtml!CFlowLayout::Draw+0x1c 018a9504 436ad37e mshtml!CLayout::DrawClient+0x52 018a98e0 436abdc0 mshtml!CDispLeafNode::DrawSelf+0x42c 018a9bc4 436ace26 mshtml!CDispNode::Draw+0x10d 018a9be4 436ac529 mshtml!CDispContainer::DrawChildren+0x3f 018a9f80 436abdc0 mshtml!CDispContainer::DrawSelf+0x290 018aa264 436ace26 mshtml!CDispNode::Draw+0x10d 018aa284 436ad86a mshtml!CDispContainer::DrawChildren+0x3f 018aa620 436abdc0 mshtml!CDispContainer::DrawSelf+0x2be 018aa904 436ace26 mshtml!CDispNode::Draw+0x10d 018aa924 436ad86a mshtml!CDispContainer::DrawChildren+0x3f 018aacc0 436abdc0 mshtml!CDispContainer::DrawSelf+0x2be 018aafa4 43667b64 mshtml!CDispNode::Draw+0x10d 018acf50 43667a64 mshtml!CDispRoot::DrawBand+0xc7 [...]
The stacks look quite similar for the one we had for a couple of old bugs already fixed for Internet Explorer when dynamically adding cells via DHTML (back in 2002-2003); there was also another bug in .NET 2.0 which has been fixed in KB# 928365.
For both customers the problem reproduced only with IE7, IE was running in Strict mode (but not in Quirks mode) and the page layout was made with nested Tables and DIVs whose dimensions (width and/or height) where expressed in percentage.
Since we needed the page layout to adapt gracefully to the size of browser’s window, we could not avoid using percentages for Tables and DIVs, so we decided to not use Strict mode instead. This is controlled with the !DOCTYPE page declaration, in particular:
You can use this declaration to switch Microsoft Internet Explorer 6 and later into strict standards-compliant mode. You turn the switch on by including the !DOCTYPE declaration at the top of your document, specifying a valid Label in the declaration, and in some cases, specifying the Definition and/or URL. The following table shows when standards-compliance is on or off. DOCTYPE URL Present URL Not Present No DOCTYPE present off off HTML (no version) off off HTML 2.0 off off HTML 3.0 off off HTML 4.0 on on HTML 4.0 Frameset on off HTML 4.0 Transitional on off HTML 4.0 Strict on on XHTML on on XML on on Unrecognized DOCTYPE on on
You can use this declaration to switch Microsoft Internet Explorer 6 and later into strict standards-compliant mode. You turn the switch on by including the !DOCTYPE declaration at the top of your document, specifying a valid Label in the declaration, and in some cases, specifying the Definition and/or URL. The following table shows when standards-compliance is on or off.
To make things easy we simply removed the DOCTYPE declaration from the problematic pages, and the CPU usage went back to normal.
In my tests, the problem does not reproduce in IE8.
Ajax is a flexible and powerful technology, but sometimes “With great power comes great responsibility” . And also some kind of fragility and sensitivity to external factors.
In this case browsing the application worked somehow fine, but triggering a partial postback we were getting the following error on the browser:
The message received from the server count not be parser. Common causes for this error are when the response is modified by calls to Response.Write(), response filters, HttpModules, or server trace is enabled.
If you get weird Ajax script errors, my experience tell me to always first check if Http compression is enabled on the web server (this is not the first time I tackle this topic), but this was not the case. An important thing to note is that the application was working perfectly fine when browser from the LAN, while it was failing when browsed from the Internet (no matter the browser we were using).
An important thing to note is that browsing from the Internet, our request was going through IAG:
Intelligent Application Gateway (IAG) is a remote access gateway that boosts productivity by giving mobile and remote workers, partners, and customers easy, flexible, and secure access to virtually any application from a broad range of devices and locations. Using a combination of SSL VPN (secure socket layer virtual private network), a Web application firewall, and endpoint security management, IAG provides employees, partners, vendors, and customers with secure and easy access from a broad range of devices and locations including kiosks, PCs, and mobile devices.
Basically IAG allows access to internal web applications over the internet. It does this by translating every HTML/script link it finds relating to the published Intranet server in the HTTP Response (both Header and Body).
To make a not so long story even shorter, there was a bug which has been fixed in .NET 3.5 SP1, which cause this error when “something” was modifying the headers collection of a request. But we were getting the error anyway (the customer was already running .NET 3.5 SP1) because IAG was modifying the body of a web request and this still supposed to cause the exception.
And just to clarify, this latter behavior is by design. The reason is proxy rewriting URLs on the fly (so we’re not just talking about IAG but every kind of proxy/firewall or even an HttpModule with this capability); the format that the UpdatePanel control uses to send the client updates is very sensitive to any modification made along the way to the client, and it is impossible to support every kind of possible modification that can be made by potentially countless applications and filters.
So there are three possible solutions in my opinion, which one to choose is up to you and your needs:
Getting feedback always nice, as it is getting suggestions about new posts as happened this morning with my colleague Michael Clemens after he read my latest entry from yesterday (so thanks Mike for sharing this!). The point is, you may get a similar experience (exactly vice versa) after having installed the “IIS media pack 1.0 - Web playlists and bit Rate Throttling package” as part of the Microsoft Web Platform installer" on a x64 OS version.
The media pack installs two 32-bit modules ("BitrateModule" and "PlaylistHandler") that prevent a 64-bit application pool from starting up, resulting in a fail-fast disabled application pool. A look at the Application event log first tells you the application pool couldn't load "%ProgramFiles%\IIS\Media\playlisthandler.dll”. Searching applicationHost.config for this module occurrence, you can simply adjust the affected <add> element with the preCondition attribute set to bitness32.
After this, the affected app pool still cannot start…
Back to the Application event log, you’ll see another module load failure for "%ProgramFiles%\IIS\Media\bitratemodule.dll". Changing this in the same manner as for the PlaylistHandler module will give you the following entry in applicationhost.config added below the <globalmodules> section:
<add name="BitrateModule" image="%ProgramFiles%\IIS\Media\bitratemodule.dll" preCondition="bitness32" />
Do not forget to search the whole “%windir%\System32\inetsrv\config\applicationHost.config” for any additional module entries (in <add> elements) on these two modules to ensure they all specify the preCondition attribute with bitness32.
And last but not least, if you did end up with a fail-fast disabled application pool, do not forget to start it again…
As sometime happens, while setting up a repro for a customer (I’m working with him on a completely different problem) I wanted to test my sample code on a 32 bit w3wp.exe instance (I am running Windows 2008 x64); nothing easier on IIS7, just create a new application pool and change its “Enable 32.Bit Applications” property to true in IIS Manager and you’re done:
But when I tried to run my code, I got this a “HTTP Error 503. The service is unavailable” message and the application pool was stopped. The Application event log contains a few entries like the following:
Log Name: Application Source: Microsoft-Windows-IIS-W3SVC-WP Date: 02/02/2009 14.19.05 Event ID: 2280 Task Category: None Level: Error Keywords: Classic User: N/A Computer: <computername> Description: The Module DLL C:\Windows\system32\RpcProxy\RpcProxy.dll failed to load. The data is the error.
I also got some of this warning:
Log Name: System Source: Microsoft-Windows-WAS Date: 03/02/2009 20.42.55 Event ID: 5139 Task Category: None Level: Warning Keywords: Classic User: N/A Computer: <computername> Description: A listener channel for protocol 'http' in worker process '4580' serving application pool '32bitPool' reported a listener channel failure. The data field contains the error number
If I changed the application pool back to a 64 bit process, everything worked fine again.
I save you the tedious details, but the point is: RpcProxy.dll is 64 bit, obviously it cannot by loaded into a 32 bit process…
As I usually do as part of my “learning experience”, I installed all the available roles for IIS7 and one of them is “RPC over HTTP Proxy”:
This installs RpcProxy.dll (and other files) and changes applicationHost.config adding a new global module:
<add name="PasswordExpiryModule" image="C:\Windows\system32\RpcProxy\RpcProxy.dll" />
How can I prevent IIS from loading it? Easy, just add the preCondition=”bitness64” attribute as follows:
<add name="PasswordExpiryModule" image="C:\Windows\system32\RpcProxy\RpcProxy.dll" preCondition="bitness64" />
Bingo, I finally have my 32 bit worker process up and running!