February, 2009

  • Never doubt thy debugger

    minWorkerThreads and autoConfig


    A couple of days ago I was helping a colleague of mine with some tests and I had the need to change the ThreadPool configuration; more precisely, I wanted to increase the minWorkerThreads value. Everything was working fine at first, because I was changing the value by code using the SetMinThreads method, but I also wanted to test it changing the values from my machine.config.

    First thing, to be on the safe side I set autoConfig=”false” (see why here) and set my <processModel> element as follows:

    <processModel autoConfig=”false” minWorkerThreads=”23” />

    Why set it to 23? Simply because I did not want it to match any default.

    I run my test page, attached Windbg to the process and had a look at !threadpool output: something was definitely wrong, as it was telling that minWorkerThreads was set to 2 (the default value is 1 * CPU number = 2 for my machine):

    0:029> !threadpool
    CPU utilization 48%
    Worker Thread: Total: 2 Running: 0 Idle: 2 MaxLimit: 40 MinLimit: 2
    Work Request in Queue: 0
    Number of Timers: 8
    Completion Port Thread:Total: 1 Free: 1 MaxFree: 4 CurrentLimit: 0 MaxLimit: 40 MinLimit: 2

    Now I see from here and with some reasoning, it is already possible to understand why I was not seeing MinLimit=46 as I was expecting… Who has the eagle eye? Smile

    I was in a hurry (as usual) and I tried a few other things, one of them was to set autoConfig back to true, here is what I got:

    0:030> !threadpool
    CPU utilization 7%
    Worker Thread: Total: 2 Running: 0 Idle: 2 MaxLimit: 200 MinLimit: 46
    Work Request in Queue: 0
    Number of Timers: 8
    Completion Port Thread:Total: 1 Free: 1 MaxFree: 4 CurrentLimit: 0 MaxLimit: 200 MinLimit: 2

    This time I’m getting it as I wanted! I’m sure someone already got it…

    But before going to the solution, let’s just explain why this cannot be because of autoConfig as you may think Nerd smile

    Here is how MSDN defines autoConfig:

    Specifies whether to automatically configure the following settings to achieve optimal performance based on the machine configuration:

    The values are set according to the KB article at http://support.microsoft.com/?id=821268.
    This attribute does not affect the .NET Framework client applications; only ASP.NET applications.
    The autoConfig attribute can be one of the following values.

    • True: Indicates that ASP.NET automatically configures the attributes in the preceding list to achieve optimal performance based on the machine configuration.
    • False: Indicates that ASP.NET should use the explicitly defined values for the attributes in the preceding list.

    The default in the Machine.config file is True, unless there is a previously existing configuration.

    minWorkerThreads is not one of the values automatically configured by the runtime, so whether autoConfig is true or false does not make any difference.

    Not quite…

    Have another look at the !threadpool output above: with autoConfig=”true” the ThreadPool limit for maxWorkerThreads is 200 (100 * 2 CPUs), while with autoConfig=”false” maxWorkerThreads is set to 40 (default 20 * 2 CPUs).

    As you can imagine, it is not possible to set minWorkerThreads higher than maxWorkerThreads… Nerd smile The runtime detects the inconsistency and simply set the “min” value to its default.

    So I have 3 points to make:

    1. Always keep an eye on default values
    2. Keep the other eye on how default values influence each other
    3. Ok, I lied, just a bit… autoConfig does not directly influence minWorkerThreads and minIoThreads, but they are instead affected by their counterparts such as maxWorkerThreads and maxIoThreads



    Quote of the day:
    Technology is dominated by two types of people: those who understand what they do not manage, and those who manage what they do not understand. – Putt's law
  • Never doubt thy debugger

    Strict mode and 100% CPU in IE 7


    I got a couple of cases of this kind recently, one where the web pages were served by PerformancePoint and the other one was a completely custom web site, but both of them shared the same common root issue: browsing some specific pages, the CPU on client got constantly at 100% and the GUI was completely frozen

    From the Internet Explorer dump we captured it was very clear that thread 2 was the one burning the CPU:

    0:000> !runaway
     User Mode Time
      Thread       Time
       2:1654      0 days 0:07:55.954
       0:1610      0 days 0:00:00.420
       7:142c      0 days 0:00:00.070
       9:1788      0 days 0:00:00.050
       4:1388      0 days 0:00:00.010
      17:8f0       0 days 0:00:00.000
      16:11cc      0 days 0:00:00.000
      15:1180      0 days 0:00:00.000
      14:11ec      0 days 0:00:00.000
      13:1778      0 days 0:00:00.000
      12:f8        0 days 0:00:00.000
      11:7dc       0 days 0:00:00.000
      10:e0c       0 days 0:00:00.000
       8:1784      0 days 0:00:00.000
       6:1420      0 days 0:00:00.000
       5:1478      0 days 0:00:00.000
       3:1338      0 days 0:00:00.000
       1:1634      0 days 0:00:00.000


    And thread 2 has his stack:

    0:002> kpL1000
    ChildEBP RetAddr  
    01bcf0bc 43d0e4d7 mshtml!CFormatInfo::GetMatchedBy
    01bcf170 43cf914f mshtml!CStyleSelector::Match+0x34d
    01bcf694 43d12175 mshtml!CStyleSheetArray::Apply+0x1e1
    01bcf6f4 43d17298 mshtml!CElement::ApplyDefaultFormat+0x490
    01bcf708 43cf8958 mshtml!CAnchorElement::ApplyDefaultFormat+0x9f
    01bcf748 43cf53e2 mshtml!CElement::ComputeFormatsVirtual+0xaa3
    01bcf764 43d0cb08 mshtml!CElement::ComputeFormats+0x3a
    01bcf9f4 43d3876b mshtml!CTreeNode::GetFancyFormatHelper+0x4b
    01bcfb10 43e167ab mshtml!CElement::UpdateFormats+0x2da
    01bcfb2c 43d3d908 mshtml!CElement::HandleMouseHoverForStyle+0x1a7
    01bcfbac 43e3f53c mshtml!CDoc::PumpMessage+0xa3f
    01bcfcf8 43d3c4a1 mshtml!CDoc::OnMouseMessage+0x4df
    01bcfe1c 43d7e137 mshtml!CDoc::OnWindowMessage+0x748
    01bcfe48 7e398734 mshtml!CServer::WndProc+0x78
    01bcfe74 7e398816 user32!InternalCallWinProc+0x28
    01bcfedc 7e3989cd user32!UserCallWinProcCheckWow+0x150
    01bcff3c 7e398a10 user32!DispatchMessageWorker+0x306
    01bcff4c 4367e6c3 user32!DispatchMessageW+0xf
    01bcffb4 7c80b683 ieframe!CTabWindow::_TabWindowThreadProc+0x189
    01bcffec 00000000 kernel32!BaseThreadStart+0x37


    Something similar also happens if you have a .NET control inside a table using percentage width and height for the control, and using percentage width for the wrapping table cell:

    ChildEBP RetAddr 
    018a84f8 77f17fda ntdll!KiFastSystemCallRet 
    018a854c 629c31f7 gdi32!NtGdiExtTextOutW+0xc 
    018a8588 629c322c lpk!InternalTextOut+0x1d4 
    018a85bc 77f31dc4 lpk!LpkExtTextOut+0x29 
    018a860c 436761bc gdi32!ExtTextOutW+0x26a 
    018a8890 43676038 mshtml!XHDC::ExtTextOutW+0x13e 
    018a88b8 43675ffa mshtml!ExtTextOutW+0x24 
    018a8e64 43675f06 mshtml!LSReExtTextOut+0x79 
    018a8f38 436763a5 mshtml!CLSRenderer::TextOutW+0x7fe 
    018a90d8 436b03e4 mshtml!CLSRenderer::BlastLineToScreen+0x4a7 
    018a9150 436aff59 mshtml!CLSRenderer::RenderLine+0x3f7 
    018a94b8 436afc4b mshtml!CDisplay::Render+0x4e0 
    018a94d0 436b004f mshtml!CFlowLayout::Draw+0x1c 
    018a9504 436ad37e mshtml!CLayout::DrawClient+0x52 
    018a98e0 436abdc0 mshtml!CDispLeafNode::DrawSelf+0x42c 
    018a9bc4 436ace26 mshtml!CDispNode::Draw+0x10d 
    018a9be4 436ac529 mshtml!CDispContainer::DrawChildren+0x3f 
    018a9f80 436abdc0 mshtml!CDispContainer::DrawSelf+0x290 
    018aa264 436ace26 mshtml!CDispNode::Draw+0x10d 
    018aa284 436ad86a mshtml!CDispContainer::DrawChildren+0x3f 
    018aa620 436abdc0 mshtml!CDispContainer::DrawSelf+0x2be 
    018aa904 436ace26 mshtml!CDispNode::Draw+0x10d 
    018aa924 436ad86a mshtml!CDispContainer::DrawChildren+0x3f 
    018aacc0 436abdc0 mshtml!CDispContainer::DrawSelf+0x2be 
    018aafa4 43667b64 mshtml!CDispNode::Draw+0x10d 
    018acf50 43667a64 mshtml!CDispRoot::DrawBand+0xc7 


    The stacks look quite similar for the one we had for a couple of old bugs already fixed for Internet Explorer when dynamically adding cells via DHTML (back in 2002-2003); there was also another bug in .NET 2.0 which has been fixed in KB# 928365.

    For both customers the problem reproduced only with IE7, IE was running in Strict mode (but not in Quirks mode) and the page layout was made with nested Tables and DIVs whose dimensions (width and/or height) where expressed in percentage.

    Since we needed the page layout to adapt gracefully to the size of browser’s window, we could not avoid using percentages for Tables and DIVs, so we decided to not use Strict mode instead. This is controlled with the !DOCTYPE page declaration, in particular:

    You can use this declaration to switch Microsoft Internet Explorer 6 and later into strict standards-compliant mode. You turn the switch on by including the !DOCTYPE declaration at the top of your document, specifying a valid Label in the declaration, and in some cases, specifying the Definition and/or URL. The following table shows when standards-compliance is on or off.

    DOCTYPE URL Present URL Not Present
    No DOCTYPE present off off
    HTML (no version) off off
    HTML 2.0 off off
    HTML 3.0 off off
    HTML 4.0 on on
    HTML 4.0 Frameset on off
    HTML 4.0 Transitional on off
    HTML 4.0 Strict on on
    XHTML on on
    XML on on
    Unrecognized DOCTYPE on on

    To make things easy we simply removed the DOCTYPE declaration from the problematic pages, and the CPU usage went back to normal.

    In my tests, the problem does not reproduce in IE8.



    Quote of the day:
    When ideas fail, words come in very handy. – Johann Wolfgang von Goethe
  • Never doubt thy debugger



    Ajax is a flexible and powerful technology, but sometimes “With great power comes great responsibilityNerd smile. And also some kind of fragility and sensitivity to external factors.

    In this case browsing the application worked somehow fine, but triggering a partial postback we were getting the following error on the browser:


    The message received from the server count not be parser. Common causes for this error are when the response is modified by calls to Response.Write(), response filters, HttpModules, or server trace is enabled.

    If you get weird Ajax script errors, my experience tell me to always first check if Http compression is enabled on the web server (this is not the first time I tackle this topic), but this was not the case. An important thing to note is that the application was working perfectly fine when browser from the LAN, while it was failing when browsed from the Internet (no matter the browser we were using).

    An important thing to note is that browsing from the Internet, our request was going through IAG:

    Intelligent Application Gateway (IAG) is a remote access gateway that boosts productivity by giving mobile and remote workers, partners, and customers easy, flexible, and secure access to virtually any application from a broad range of devices and locations. Using a combination of SSL VPN (secure socket layer virtual private network), a Web application firewall, and endpoint security management, IAG provides employees, partners, vendors, and customers with secure and easy access from a broad range of devices and locations including kiosks, PCs, and mobile devices.

    Basically IAG allows access to internal web applications over the internet. It does this by translating every HTML/script link it finds relating to the published Intranet server in the HTTP Response (both Header and Body).

    To make a not so long story even shorter, there was a bug which has been fixed in .NET 3.5 SP1, which cause this error when “something” was modifying the headers collection of a request. But we were getting the error anyway (the customer was already running .NET 3.5 SP1) because IAG was modifying the body of a web request and this still supposed to cause the exception.

    And just to clarify, this latter behavior is by design. The reason is proxy rewriting URLs on the fly (so we’re not just talking about IAG but every kind of proxy/firewall or even an HttpModule with this capability); the format that the UpdatePanel control uses to send the client updates is very sensitive to any modification made along the way to the client, and it is impossible to support every kind of possible modification that can be made by potentially countless applications and filters.

    So there are three possible solutions in my opinion, which one to choose is up to you and your needs:

    1. Do not use the UpdatePanel: this means you cannot use Ajax and you must always do a full page postback
    2. Do not use the proxy, or make sure it is not tampering the request’s body (if possible, of course)
    3. Write your URLs so that they do not need to be rewritten by the proxy: this is the most complex, and maybe not really applicable solution (simply because of the names translation needed to access internal resources from the Internet)



    Quote of the day:
    An expert is a person who has made all the mistakes that can be made in a very narrow field. – Niels Bohr
  • Never doubt thy debugger

    Again on 503, “Service Unavailable” in IIS7 x64


    Getting feedback always nice, as it is getting suggestions about new posts as happened this morning with my colleague Michael Clemens after he read my latest entry from yesterday (so thanks Mike for sharing this!). The point is, you may get a similar experience (exactly vice versa) after having installed the “IIS media pack 1.0 - Web playlists and bit Rate Throttling package” as part of the Microsoft Web Platform installer" on a x64 OS version.

    The media pack installs two 32-bit modules ("BitrateModule" and "PlaylistHandler") that prevent a 64-bit application pool from starting up, resulting in a fail-fast disabled application pool. A look at the Application event log first tells you the application pool couldn't load "%ProgramFiles%\IIS\Media\playlisthandler.dll”. Searching applicationHost.config for this module occurrence, you can simply adjust the affected <add> element with the preCondition attribute set to bitness32.

    After this, the affected app pool still cannot start… Thinking smile

    Back to the Application event log, you’ll see another module load failure for "%ProgramFiles%\IIS\Media\bitratemodule.dll". Changing this in the same manner as for the PlaylistHandler module will give you the following entry in applicationhost.config added below the <globalmodules> section:

    <add name="BitrateModule" image="%ProgramFiles%\IIS\Media\bitratemodule.dll" preCondition="bitness32" />

    Do not forget to search the whole “%windir%\System32\inetsrv\config\applicationHost.config” for any additional module entries (in <add> elements) on these two modules to ensure they all specify the preCondition attribute with bitness32.

    And last but not least, if you did end up with a fail-fast disabled application pool, do not forget to start it again… Nerd smile



    Quote of the day:
    Old age is not so bad when you consider the alternatives. – Maurice Chevalier
  • Never doubt thy debugger

    “Service Unavailable” in IIS7 with 32 bit application pool


    As sometime happens, while setting up a repro for a customer (I’m working with him on a completely different problem) I wanted to test my sample code on a 32 bit w3wp.exe instance (I am running Windows 2008 x64); nothing easier on IIS7, just create a new application pool and change its “Enable 32.Bit Applications” property to true in IIS Manager and you’re done:

     Enable 32 bit application

    But when I tried to run my code, I got this a “HTTP Error 503. The service is unavailable” message and the application pool was stopped. The Application event log contains a few entries like the following:

    Log Name:      Application
    Source:        Microsoft-Windows-IIS-W3SVC-WP
    Date:          02/02/2009 14.19.05
    Event ID:      2280
    Task Category: None
    Level:         Error
    Keywords:      Classic
    User:          N/A
    Computer:      <computername>
    Description: The Module DLL C:\Windows\system32\RpcProxy\RpcProxy.dll failed to load.  The data is the error.

    I also got some of this warning:

    Log Name:      System
    Source:        Microsoft-Windows-WAS
    Date:          03/02/2009 20.42.55
    Event ID:      5139
    Task Category: None
    Level:         Warning
    Keywords:      Classic
    User:          N/A
    Computer:      <computername>
    Description: A listener channel for protocol 'http' in worker process '4580' serving application pool '32bitPool' reported a listener channel failure.  The data field contains the error number

    If I changed the application pool back to a 64 bit process, everything worked fine again.

    I save you the tedious details, but the point is: RpcProxy.dll is 64 bit, obviously it cannot by loaded into a 32 bit process…


    As I usually do as part of my “learning experience”, I installed all the available roles for IIS7 and one of them is “RPC over HTTP Proxy”:

    RPC over HTTP

    This installs RpcProxy.dll (and other files) and changes applicationHost.config adding a new global module:

    <add name="PasswordExpiryModule" image="C:\Windows\system32\RpcProxy\RpcProxy.dll" />

    How can I prevent IIS from loading it? Easy, just add the preCondition=”bitness64” attribute as follows:

    <add name="PasswordExpiryModule" image="C:\Windows\system32\RpcProxy\RpcProxy.dll" preCondition="bitness64" />

    Bingo, I finally have my 32 bit worker process up and running! Nerd smile



    Quote of the day:
    We are born charming, fresh and spontaneous and must be civilized before we are fit to participate in society – Judith Martin
Page 1 of 1 (5 items)