• Never doubt thy debugger

    How to automate Process Monitor


    The other day I needed to capture a Process Monitor trace on a machine to troubleshoot a problem where the entire OS GUI was “broken” after the resume from screensaver; with “GUI broken” I mean that clicking on any icon on the desktop has no effect, the right click menu does not work etc…, but we are still able to access the Task Manager and run tasks (command line etc…) from the File > New Task (Run…) menu. This suggests the user’s registry is for some reason unloaded during when the screensaver is running, but the point is: if I run Procmon before the screensaver starts and wait for the problem to reproduce (to have a complete trace) then I’m unable to save the trace because I cannot access any running task which a GUI unless I start a new one or reboot the machine. Either way the trace is lost.

    Coincidentally also my colleague Stefano had a similar problem and we (he, actually) found a couple of command line switches that can be used to control Procmon for such situations, in particular “/BackingFile” and “/Terminate”: /backingfile tells Procmon where to automatically save the trace, while /terminate actually starts a new instance of Procmon, terminates all other Procmon instances and exits.

    So now I can start Procmon like:

    procmon /quiet /minimized /backingfile c:\temp\trace.pml

    before the screensaver starts, wait for the problem to reproduce, resume the OS and even without access to the GUI and from the Task Manager run:

    procmon /terminate

    Now I can safely reboot the machine and have my trace to analyze; of course you can also use those (and other) command line switches in a batch file or scripts if you need to automate the tool.



    Quote of the day:
    The first question I ask myself when something doesn't seem to be beautiful is why do I think it's not beautiful. And very shortly you discover that there is no reason. - John Cage
  • Never doubt thy debugger

    HashTable insert failed. Load factor too high


    I already encountered a few recurrences of this errors and every time it has been a torture to capture a correct set of data (especially memory dumps) to review because this usually happens sporadically and you could have to wait weeks if not months before getting it again. There are a couple of hotfix available for this error message (see http://support.microsoft.com/default.aspx?scid=kb;EN-US;831730 and http://support.microsoft.com/default.aspx?scid=kb;EN-US;893314) but you already have them if you’re on the latest Service Pack for the .NET Framework.

    Unless you’re using HashTable objects yourself… If that’s the case you might be running into a race condition which leads to the error, so you might want to consider to use the following code to synchronize the HashTable using SyncLock:

    Dim MyHashTable as new Hashtable()
    SyncLock MyHashtable.Syncroot
    MyHashtable.Add (myKey, myValue)
    End SyncLock

    As an alternative you can use the Hashtable.Synchronized() method to return a synchronized (thread-safe) wrapper for the HashTable object.



    Quote of the day:
    Everyone is a genius at least once a year. The real geniuses simply have their bright ideas closer together. - Georg Christoph Lichtenberg
  • Never doubt thy debugger

    /bin is not for native assemblies


    After 4 years as a Support Engineer for ASP.NET, when I get a new case I usually can make a decently reliable idea pretty soon (often from the problem description the customer gives us, if it has enough details) about the problem complexity, how long it will take to resolve etc…, but every now and then (luckily not too often) there are occasions, as the one I’m going to talk about, where a fairly common problem as a “File not found” issue turns into something able to leave you completely puzzled and after you have considered almost every possibility you are left without any clue or idea about how to continue with troubleshooting. And again most of the times in those situations the solution comes from a fresh pair of eyes (and a new, well skilled brain) which looking at the problem from a different perspective can spot that small clue which makes the rest appear as obvious as the easiest of the problems… smile_thinking

    The problem

    This time the problem description was well written: the customer developed a custom authentication mechanism for web applications, defined an authentication interface which can be implemented by other developers etc… The product exposes a “DoWorkProc” interface developed with Visual Studio 2003; they also developed an HttpModule “My Authentication” filter that allows customers to setup authentication by defining an assembly that implements an IAuth interface, and the customer shipped his product with a sample authentication assembly named MyAuthenticationSample.dll. My Authentication.dll and MyAuthenticationSample.dll assemblies are copied into the bin folder of the DoWorkProc application. When DoWorkProc is first referenced, the ASP.NET worker process binds the assemblies from the DoWorkProc /bin folder along with various system assemblies into the C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\Temporary ASP.NET Files folder. This all works properly and an intermediate binding assembly is created. Now the problem: if I create a new project under Visual Studio to implement the IAuth interface and I change the assembly name and project namespace to MyAuthenticationSample2 but making sure that everything else in the project is the same, the binding process fails with message “File or assembly name masDotNet, or one of its dependencies not found” and it highlights the “<add assembly="*"/>” line in machine.config; it appears that not all assemblies get copied to the “Temporary ASP.NET Files” folder as well.


    This smelt of a versioning problem, partially supported by the fact that using the compiled sample assembly resulted in the error above, while if the customer added the source project for the control to the new solution and compiled the them altogether, it worked…

    The customer sent a repro I was able to run on my machine (it is always easier when we can repro, it’s much easier to debug and have a close look at what’s going wrong), and at first I actually found a versioning problem which fitted quite well with the error we got. The runtime was not able to find the appropriate version of a file it was looking for, hence the error: in the failing sample we were trying to load masDotNet.dll version 10.6.2719.37534, while the file version I had on disk (and the same loaded in the working sample) was version

    But… why does the error message mentions masDotNet.dll while to repro I had to use MyAuthenticationSample.dll and MyAuthenticationSample2.dll? smile_eyeroll Well, checking with the customer it turned out that masDotNet.dll is a mixed mode (C++ .NET) dll so we actually have two version numbers for that file: the version number in the PE header (what WinDBG shows, or what you can see from Windows Explorer if you look at the dll properties) and the version number in the assembly manifest have nothing to do with each other, it is the one in the manifest that is important for .NET assembly loading (binding), and at a more careful look that version was correct.

    As often in this kind of cases I asked Doug to have a look at the problem with me and after some time spent working on the repro (basically to run again through the troubleshooting I did until that moment to double check) we confirmed in neither the non-working case nor the working case, neither MyAuthentication.dll nor MyAuthenticationSample.dll (nor MyAuthenticationSample2.dll in the non-working case) reference any mas* assembly in their manifest, neither directly nor indirectly. So why having differing versions of MyAuthentication.dll and MyAuthenticationSample.dll affects the version of masDotNet.dll that we look for? Very odd… smile_thinking

    We then had a look at masDotNet.dll (which is managed C++) and we saw that from a native DLL dependency point of view it depends two third party native dlls we had in the /bin folder. Well, what are they doing in the BIN folder? ASP.NET’s BIN folder is for .NET assemblies, not native DLLs and the native OS loader will not find the native DLLs after they get shadow copied. So we did what should be done with native DLLs: put them in a location that was discoverable on the system PATH and removed them from the BIN folder.

    I then checked and everything worked (also clearing out “temporary asp.net files” in between each test just to be sure), in both the previously working and non-working cases!


    There would be some additional questions to answer, such as why under some circumstances the native dlls in /bin where not a problem, while in other cases it was and consistently failed; unfortunately we did not had enough time left to continue our investigations (the customer needed urgently a working solution and preferred to not spend more time on the theory behind the issue), but here is a good bet: of the two native third party dlls, one (let’s call it A.dll) has a static dependency on the other one (call it B.dll). In the working case, B.dll was loaded in memory before A.dll, so when it was its turn it was able to find all the needed dependencies already in memory. But in the failing case for some reason (maybe because of timing, but I’m not really sure) A.dll was loaded first and then load of B.dll failed because it was not discoverable on the normal loader DLL search path.

    Bottom line: the /bin folder is a sensitive place, for the above and other ones (check here for another example), be careful what you store in there! smile_regular



    Quote of the day:
    The real problem is not whether machines think but whether men do. - B. F. Skinner
  • Never doubt thy debugger

    DropDownList with tooltip


    I recently got this question from a customer whom is binding some data to a DropDownList control inside a GridView; the various items inside the DropDownList has variable length while the control itself per their page resign requirements must be fixed, making it difficult to fully read and understand the text associated with the selected item, so the customer asked for some hints to add a tooptip functionality to the DropDownList. Before raising the call he had already searched the Internet but without much luck; I did the same myself and apparently there are plenty of samples showing how to display a tooltip when the mouse is over an item already selected in a DropDown control, but the customer wanted something different: he needed to show the tooltip over each item in the DropDownList before actually selecting them to help users make the appropriate selection.

    Well, for Internet Explorer 7 (al later) it is possible to use the “Title” attribute for the <option> tag:

    for (int i = 0; i < GridView1.Rows.Count; i++)
        //Add logic to fill the GridView
        //Now let's iterate through the DropDown controls to add a "Title" to the
        //resulting HTML "<option>" elements: this will add the toolip to highlighted element
        for (int i2 = 0; i2 < dropDownList.Items.Count; i2++)
            dropDownList.Items[i2].Attributes.Add("Title", dropDownList.Items[i2].Text);


    Here is how it looks like in IE7:

    Tooltip in IE7


    Even if Firefox (at least the version I’ve tried) does not respect the control width we have set through CSS and the DropDown for this sample is large enough to read the complete text, the tooltip is displayed:

    Tooltip in Firefox


    Unfortunately this is not supported by previous versions if IE, though… if anyone has a suggestion, feel free to add a comment! smile_nerd



    Quote of the day:
    Calamities are of two kinds: misfortunes to ourselves, and good fortune to others. - Ambrose Bierce
  • Never doubt thy debugger

    The show goes on even if I’m on vacation


    … so while I was taking a break from computers to move to a new house (I’m still packing and unpacking things), a few notable things happened: Internet Explorer 8 beta 2 has been released, Google released its new browser Chrome and my colleagues from the Italian Enterprise Support Group have started a team blog! So if you can at least read Italian I suggest you to keep an eye on it, knowing that smart people you can expect a lot of good content from them smile_regular 



    Quote of the day:
    I was always taught to respect my elders and I've now reached the age when I don't have anybody to respect. - George Burns
  • Never doubt thy debugger

    How many application pools can you cope with?


    This is an interesting question I have to admit I had not thought to until we got a case from a customer whom was in trouble with his production servers: how many application pools can you run on IIS? Event better, how many w3wp.exe instances can the OS cope with? As quite often happens in our job, the correct answer is: it depends…

    First, the problem. The customer was a hosting company so they had a large numbers of sites on each web server, around 450, each of them with its own application pool, each application pool run under its own specific account and aspExecuteInMTA was set to 1 on each server. Under some circumstances asp applications from some web sites are failing with the following message (roughly translated from Spanish):

    Server error 'ASP 0177 : 80070057'
    Error in Server.CreateObject
    /site/page.asp, line 338

    When the problem occurred the customer recycled the two or three top consuming web sites in terms of memory, threads and handles; then the failing web site was usually back online. Note that the failing web site was always only affected by other problematic ones, it was almost never the culprit.

    At first we were distracted by the fact that the failing web sites where using Access as the database and we also got some errors like:

    Server error 'ASP 0177 : 80070583'
    Error in Server.CreateObject
    /database.asp, line 9 -- "Provider=Microsoft.Jet.OLEDB.4.0;Data source=" & server.MapPath("../data/bd.mdb")

    This is a deprecated configuration, see Using Microsoft JET with IIS:

    The Access ODBC Driver and Microsoft OLE DB Provider for Microsoft Jet are not intended to be used with high-stress, high-concurrency, 24x7 server applications, such as web, commerce, transactional, messaging servers, and so on.

    Anyway we soon found that was not the right direction. 450 application pools are a lot, plus there was the thing about recycling the most handles and threads consuming pools… this suggests desktop heap exhaustion.

    A bit of theory

    Desktop Heap overview has a good explanation of what Desktop Heap is and how things might start to go wrong when the system runs low in Desktop Heap storage; it also has a few interesting bits about IIS:

    Regarding non-ASP.NET applications, IIS 6.0 has been tested on a well-configured mainframe server running up to 2,000 concurrent worker processes, each serving one application pool, but not using unique identities. In practice, a design of up to 500 simultaneous application pools is achievable, depending on the application requirements and assuming hardware resources are not a significant constraint. It is important to configure the UseSharedWPDesktop registry setting mentioned above when using more than 60 application pools with unique identities

    My colleague Nicolas also has a post about Desktop Heap and Windows Stations here. Having one shared desktop can theoretically bring some security issues, but David Wang explains well this scenario:

    "we should use a common desktop when the number of application pools with unique identities configured on a server is greater than 60". In other words, IIS does not have a problem running >60 application pools which uses <60 unique identities. The limitation in question is the number of user desktops associated with each unique application pool identity that Windows simultaneously supports.

    The following are the interesting combinations and consequences:

    1. If UseSharedWPDesktop=0 and all application pools launch as Network Service, then there is exactly one Desktop (of Network Service) shared by all the application pools. You have no isolation of desktop nor process identity between application pools.
    2. If UseSharedWPDesktop=0 and all application pools launch as unique user identity, then there is a unique Desktop for each user identity, which is obviously not shared. This is limited to a number around 60 due to Desktop heap limit in Windows Server 2003. This means that you can only have ~60 application pools perfectly isolated by desktop and process identity simultaneously running.
    3. If UseSharedWPDesktop=1 and all application pools launch as unique user identity, then there is one Desktop that is shared by all the user identities. This removes the limit of 60 at the cost of not perfectly isolated application pools. You have no desktop isolation but retain process identity isolation between application pools.
    4. UseSharedWPDesktop=1 and all application pools launch as Network Service is pretty much a degenerate case that is not interesting. :-)

    Back to the problem

    The customer was already aware of those blog posts so they were already using UseSharedWPDesktop=1 (otherwise they were not able to run more than 50 concurrent application pools), set sessionViewSize to 64 Mb and the default value for “HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\SubSystems\Windows”.

    To know more about Desktop Heap status on the server we used Desktop Heap Monitor: this comes with a comprehensive usage guide, anyway here are some quick steps to use it:

    1. Log in with an account that has Administrator privilege. Note that you must either do this at the physical interactive console (i.e. not through Terminal Services) OR connect to the console session using mstsc /console (only possible on XP or Server 2003)
    2. Open a command prompt and change to "c:\kktools\dheapmon8.0"
    3. For uni-processor systems run
      • DHEAPINST -f
    4. For multi-processor systems (including single processor hyper-threaded systems) run
      • SET _NT_SYMBOL_PATH=symsrv*symsrv.dll*C:\Symbols*http://msdl.microsoft.com/download/symbols
      • You will be prompted to agree to the symbol server terms of use. You must accept these for the tool to work
        The server in question must be able to make an outgoing connection to the Internet for this to work
    5. For either 3. or 4. you should see
      • dheapinst - Desktop Heap Monitor installed successfully
    6. DHEAPMON uses a driver in order to read information from the kernel memory. This must be loaded by the following command:
      • DHEAPMON –l
      • You should see this confirmation message: Dheapmon - Driver loaded successfully
    7. You are now ready to run the tool and capture the desktop heap information. Run the following command:
      • DHEAPMON -v > desktopheap.txt
      • Important: if the output text file does not contain a line like this: “Windowstation: (Service-0x0-3e7$) SessionID: 0” (without quotation marks) then it is almost certain that you did not connect to the server using Terminal Services Client using the /console switch. You must connect using /console to get the full information required

    To automate the process of gathering Desktop Heap information, create a batch file called HeapMon.bat containing the following:

    @echo off
    rem vars C B A are day month year in the for line below, change them to change date
    for /F "tokens=2,3,4 delims=/- " %%A in ('date /T') do set DT=%%C_%%B_%%A
    for /F "tokens=1,2 delims=/: " %%A in ('time /T') do set TM=%%A-%%B
    echo %DT%
    echo %TM%
    Dheapmon -v > dheapmon_out_%DT%_%TM%.txt

    This will create an output file uniquely named with the date and time included; you can use task scheduler to run this batch file at regular intervals, e.g. once per hour.

    Once finished all diagnostic activity with DHeapMon, unload the device driver with:


    To uninstall dheapmon, run:


    From the report I got is was clear that the customer was really using only 1 desktop within IIS WinSTA (remember he was using a shared desktop across his application pools); as IIS WinSTA is created as non interactive, this single desktop is 512 Kb but it was exhausted at 99.2% when then problem appeared:

    Windowstation: (WP_WINSTA-054f77ac-6ece-4080-b3a2-b0bf08dacaab) SessionID: 0
    Desktop: (Default) Addr: bbc30000
    Desktop Heap          524288 (0x   80000) Bytes
    Committed             520192 (0x   7f000) Bytes
    Uncommitted             4096 (0x    1000) Bytes
    Allocated             520072 (0x   7ef88) Bytes
    Total Freed              120 (0x      78) Bytes
    Unused                  4216 (0x    1078) Bytes
    Used Rate                    (      99.2) %

    To calculate SessionViewState size we should do something like:

    312Kb (Winsta0/Default) +
    64Kb (Winsta0/Disconnect) +
    128Kb (Winsta0/Winlogon) + 
    7 * 512Kb (all other non interactive winsta) +
    512Kb (IIS non interactive winsta WP_WINSTA-xxx)
    == 4600 Kb  (< 5 Mb)

    So apparently the problem is not on SessionViewSize bur rather just on the non interactive Desktop Heap size; moreover setting this value to 64 Mb as the customer did is not necessary because they were using a bit less than 4 Mb for that value so also the default (20 Mb) were far enough.

    The problem of increasing this one too much is that not only IIS owns one, so we first tried using 1024 Kb:

                    312 + 64 + 128 + 7 * 1024 = 7672 Kb (< 7 Mb)

    So we tried with the following (not the customer was not using /3gb):

    • UseSharedWPDesktop enabled
    • Delete the SessionViewSize registry key to get the default 20 Mb
    • SharedSection=1024,3072,1024

    This first try allowed the server to run fine a bit longer than before but at the end the problem was there again, and another heapmon trace showed we were still exhausting Desktop Heap… we gave another try with SharedSection=1024,3072,2048 (312 + 64 + 128 + 7 * 1024 = 14840 Kb (< 15 Mb).

    Bingo, this time we did it! smile_regular 

    Of course be very careful when you manipulate the Desktop Heap configuration and if you’re not sure how to do (or if you’re doing it right, or if you need help to fine tune your settings etc…) give Microsoft Support a call! smile_regular


    To close with some general recommendations:

    1. Any kind of platform that has to support more than concurrent 450 processes should be considered to migrate under x64, and to really care about context switching
    2. Windows 2008 Server with Dynamic Session Space removes the SessionViewSize limits, but desktop heap limits would still apply
    3. 450 processes identity is also a very big because logon session & token cache might duplicate kernel memory
    4. Always be very careful when changing those values since this might have effect on other kernel quota

    The point is we will always have inherent x32 bit limitations that we cannot workaround and 450 application pools is a pretty close “practically tested” limit…




    Quote of the Day:
    Women like silent men. They think they're listening. - Marcel Archard
  • Never doubt thy debugger

    Visual Studio 2008 SP1 and .NET 3.5 SP1 available for download


    Microsoft Visual Studio 2008 Service Pack 1 (exe)

    This download installs Visual Studio 2008 Service Pack 1 (SP1) and the .NET Framework 3.5 SP1. SP1 addresses issues that were found through a combination of customer and partner feedback, as well as internal testing. These service packs offer Visual Studio and .NET Framework users improvements in responsiveness, stability and performance.
    Click here for more information regarding about these service packs.

    Microsoft .NET Framework 3.5 Service Pack 1

    .NET Framework version 3.5 Service Pack 1 provides the following new features and improvements:

    • ASP.NET Dynamic Data, which provides a rich scaffolding framework that enables rapid data driven development without writing code, and a new addition to ASP.NET AJAX that provides support for managing browser history (back button support). For more information, see What’s New in ASP.NET and Web Development.
    • Core improvements to the CLR (common language runtime) that include better layout of .NET Framework native images, opting out of strong-name verification for fully trusted assemblies, improved application startup performance, better generated code that improves end-to-end application execution time, and opting managed code to run in ASLR (Address Space Layout Randomization) mode if supported by the operating system. Additionally, managed applications that are opened from network shares have the same behavior as native applications by running with full trust.
    • Performance improvements to WPF (Windows Presentation Foundation), including a faster startup time and improved performance for Bitmap effects. Additional functionality for WPF includes better support for line of business applications, native splash screen support, DirectX pixel shader support, and the new WebBrowser control.
    • ClickOnce application publishers can decide to opt out of signing and hashing as appropriate for their scenarios, developers can programmatically install ClickOnce applications that display a customized branding, and ClickOnce error dialog boxes support links to application-specific support sites on the Web.
    • The Entity Framework is an evolution of the existing suite of ADO.NET data access technologies. The Entity Framework enables developers to program against relational databases in according to application-specific domain models instead of the underlying database models. For more information, see Getting Started with the Entity Framework. The Entity Framework introduces some additional features, including support for new SQL Server 2008 types, default graph serialization of Entities, and the Entity Data Source. This release of the Entity Framework supports the new date and file stream capabilities in SQL Server 2008. The graph serialization work helps developers who want to build Windows Communication Foundation (WCF) services that model full graphs as data contracts. The Entity Data Source provides a traditional data source experience for ASP.NET Web application builders who want to work with the Entity Framework.
    • LINQ to SQL includes new support for the new date and file stream capabilities in SQL Server 2008.
    • The ADO.NET Data Services Framework consists of a combination of patterns and libraries, which enable data to be exposed as a flexible REST (Representational State Transfer)-based data service that can be consumed by Web clients in a corporate network or across the Internet. The ADO.NET Data Services Framework makes data service creation over any data source. A conceptual view model of the underlying storage schema can easily be exposed through rich integration with the ADO.NET Entity Framework. Services created by using the ADO.NET Data Services Framework, and also compatible Windows Live (dev.live.com) services, can be easily accessed from any platform. For client applications that are running on Microsoft platforms, a set of client libraries are provided to make interaction with data services simple. For example, .NET Framework-based clients can use LINQ to query data services and a simple .NET Framework object layer to update data in the service.
    • Windows Communication Foundation now makes the DataContract Serializer easier to use by providing improved interoperability support, enhancing the debugging experience in partial trust scenarios, and extending syndication protocol support for wider usage in Web 2.0 applications.
    • The .NET Framework Data Provider for SQL Server (SqlClient) adds new support for file stream and sparse column capabilities in SQL Server 2008.

    Here is the link to download the full package: Microsoft .NET Framework 3.5 Service Pack 1


    Quote of the Day:
    Resentment is like taking poison and hoping the other person dies. - St. Augustine

  • Never doubt thy debugger

    Charting with LogParser


    A picture worth 1000 words and this is true also when you have to analyze IIS logs to get some statistics on your site; as you already know, LogParser is extremely flexible and very useful in such situation and its chart output format is what it’s needed.

    The first thing you need is a copy of Office Web Components installed on the machine; if you don’t, LogParser will complain and return this error:

    Error creating output format "chart": This output format requires a licensed Microsoft Office Chart Web Component to be installed on the local machine

    From LogParser help:


    name of chart type


    Chart type

    The set of available chart types depends on the version of the Microsoft Office Web Components installed on the local computer.
    For a list of the available chart types, type the following
    help command from the command-line shell: 

         LogParser -h -o:CHART

    Currently the latest Office Web Components version available (the newest I’ve been able to find on the Internet) is version 11 which has been released for Office 2003.

    Changes in the 2007 Office System

    Description: OWC11 is removed from the 2007 Office system and now ships only with Microsoft Office Project 2007. The DataFinder functionality in OWC11 is also removed. This affects solutions that use the spreadsheet, chart, pivot table, and data source control functionality of OWC11 in a Web page or client program that requires installation of this ActiveX control on the user's computer. The user experience varies depending upon the host container of the OWC11 ActiveX control.

    For Web pages, users might see a broken icon or a notification that OWC is required. For client programs, users might receive a runtime error or other notification. In these situations, users download OWC11 from the Microsoft Download Center. The notification message generated by Office programs provides a link to the download location for OWC11. Neither Microsoft Office Excel 2007 nor SharePoint Designer 2007 now generate Web pages that use OWC.

    Reason for change: New technologies such as Windows SharePoint Services 3.0 with Excel server side capabilities and the new charting component in the 2007 Office system replace some features of OWC11.

    And if you have Office 2007 like me? I installed OWC11 anyway and everything is still working fine so you should be fine; just in case you need the download links, here they are: Office 2003 Add-in: Office Web Components and Microsoft Office 2003 Web Components Service Pack 1 (SP1) for the 2007 Microsoft Office System.

    A few samples

    Bytes per data type as exploded pie

        "SELECT TO_UPPERCASE(EXTRACT_EXTENSION(cs-uri-stem)) AS PageType, MUL(PROPSUM(sc-bytes),100.0) AS Bytes 
        INTO Pie.gif FROM *.log 
        GROUP BY PageType ORDER BY Bytes DESC" 
        -chartType:PieExploded -chartTitle:"Bytes per data type" -categories:off -o:chart -i:iisw3c

    Bytes per data type as exploded pie


    Calculating percentages per data type

        "select extract_extension(cs-uri-stem) as Resource, mul(propcount(*),100.0) as ResourceHits 
        into percentage.jpg from *.log 
        group by Resource order by ResourceHits desc" 
        -chartType:PieExploded3D -chartTitle:"Percentages per data type" -categories:off -o:chart -i:iisw3c
    Percentage per data type


    .aspx page hits per minute

        "select quantize(time, 60) as TimeGenerated, count(*) as Hits 
        into qnt.gif from *.log 
        where to_lowercase(extract_extension(cs-uri-stem))='aspx' group by TimeGenerated" 
        -i:iisw3c -o:chart -chartType:Line
    aspx hits per minute


    Getting the number of unique visitors (actually the number of unique client IP addresses) is a bit more complex. We could try the following:

    select date, count(distinct c-ip) into UniqueIPs.gif from *.log group by date" -i:iisw3c -o:chart -chartType:Line

    but we get this error:

    Error: Semantic Error: aggregate functions with DISTINCT arguments are not supported with GROUP BY clauses

    The solution in this case is to split the query into two separate commands; for example you could create a .bat file (or script, or whatever method you prefer) and run the following:

    @echo off 
        "select distinct date, c-ip into dates.txt from *.log" -i:iisw3c -o:w3c
        "select date, count(c-ip) as UniqueIPs into UniqueIPs.gif 
        from dates.txt group by date order by date" 
        -i:iisw3c -o:chart -chartType:SmoothLine -view:on

    unique IPs per day

    Advanced carting with configuration scripts

    Using an external script file (JScript or VBScript) it is possible to have a deeper control over the produced chart; this is based on two global objects which expose methods and properties that can be used to modify parameters such as the chart colors, the chart fonts, and many other attributes. Those two global objects are instances of the chartSpace and chart objects of the Microsoft Office Web Components ChartSpace object model, and they are named "chartSpace" and "chart"; for more information on those objects you can have a look at the MSDN ChartSpace Object Model documentation.

    Taken from the LogParser documentation, we can use the following sample script to add a caption to the previous UniqueIPs chant and make the background color transparent:

    // Add a caption
    chartSpace.HasChartSpaceTitle = true;
    chartSpace.ChartSpaceTitle.Caption = "Generated by Log Parser 2.2";
    chartSpace.ChartSpaceTitle.Font.Size = 8;
    chartSpace.ChartSpaceTitle.Position = chartSpace.Constants.chTitlePositionBottom;
    // Change the background color
    chart.PlotArea.Interior.Color = chartSpace.Constants.chColorNone;
    @echo off 
        "select distinct date, c-ip into dates.txt from *.log" -i:iisw3c -o:w3c
        "select date, count(c-ip) as UniqueIPs into UniqueIPs.gif 
        from dates.txt group by date order by date" 
        -i:iisw3c -o:chart -chartType:SmoothLine -config:MyScript.js -groupSize: 800x600 -view:on

    unique IPs per day with config script

    From here you can experiment using the ChartSpace object model to further customize the appearance of your charts: the limit is your imagination smile_regular




    Quote of the day:
    Ability will never catch up with the demand for it. - Malcolm Forbe
  • Never doubt thy debugger

    LogParser did it again: application pool recycle


    This started as a “standard” w3wp.exe crash, but this time IIS was also randomly disabling one of the application pools on the server. This is a protection mechanism we have since IIS 6 called Rapid Fail Protection that disables ad application pool (for security and stability reasons) if it fails (encounters severe errors) too often: how often and how much time it’s configurable (see Configuring Rapid-Fail Protection in IIS 6.0), the default is 5 failures in 5 minutes.

    Looking at the System event log there were plenty of messages like:

    A worker process with process id of 'xxx' serving application pool 'MyAppPool' has requested a recycle because it reached its virtual memory limit.

    And also:

    A process serving application pool 'MyAppPool' exceeded time limits during shut down. The process id was 'xxx'.

    How often? As usual, LogParser comes at hand.

        "select quantize(TimeGenerated, 300) as TimeGen, count(*) as NumberOfrecycles
        into memory_limit_recycles.log from system.evt 
        where EventID = 1077 and Message like '%\'MyAppPool%' group by TimeGen order by TimeGen" 
        -i:evt -o:nat -rtp:-1

    Note the “quantize” function: LogParser help states “Truncates a value to the nearest multiple of another value”; what it actually does in the command above is count the number of recurrences of a certain event in “blocks” of 5 minutes (300 seconds in my sample).

    The output is something like:

    Generated           NumberOfRecycles 
    ------------------- ----------------
    2008-07-06 21:55:00 5
    2008-07-06 22:00:00 5
    2008-07-06 22:05:00 5
    2008-07-06 22:10:00 4
    2008-07-06 22:15:00 4
    2008-07-06 22:20:00 3
    2008-07-06 22:25:00 1
    2008-07-06 22:30:00 2
    2008-07-06 22:35:00 1
    2008-07-06 22:40:00 2
    2008-07-06 22:45:00 1
    2008-07-06 22:50:00 2
    2008-07-06 22:55:00 1
    2008-07-06 23:00:00 2
    2008-07-06 23:05:00 1
    2008-07-06 23:10:00 2
    2008-07-06 23:15:00 1
    2008-07-06 23:20:00 2
    2008-07-06 23:25:00 1
    2008-07-06 23:30:00 2
    2008-07-06 23:35:00 3

    As you can see the pool is recycled quite often, event 4-5 times every 5 minutes, exactly the default limit configured for pool heal checking in IIS… Moreover we don’t have to forget about the second error message:

        "select quantize(TimeGenerated, 300) as TimeGen, count(*) as ShutDownLimit into ShutDownLimit.log 
        from system.evt where Message like '%\'MyAppPool%' and Message like '%exceeded time limits during shut down%' 
        group by TimeGen order by TimeGen" -i:evt -o:nat -rtp:-1

    And the output is:

    TimeGen             ShutDownLimit 
    ------------------- -----
    2008-07-09 09:05:00 3
    2008-07-09 10:00:00 5
    2008-07-11 14:40:00 3
    2008-07-15 10:50:00 5
    2008-07-15 11:30:00 3
    2008-07-15 13:00:00 3
    2008-07-15 14:45:00 3
    2008-07-15 14:50:00 4
    2008-07-15 14:55:00 3
    2008-07-15 16:15:00 3
    2008-07-15 16:20:00 4
    2008-07-15 16:40:00 3
    2008-07-15 16:45:00 3
    2008-07-15 17:00:00 3
    2008-07-15 17:25:00 3
    2008-07-15 17:30:00 3
    2008-07-15 17:40:00 3
    2008-07-15 17:45:00 3
    2008-07-15 17:55:00 3
    2008-07-15 18:00:00 4

    Sometimes we’re clearly exceeding the configured pool health checking so really no surprise IIS disables it.

    But why the pool is recycled so often? Well, we found a mistake in customer’s configuration: the application pool was configured to be recycled when the virtual memory reached 50 Mb, definitely a value too low. Increasing it to resolved also this problem: no more pool recycles and as obvious side effect the users noticed an increased stability and performance when working with the application.



    Quote of the day:
    A wise man gets more use from his enemies than a fool from his friends. - Baltasar Gracian
  • Never doubt thy debugger

    Time-taken and LogParser for web site statistics


    The time-taken field is quite handy if your web application or IIS web server is performing poorly (or slower that you expect) or even for simple statistics and monitoring purposes, if you want to keep under how long it takes every web request to be processed and the page sent back to the client.

    In earlier versions of IIS the time-taken field was not selected by default so I often had to request customers to first enable it, wait some more time for the problem to reoccur (to give IIS the change to create a significant amount of logs with the new value) before being able to start troubleshooting. Well, at least before being able to extract some statistics involving the duration of web requests.

    Time Taken logging property in IIS 6

    Luckily in IIS 7 time-taken is now selected by default so we already it logged and ready to use.

    Taken from the IIS online docs, here is the meaning each field in IISW3C logs:

    Date (date) Logs the date on which the request occurred. Selected by default
    Time (time) Logs the time, in Coordinated Universal Time (UTC), at which the request occurred. Selected by default

    Client IP Address (c-ip)

    Logs the IP address of the client that made the request. Selected by default.
    User Name (cs-username) Logs the name of the authenticated user who accessed the server. Anonymous users are indicated by a hyphen. Selected by default.
    Service Name (s-sitename) Logs the Internet service name and instance number that was running on the client at the time that the incident was logged.
    Server Name (s-computername) Logs the name of the server on which the log file entry was generated.
    Server IP Address (s-ip) Logs the IP address of the server on which the log file entry was generated. Selected by default.
    Server Port (s-port) Logs the server port number that is configured for the service. Selected by default.
    Method (cs-method) Logs the HTTP method, such as GET, that is used in the request. Selected by default.
    URI Stem (cs-uri-stem) Logs the Uniform Resource Identifier (URI) that is the target of the action, such as Default.htm. Selected by default.
    URI Query (cs-uri-query) Logs the query, if any, that the client was trying to perform. A URI query is necessary only for dynamic pages. Selected by default.
    Protocol Status (sc-status) Logs the HTTP status code. Selected by default.
    Protocol Substatus (sc-substatus) Logs the HTTP substatus code. Selected by default.
    Win32 Status (sc-win32-status) Logs the Windows status code. Selected by default.
    Bytes Sent (sc-bytes) Logs the number of bytes that the server sent.
    Bytes Received (cs-bytes) Logs the number of bytes that the server received.
    Time Taken (time-taken) Logs the length of time that the action took in milliseconds. Selected by default.
    Protocol Version (cs-version) Logs the protocol version, HTTP or FTP, that the client used.
    Host (cs-host) Logs the host header name, if any.

    The host name that you configure for your Web site might appear differently in the log files because HTTP.sys logs host names by using the punycode-encoded format.

    User Agent (cs(User-Agent)) Logs the browser from which the request came. Selected by default.
    Cookie (cs(Cookie)) Logs the content of the cookie sent or received, if any.
    Referer (cs(Referer) ) Logs the site that the user last visited. This site provided a link to the current site.

    What does time-taken actually measures? It measures the length of time that it takes for a request to be processed. The client-request time stamp is initialized when HTTP.sys receives the first byte of the request. HTTP.sys is the kernel-mode component that is responsible for HTTP logging for IIS activity. The client-request time stamp is initialized before HTTP.sys begins parsing the request. The client-request time stamp is stopped when the last IIS response send completion occurs. Beginning in IIS 6.0, the time-taken field typically includes network time. Before HTTP.sys logs the value in the time-taken field, HTTP.sys usually waits for the client to acknowledge the last response packet send operation or HTTP.sys waits for the client to reset the underlying TCP connection. Therefore, when a large response or large responses are sent to a client over a slow network connection, the value of the time-taken field may be more than expected.

    A few samples

    If you are either troubleshooting or just monitoring your site/IIS, LogParser can be extremely useful to elaborate statistics and charts out of your IIS logs, find average and peak execution time for your pages and web services etc…

    For example if you want to get the average execution time for your .aspx (or .asmx) pages/services you can run something like:

        "SELECT AVG(time-taken) As AverageTimeTaken, MAX(time-taken) As MaxTimeTaken, COUNT(*) As Hits, TO_LOWERCASE(cs-uri-stem) As Uri
        INTO average_uri_execution_time.log FROM ex*.log 
        GROUP BY TO_LOWERCASE(cs-uri-stem) 
        ORDER BY AverageTimeTaken DESC"  
        -i:IISW3C -o:TSV

    Note: the command must be placed on a single line, I split it here just to be easier to read

    Using the IISW3C input format, if you have your logs organized in subfolders you can parse them all at once appending “-recurse:-1” to the command above.

    Here you can get the top n slowest pages to complete:

        "SELECT TOP 100 time-taken, cs-uri-stem 
        INTO top_time_taken.log FROM ex*.log 
        WHERE extract_extension(to_lowercase(cs-uri-stem)) = 'aspx' 
        ORDER BY time-taken DESC" 
        -i:iisw3c -o:tsv

    If you are interested only on a specific timeframe (say you had a load pick and want to know more) you can take the cue from here:

        "SELECT AVG(time-taken) As AverageTimeTaken, MAX(time-taken) As MaxTimeTaken, COUNT(*) As Hits, TO_LOWERCASE(cs-uri-stem) As Uri 
        INTO date_range.txt FROM *.log 
        WHERE TO_TIMESTAMP(date,time) BETWEEN TIMESTAMP('2008/05/27 10:11:00', 'yyyy/MM/dd hh:mm:ss') AND TIMESTAMP('2008/05/27 10:13:00', 'yyyy/MM/dd hh:mm:ss') 
        GROUP BY TO_LOWERCASE(cs-uri-stem)" 
        -i:IISW3C -RECURSE:-1 -o:NAT -RTP:-1

    If you want an overview of resource types served by your IIS (with resource I mean every file extension IIS logs) you can use something like this:

        "select to_lowercase(extract_extension(cs-uri-stem))as ResourceType, MIN(time-taken) as MinTimeTaken, MAX(time-taken) as MaxTimeTaken, AVG(time-taken) as AvgTimeTaken, COUNT(*) as Hits 
        INTO timers.txt FROM time-taken.log GROUP BY ResourceType" 
        -i:IISW3C -o:NAT -RTP:-1

    A few words about the load you add on the serve enabling additional fields for IIS logs: the default files are essentially the same in IIS 6 and 7 except for time-taken which is now enabled by default on IIS 7. This suggests the load you’ll add to the server is negligible, but as always is really depends on your server and application; to stay on the safe side you can add fixed-size fields, for example handle with case the cookie field, enable it if you need to troubleshoot specific problems involving cookies (session mixed-up etc…), otherwise just leave it off.



    Quote of the day:
    Art is science made clear. - Jean Cocteau
  • Never doubt thy debugger

    Visual Studio 2008 crashes in “split view”


    “Split view” is one of the new features in Visual Studio 2008 web designer: this is the possibility to have Design View and Source View of your page at the same time (see What's New in ASP.NET and Web Development, “Visual Web Developer Enhancements” paragraph). A few days ago a customer called in to report a problem with his Visual Studio and split view: when trying to view a file (even a new one) in split view or design view the IDE either crashed or frozen.

    As you can imagine we took a crash dump with adplus, but a first look at the threads, stack, exceptions etc… (the usual stuff) did not show anything interesting. Then I thought to some external process (like an antivirus, third parties add-ons etc…) that might meddle in and had a look at the list of modules loaded within the process with lmf (list loaded modules with full path). There where only a couple of odd dlls loaded, from Office 2003:

    33f20000 34118000   FPCUTL     (deferred)             
        Image path: C:\Program Files\Microsoft Office\OFFICE11\FPCUTL.DLL 
        Image name: FPCUTL.DLL 
        Timestamp:        Wed Jun 06 19:44:55 2007 (4666F297) 
        CheckSum:         001EC356 
        ImageSize:        001F8000 
        File version:     11.0.8170.0 
        Product version:  11.0.8170.0 
        File flags:       0 (Mask 3F) 
        File OS:          40004 NT Win32 
        File type:        2.0 Dll 
        File date:        00000000.00000000 
        Translations:     0409.04e4 
        CompanyName:      Microsoft Corporation 
        ProductName:      Microsoft Office 2003 
        InternalName:     FP40CUTL 
        OriginalFilename: FP40CUTL.DLL 
        ProductVersion:   11.0.8170 
        FileVersion:      11.0.8170 
        FileDescription:  Microsoft Office FrontPage Client Utility Library 
        LegalCopyright:   Copyright © 1995-2003 Microsoft Corporation.  All rights reserved. 
    37050000 37157000   OMFC       (deferred)             
        Image path: C:\Program Files\Microsoft Office\OFFICE11\OMFC.DLL 
        Image name: OMFC.DLL 
        Timestamp:        Fri Apr 13 21:43:21 2007 (461FDD59) 
        CheckSum:         0010A869 
        ImageSize:        00107000 
        File version:     11.0.8164.0 
        Product version:  11.0.8164.0 
        File flags:       0 (Mask 3F) 
        File OS:          40004 NT Win32 
        File type:        2.0 Dll 
        File date:        00000000.00000000 
        Translations:     0000.04e4 
        CompanyName:      Microsoft Corporation 
        ProductName:      Microsoft Office 2003 
        InternalName:     OMFC.DLL 
        OriginalFilename: OMFC.DLL 
        ProductVersion:   11.0.8164 
        FileVersion:      11.0.8164 
        FileDescription:  Microsoft Office MFC 
        LegalCopyright:   Copyright © 1993-2003 Microsoft Corporation.  All rights reserved. 

    That rang a bell, especially about fpcutl.dll… This dll comes with both Visual Studio and Office 2003 but the two versions are incompatible: Visual Studio in this case was loading the wrong one. Why?

    Well, it happens that Visual Studio looks at the system PATH to load this (and other) component, while Office doesn’t… Here is the PATH value from the dump (use !procinfo and then check the path value):

    C:\Program Files\Microsoft SQL Server\90\Tools\binn\;
    C:\Program Files\Microsoft SQL Server\90\DTS\Binn\;
    C:\Program Files\Microsoft SQL Server\90\Tools\Binn\VSShell\Common7\IDE\;
    C:\Program Files\Microsoft Visual Studio 8\Common7\IDE\PrivateAssemblies\;

    The Office 11 entry is the last one… but there is something wrong anyway, who spots it? smile_regular

    We have an entry for Visual Studio 8 (Visual Studio 2005), but where is the entry for Visual Studio 9 (VS 2008)? smile_nerd

    For some reason the PATH value has not been updated (or maybe it has modified after the setup) and we don’t know exactly why; the point is that we need to fix it. A quick search on the machine found three copies of fpcutl.dll:

    fpcutil paths

    System Path

    We added the missing folders at the beginning of the PATH string which now reads as:

    C:\Program Files\Microsoft Web Designer Tools\VWD;
    C:\Program Files\Microsoft Visual Studio 9.0\Common7\IDE;
    C:\Program Files\Microsoft Visual Studio 9.0\VC\BIN;
    C:\Program Files\Microsoft Visual Studio 9.0\Common7\Tools;
    C:\Program Files\Microsoft Visual Studio 9.0\VC\VCPackages;
    C:\Program Files\Microsoft SQL Server\90\Tools\binn\;
    C:\Program Files\Microsoft SQL Server\90\DTS\Binn\;
    C:\Program Files\Microsoft SQL Server\90\Tools\Binn\VSShell\Common7\IDE\;
    C:\Program Files\Microsoft Visual Studio 8\Common7\IDE\PrivateAssemblies\;

    The most important one on this case is “C:\Program Files\Microsoft Web Designer Tools\VWD”: Visual Studio will look into this folder to load the right version of fpcutl.dll (PATH is a first in – first serve list of folders, the first match will be the one we use) and Office continues working happily anyway. smile_nerd



    Quote of the day:
    Any sufficiently advanced technology is indistinguishable from magic. - Arthur C. Clarke
  • Never doubt thy debugger

    An interesting fusion cache lock: that’s what the GAC is meant for


    This has been an interesting case where we had an ASP.NET 2.0 application which under load was completely blocked after a few minutes. Since we were talking about a hang/deadlock (as reported from the customer), the !critlist command (you can find the command within the SieExtPub.dll extension) is a good start:

    0:021> !critlist
    CritSec at 7a393800.  Owned by thread 21.  
      Waiting Threads: 6 7 8 9 10 11 12 13 42 72 104 107 108 109 110 111 112 113 114 115 116 
    CritSec at    e194c.  Owned by thread 32.  
      Waiting Threads: 21

    Thread 32 holds a critical section and thread 21 is waiting on it, but thread 21 is also owning another critical section, and about 20 other threads are waiting there… so apparently this is not a real deadlock (21 is waiting on 32 but 32 is not waiting on 21); but what is thread 32 doing?

    0:032> kpL2000
    ChildEBP RetAddr  
    0327fd30 7c822124 ntdll!KiFastSystemCallRet
    0327fd34 77e6baa8 ntdll!NtWaitForSingleObject+0xc
    0327fda4 79e718fd kernel32!WaitForSingleObjectEx+0xac
    0327fde8 79e718c6 mscorwks!PEImage::LoadImage+0x199
    0327fe38 79e7187c mscorwks!CLREvent::WaitEx+0x117
    0327fe48 7a0e288e mscorwks!CLREvent::Wait+0x17
    0327fe9c 7a086e76 mscorwks!Thread::SysSuspendForGC+0x52a
    0327ff88 7a0d867b mscorwks!SVR::GCHeap::SuspendEE+0x16c
    0327ffa8 7a0d8987 mscorwks!SVR::gc_heap::gc_thread_function+0x3b
    0327ffb8 77e66063 mscorwks!SVR::gc_heap::gc_thread_stub+0x9b
    0327ffec 00000000 kernel32!BaseThreadStart+0x34

    It is waiting to start GC but it cannot, because thread 95 has PreEmptive disabled:

    0:032> !threads
    ThreadCount: 69
    UnstartedThread: 0
    BackgroundThread: 69
    PendingThread: 0
    DeadThread: 0
    Hosted Runtime: yes
                                          PreEmptive   GC Alloc           Lock
           ID OSID ThreadOBJ    State     GC       Context       Domain   Count APT Exception
      23    1 1288 0012efb8   1808220 Enabled  1048ebfc:1048efe8 0011f578     0 MTA (Threadpool Worker)
      95   35 1114 05424c30   180b222 Disabled 16577334:16579008 00162f38     1 MTA (Threadpool Worker)
      96   36 1b54 05407bf8   180b220 Enabled  145124ec:14512fe8 0011f578     0 MTA (Threadpool Worker)

    As you might know, if a thread has PreEmptive GC disabled is because it does not want to be interrupted by the GC; loading a file is a good reason to disable PreEmptive and that’s exactly the case:

    0:095> !clrstack
    OS Thread Id: 0x1114 (95)
    ESP       EIP     
    05e5ecd0 7c82ed54 [HelperMethodFrame_PROTECTOBJ: 05e5ecd0] System.Reflection.Assembly.nLoadFile(System.String, System.Security.Policy.Evidence)
    05e5ef88 793f583f System.Reflection.Assembly.LoadFile(System.String)
    05e5ef9c 04674f82 MyApp.Core.Lib.Managers.LangMan.ResourceFile(System.String, System.String, MyApp.Core.Lib.Managers.ResourceMode)
    05e5efd8 04674f0c MyApp.Core.Lib.Managers.LangMan.ResourceFile(System.String, System.String)
    05e5efe0 04674ee5 MyApp.FPCore.Pages.Page.ResourceFile(System.String, System.String)
    05e5efec 04674d69 FP_Login.formslogin.Page_Load(System.Object, System.EventArgs)
    05e5f028 031bb2b5 [MulticastFrame: 05e5f028] System.EventHandler.Invoke(System.Object, System.EventArgs)

    !showstringw from SieExtPub on the first parameter passed to mscorwks!ExplicitBind we can get the name of the file we’re trying to load:

    0:095> kb
    ChildEBP RetAddr  Args to Child              
    05e5e860 7c822124 77e6baa8 00001348 00000000 ntdll!KiFastSystemCallRet
    05e5e864 77e6baa8 00001348 00000000 00000000 ntdll!NtWaitForSingleObject+0xc
    05e5e8d4 79e718fd 00001348 ffffffff 00000000 kernel32!WaitForSingleObjectEx+0xac
    05e5e918 79e718c6 00001348 ffffffff 00000000 mscorwks!PEImage::LoadImage+0x199
    05e5e968 79e7187c ffffffff 00000000 00000000 mscorwks!CLREvent::WaitEx+0x117
    05e5e978 79f90f81 ffffffff 00000000 00000000 mscorwks!CLREvent::Wait+0x17
    05e5e988 79f917ba 7a390970 03c55358 ffffffff mscorwks!CExecutionEngine::WaitForEvent+0x37
    05e5e99c 7a15cfce 03c55358 ffffffff 00000000 mscorwks!ClrWaitEvent+0x17
    05e5e9ac 7a153339 05e5ed7c 00000000 00000000 mscorwks!CSimpleFusionBindSink::Wait+0xe
    05e5e9c8 7a1545d2 03cecd70 0016e3d0 00000200 mscorwks!BindHelper+0x7d
    05e5ec64 7a2d0c7c 05e5ed7c 0016e3d0 00000000 mscorwks!ExplicitBind+0x1a6
    05e5ef80 793f583f 00000000 16577088 1657703c mscorwks!AssemblyNative::LoadFile+0x1de
    0:095> !showstringw 05e5ed7c 

    Looking around in the stack it’s interesting to note that also thread 81 was loading the same file from from ASP.NET temporary folder:

    0:081> kb
    ChildEBP RetAddr  Args to Child              
    0573cf58 7c822124 77e6baa8 000004f4 00000000 ntdll!KiFastSystemCallRet
    0573cf5c 77e6baa8 000004f4 00000000 00000000 ntdll!NtWaitForSingleObject+0xc
    0573cfcc 79e718fd 000004f4 ffffffff 00000000 kernel32!WaitForSingleObjectEx+0xac
    0573d010 79e718c6 000004f4 ffffffff 00000000 mscorwks!PEImage::LoadImage+0x199
    0573d060 79e7187c ffffffff 00000000 00000000 mscorwks!CLREvent::WaitEx+0x117
    0573d070 7a0851cb ffffffff 00000000 00000000 mscorwks!CLREvent::Wait+0x17
    0573d080 79f40e96 00000000 5ed28252 79eaebea mscorwks!SVR::GCHeap::WaitUntilGCComplete+0x32
    0573d0bc 79e797c6 5ed2822a 79eaebea 053ee59a mscorwks!Thread::RareDisablePreemptiveGC+0x1a1
    0573d0e8 79ea91b2 7a3879e0 5ed283e6 79eaebea mscorwks!CrstBase::AcquirePreempLock+0x2e
    0573d124 79eb4132 0573d154 00000000 0573d1b4 mscorwks!PEImage::OpenImage+0x92
    0573d1c0 79eb3d78 053ee598 0573d1f8 0573d1f4 mscorwks!RuntimeOpenImageInternal+0xcd
    0573d208 79eb3cee 053ee598 79eaebb0 00000000 mscorwks!GetAssemblyMDInternalImportEx+0x9d
    0573d21c 79ecf4b9 053ee598 00000000 0573d238 mscorwks!CreateMetaDataImport+0x16
    0573d23c 79ecf462 053ee598 5ed280a6 00000000 mscorwks!CAssemblyManifestImport::Init+0x35
    0573d264 79f2e637 053ee598 0573d27c 000f2a20 mscorwks!CreateAssemblyManifestImport+0x53
    0573d280 7a161a1d 053ee598 03d09c00 0573d9a8 mscorwks!CreateAssemblyFromManifestFile+0x48
    0573d930 7a161e12 0573dcc4 03d09c00 0573d9a8 mscorwks!CAsmDownloadMgr::CreateAssembly+0x8e2
    0573d96c 7a162550 0573dcc4 03d09c00 0573d9a8 mscorwks!CAsmDownloadMgr::DoSetupPushToCache+0x50
    0573dbf4 79f90e65 000f2a20 03d09c00 0573dcc4 mscorwks!CAsmDownloadMgr::DoSetup+0x26a
    0573dc40 79f8f2cc 0573dcc4 00000000 5ed28eb6 mscorwks!CAssemblyDownload::DoSetup+0x7b
    0:081> !showstringw 053ee598 
    C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\_mem_bin\bf10f877\41cee659\assembly\dl3\9cf10f5d\0090cb26_8229c601\DescMembers.DLL
    0:081> !showstringw 0573dcc4 

    Another set of dumps helped to further clarify the situation: thread 94 holds the lock and is trying to load a previously compiled assembly from disk. It is trying to load this in relation to page D:\MyApp\App\Secure\Root\default.aspx and this this is leading to the attempt to load the assembly for global.asax: C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\App_global.asax.ylfot7zw.dll. As part of this assembly load, thread 94 is blocked waiting to acquire a critical section owned by thread 123:

    0:094> !syncblk Index SyncBlock MonitorHeld Recursion Owning Thread Info SyncBlock Owner 140 03f05a84 13 1 03ebd208 714 94 1225a688 System.Web.Compilation.BuildManager ----------------------------- Total 219 CCW 11 RCW 0 ComClassFactory 0 Free 0 0:094> !dso OS Thread Id: 0x714 (94) ESP/REG Object Name 04f7e9e0 790d6654 System.String 04f7eb70 1c571e2c System.Byte[] 04f7ecb8 1c571df0 System.Reflection.AssemblyName 04f7eccc 1c571df0 System.Reflection.AssemblyName 04f7edac 1c571df0 System.Reflection.AssemblyName 04f7edc4 1c571df0 System.Reflection.AssemblyName 04f7edf8 1c571ad0 System.String C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\App_global.asax.ylfot7zw.dll.delete 04f7ee00 1c47a640 System.Object[] (System.Reflection.AssemblyName[]) 04f7ee04 1c5715e8 System.String C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\App_global.asax.ylfot7zw.dll 04f7ee24 1c4775e0 System.Reflection.Assembly 04f7ee28 1c468e58 System.String App_Web_gf9hosur 04f7ee2c 1c428b70 System.Web.Compilation.PreservationFileReader 04f7ee30 121a2004 System.Globalization.CultureInfo 04f7ee34 1c468dbc System.String 110000 04f7ee44 1c1f7e58 System.Web.VirtualPath 04f7ee48 1c428b70 System.Web.Compilation.PreservationFileReader 04f7ee4c 1c4693bc System.Web.Compilation.BuildResultCompiledTemplateType 04f7ee58 1c1f7e58 System.Web.VirtualPath 04f7ee5c 1c428b70 System.Web.Compilation.PreservationFileReader 04f7ee60 1c4693bc System.Web.Compilation.BuildResultCompiledTemplateType 04f7ee7c 1c468cbc System.String fffffff8697ef330 04f7ee84 1c428a68 System.String C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\default.aspx.8cd6fbf4.compiled 04f7ee88 1c428a68 System.String C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\default.aspx.8cd6fbf4.compiled 04f7ee8c 1c1f7e58 System.Web.VirtualPath 04f7ee9c 1c428a68 System.String C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\default.aspx.8cd6fbf4.compiled 04f7eea4 1c428b70 System.Web.Compilation.PreservationFileReader 04f7eea8 1a200b5c System.String .compiled 04f7eeac 1c428970 System.String C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\default.aspx.8cd6fbf4 04f7eeb8 1a200b5c System.String .compiled 04f7eebc 1c428a68 System.String C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\default.aspx.8cd6fbf4.compiled 04f7eec0 1c428b70 System.Web.Compilation.PreservationFileReader 04f7eec4 1225b41c System.Web.Compilation.StandardDiskBuildResultCache 04f7eec8 1c1f7e58 System.Web.VirtualPath 04f7eed8 1c428a68 System.String C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\default.aspx.8cd6fbf4.compiled 04f7eee8 1c1f7e58 System.Web.VirtualPath 04f7eef8 1c1f7e58 System.Web.VirtualPath 04f7ef00 1225a688 System.Web.Compilation.BuildManager 04f7ef08 1c427124 System.String default.aspx.8cd6fbf4 04f7ef0c 1225a7c4 System.Web.Util.SimpleRecyclingCache 04f7ef20 1c427100 System.String 8cd6fbf4 04f7ef2c 1c1f7e58 System.Web.VirtualPath 04f7ef30 1225a688 System.Web.Compilation.BuildManager 04f7ef44 1c1f7e58 System.Web.VirtualPath 04f7ef50 1225a688 System.Web.Compilation.BuildManager 04f7ef54 1c1f7e58 System.Web.VirtualPath 04f7ef70 1c418fb8 System.Web.ApplicationImpersonationContext 04f7ef74 1c418fb8 System.Web.ApplicationImpersonationContext 04f7ef78 1c418fb8 System.Web.ApplicationImpersonationContext 04f7ef88 1c1f7e58 System.Web.VirtualPath 04f7ef9c 1c418fb8 System.Web.ApplicationImpersonationContext 04f7efa8 1c418fb8 System.Web.ApplicationImpersonationContext 04f7efac 1c1f7e58 System.Web.VirtualPath 04f7efc4 1c418fb8 System.Web.ApplicationImpersonationContext 04f7efcc 1c418f88 System.Object[] (System.Type[]) 04f7efd0 1c418fac System.Web.UI.PageHandlerFactory 04f7efe4 1c1f7e58 System.Web.VirtualPath 04f7efe8 1c1f7c2c System.Web.HttpContext 04f7f000 1c1f7c2c System.Web.HttpContext 04f7f004 1444b800 System.RuntimeType 04f7f024 1239e718 System.String System.Web.UI.PageHandlerFactory 04f7f02c 1c393abc System.Collections.Hashtable 04f7f034 1c1f73a4 System.String D:\MyApp\App\Secure\Root\default.aspx 04f7f038 1c1f7c2c System.Web.HttpContext 04f7f03c 1c1f7e58 System.Web.VirtualPath 04f7f050 1c1f7c2c System.Web.HttpContext 04f7f054 1c418fac System.Web.UI.PageHandlerFactory 04f7f058 1c1f7c2c System.Web.HttpContext 04f7f060 1c1f73a4 System.String D:\MyApp\App\Secure\Root\default.aspx 04f7f064 1c1f7e58 System.Web.VirtualPath 04f7f068 1c1f7338 System.String POST 04f7f06c 1c418fac System.Web.UI.PageHandlerFactory 04f7f070 1c4187b0 System.Web.ApplicationImpersonationContext 04f7f074 1c1f87b8 ASP.global_asax 04f7f07c 1c1f73a4 System.String D:\MyApp\App\Secure\Root\default.aspx 04f7f080 1c1f7e58 System.Web.VirtualPath 04f7f084 1c1f7338 System.String POST 04f7f088 1c4187b0 System.Web.ApplicationImpersonationContext 04f7f090 1c1f7c2c System.Web.HttpContext 04f7f0a4 1c1f87b8 ASP.global_asax 04f7f0a8 1c1f87b8 ASP.global_asax 04f7f0ac 1c1f7cd4 System.Web.HttpRequest 04f7f0b0 1c1f7c2c System.Web.HttpContext 04f7f0c0 1c1f73a4 System.String D:\MyApp\App\Secure\Root\default.aspx 04f7f0c4 1c1f7e58 System.Web.VirtualPath 04f7f0c8 1c1f7338 System.String POST 04f7f0cc 1c1f7c2c System.Web.HttpContext 04f7f0d0 1c1f87b8 ASP.global_asax 04f7f0e8 1c411f58 System.Web.HttpApplication+MapHandlerExecutionStep 04f7f0ec 1c1f87b8 ASP.global_asax 04f7f0f0 1c1f7c2c System.Web.HttpContext 04f7f104 1c1f7c2c System.Web.HttpContext 04f7f108 1c1f87b8 ASP.global_asax 04f7f11c 1c1f87b8 ASP.global_asax 04f7f120 1c412258 System.Web.HttpApplication+ThreadContext 04f7f124 1c1f7c2c System.Web.HttpContext 04f7f128 1c1f87b8 ASP.global_asax 04f7f12c 1c412238 System.Web.AspNetSynchronizationContext 04f7f134 1c411cdc System.Web.HttpApplication+ApplicationStepManager 04f7f158 1c1f7c2c System.Web.HttpContext 04f7f15c 1c412218 System.Web.HttpAsyncResult 04f7f160 1c1f87b8 ASP.global_asax 04f7f164 121c489c System.AsyncCallback 04f7f170 1c1f4ff4 System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6 04f7f174 1c1f87b8 ASP.global_asax 04f7f178 1c1f7e6c System.Web.HttpWriter 04f7f180 1c1f7c2c System.Web.HttpContext 04f7f184 121c489c System.AsyncCallback 04f7f188 1c1f7c2c System.Web.HttpContext 04f7f18c 1c1f4ff4 System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6 04f7f190 121c4138 System.Web.HttpRuntime 04f7f1a8 1c1f72dc System.String D:\MyApp\App\Secure\ 04f7f1ac 1c1f4ff4 System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6 04f7f1bc 121a1038 System.String D:\MyApp\App\Secure\ 04f7f1c0 1c1f4ff4 System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6 04f7f1f8 122f2880 System.Web.Hosting.ISAPIRuntime 0:094> !do 1c1f4ff4 Name: System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6 MethodTable: 663a6f84 EEClass: 663a6f04 Size: 256(0x100) bytes (C:\WINDOWS\assembly\GAC_32\System.Web\\System.Web.dll) Fields: MT Field Offset Type VT Attr Value Name 79105fac 400133a 4 System.DateTime 1 instance 1c1f4ff8 _startTime 7910ead0 400133b c System.Guid 1 instance 1c1f5000 _traceId 7912ab5c 400133c 194 System.String[][] 0 shared static s_HTTPStatusDescriptions >> Domain:Value 001171e8:NotInit 00159f68:1a227064 03ea3b78:122f6384 << 79124304 400133d 198 System.Object[] 0 shared static s_serverVarFromRequestHeaderNames >> Domain:Value 001171e8:NotInit 00159f68:1a2271ac 03ea3b78:122f64cc << 79124304 400133e 19c System.Object[] 0 shared static s_requestHeaderNames >> Domain:Value 001171e8:NotInit 00159f68:1a22725c 03ea3b78:122f657c << 79124304 400133f 1a0 System.Object[] 0 shared static s_responseHeaderNames >> Domain:Value 001171e8:NotInit 00159f68:1a22730c 03ea3b78:122f662c << 790feb48 4001340 1a4 ...ections.Hashtable 0 shared static s_requestHeadersLoookupTable >> Domain:Value 001171e8:NotInit 00159f68:1a227394 03ea3b78:122f66b4 << 790feb48 4001341 1a8 ...ections.Hashtable 0 shared static s_responseHeadersLoookupTable >> Domain:Value 001171e8:NotInit 00159f68:1a22745c 03ea3b78:122f677c << 790fe238 4001b3a 70 System.IntPtr 1 instance 3066598 _ecb 790fe238 4001b3b 74 System.IntPtr 1 instance 2d0 _token 7910ead0 4001b3c a8 System.Guid 1 instance 1c1f509c _traceId 790fa4b0 4001b3d 1c System.String 0 instance 1c1f7338 _method 790fa4b0 4001b3e 20 System.String 0 instance 1c1f7354 _path 790fa4b0 4001b3f 24 System.String 0 instance 1c1f7354 _filePath 790fa4b0 4001b40 28 System.String 0 instance 790d6654 _pathInfo 790fa4b0 4001b41 2c System.String 0 instance 1c1f73a4 _pathTranslated 790fa4b0 4001b42 30 System.String 0 instance 141a4b28 _appPath 790fa4b0 4001b43 34 System.String 0 instance 1c1f72dc _appPathTranslated [...] 0:094> !do 1c1f73a4 Name: System.String MethodTable: 790fa4b0 EEClass: 790fa410 Size: 150(0x96) bytes (C:\WINDOWS\assembly\GAC_32\mscorlib\\mscorlib.dll) String: D:\MyApp\App\Secure\Root\default.aspx Fields: MT Field Offset Type VT Attr Value Name 790fedf4 4000096 4 System.Int32 1 instance 67 m_arrayLength 790fedf4 4000097 8 System.Int32 1 instance 66 m_stringLength 790fbfcc 4000098 c System.Char 1 instance 44 m_firstChar 790fa4b0 4000099 10 System.String 0 shared static Empty >> Domain:Value 001171e8:790d6654 00159f68:790d6654 03ea3b78:790d6654 << 7912474c 400009a 14 System.Char[] 0 shared static WhitespaceChars >> Domain:Value 001171e8:141a0624 00159f68:141a77f8 03ea3b78:121a2374 << 0:094> !do 1c1f87b8 Name: ASP.global_asax MethodTable: 04c35adc EEClass: 04ff1dd8 Size: 140(0x8c) bytes (C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\App_global.asax.ylfot7zw.dll) Fields: MT Field Offset Type VT Attr Value Name 663a2450 4000dc8 4 ...pApplicationState 0 instance 122e14dc _state 663a2750 4000dc9 8 ...m.Web.HttpContext 0 instance 00000000 _initContext 663a359c 4000dca c ...b.HttpAsyncResult 0 instance 1c412218 _ar 663a3394 4000dcb 10 ...pModuleCollection 0 instance 1c3a648c _moduleCollection 79118214 4000dcc 14 ...ncipal.IPrincipal 0 instance 1c413d68 _savedPrincipal 79105020 4000dcd 78 System.Boolean 1 instance 1 _restorePrincipal 7a74da98 4000de8 18 ....EventHandlerList 0 instance 1c3f7408 _events 663a2aec 4000de9 1c ...ventHandlersTable 0 instance 1c401980 _asyncEvents [...]


    0:094> !critlist
    CritSec at 7a38ee88. Owned by thread 123.
    Waiting Threads: 26 66 94

    !dso output already shows objects and strings in the same order as they are stored on the stack and executed, so it’s quite easy to spot interesting file paths, URLs etc… and often that is enough to suggest which files are involved, which page or web service you’re accessing and so on; of course you can always use !do <address> to dig into the stack to be 100% sure you’re on the right path.

    Using the same commands above we found that thread 123 is trying to explicitly load "D:\MyApp\App\Secure\bin\secure.dll" whilst executing "ASP.logout_aspx"; after transitioning from managed to unmanaged code, this thread then gets blocked waiting for garbage collection to complete (because it needs to disable pre-emptive mode in order to acquire a particular lock).

    In summary, the hang was due to a complicated deadlock situation involving “D:\MyApp\App\Secure\Root\default.aspx” being compiled by ASP.NET 2.0 compilation engine. This leads to an attempt to load “C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\App_global.asax.ylfot7zw.dll” from disk (the default.aspx page has a dependency on the global.asax) which in turn leads to an attempt to acquire a critical section owned by thread 123. Thread 123 is processing the request for “logout.aspx” which involves the execution of “MyApp.Core.Library.Managers.LanguageManager.ResourceFile” which leads to an attempt to load “D:\MyApp\App\Secure\bin\secure.dll” from disk. However this load operation gets blocked waiting for garbage collection to complete as it must acquire a particular lock when GC is not running. The GC was triggered (incidentally) by thread 56, but the GC has not actually commenced yet. The primary GC thread (29) has asked all thread executing managed code to get to a 'safe' point where GC can begin, but that has not happened (and will never happen) because some of those threads are blocked doing assembly load operations and are waiting on critical section.


    Finally, the resolution: the customer confirmed the involved assemblies are “resource only” files which they access from everywhere in their quite complex application (basically those were localization assemblies they were explicitly loading when needed to localize their interface, error messages etc…); we strong-named those “sensitive” assemblies and installed them in the GAC: easy solution, when you know which is the problem…

    By the way, there are a couple hotfixes available for this situation (see http://support.microsoft.com/kb/936757/en-us and 946644, KB not yet available but you can ask for the fix to CSS), but did not help in this case.



    Quote of the day:
    Tact is the ability to describe others as they see themselves. - Abraham Lincoln
  • Never doubt thy debugger

    autoConfig=”false” effects


    My post on autoConfig triggered and interesting discussion with my colleague Cenk about which is the real effect of this setting; one of his customers asked what to do if they needed to adjust only one or two values and not all the ones discussed in Contention, poor performance, and deadlocks when you make Web service requests from ASP.NET applications, they wanted to be sure they could change only what they needed to fine tune and forget about the rest. The short answer is: you can change only what you need to and leave he rest as is, since we have default values hardcoded to improve performance so you’re safe (provided the values you’re changing make sense in the context, of course).

    If we have a look at System.Web.Configuration.ProcessModelSection with Windbg, we can see all the properties we have in <processModel> and _defaultValue corresponds to the defaults we have in machine.config.comments:

    0:000> !do 06904868 
    Name: System.Web.Configuration.ProcessModelSection
    MethodTable: 6638dcb0
    EEClass: 663a0604
    Size: 64(0x40) bytes
          MT    Field   Offset                 Type VT     Attr    Value Name
    7910be50  400014d       30       System.Boolean  1 instance        0 _bDataToWrite
    7910be50  400014e       31       System.Boolean  1 instance        0 _bModified
    7910be50  400014f       32       System.Boolean  1 instance        1 _bReadOnly
    7910be50  4000150       33       System.Boolean  1 instance        0 _bElementPresent
    7910be50  4000151       34       System.Boolean  1 instance        1 _bInited
    648eff64  4000152        4 ...ionLockCollection  0 instance 00000000 _lockedAttributesList
    648eff64  4000153        8 ...ionLockCollection  0 instance 00000000 _lockedAllExceptAttributesList
    648eff64  4000154        c ...ionLockCollection  0 instance 00000000 _lockedElementsList
    648eff64  4000155       10 ...ionLockCollection  0 instance 00000000 _lockedAllExceptElementsList
    648ea704  4000156       14 ...nfigurationValues  0 instance 069048a8 _values
    790fd8c4  4000157       18        System.String  0 instance 00000000 _elementTagName
    648f032c  4000158       1c ...lementInformation  0 instance 00000000 _evaluationElement
    648eee30  4000159       20 ...onElementProperty  0 instance 068fce20 _elementProperty
    648f53d0  400015a       2c         System.Int32  1 instance        0 _fItemLocked
    648f0a48  400015b       24 ...ontextInformation  0 instance 00000000 _evalContext
    648eda8c  400015c       28 ...nfigurationRecord  0 instance 028bea78 _configRecord
    7912d8f8  4000148        8      System.Object[]  0   shared   static s_lockAttributeNames
        >> Domain:Value  023d04b0:NotInit  024169d0:068fcd1c <<
    79101fe4  4000149        c ...ections.Hashtable  0   shared   static s_propertyBags
        >> Domain:Value  023d04b0:NotInit  024169d0:068fcd40 <<
    00000000  400014a       10                       0   shared   static s_perTypeValidators
        >> Domain:Value  023d04b0:NotInit  024169d0:00000000 <<
    790fd0f0  400014b       14        System.Object  0   shared   static s_nullPropertyValue
        >> Domain:Value  023d04b0:NotInit  024169d0:068fce08 <<
    648eee30  400014c       18 ...onElementProperty  0   shared   static s_ElementProperty
        >> Domain:Value  023d04b0:NotInit  024169d0:068fce20 <<
    648eee84  400015d       38 ...ectionInformation  0 instance 069049b8 _section
    648eee30  4001928      74c ...onElementProperty  0   shared   static s_elemProperty
        >> Domain:Value  023d04b0:NotInit  024169d0:06903da8 <<
    7911228c  4001929      748      System.TimeSpan  1   shared   static DefaultClientConnectedCheck
        >> Domain:Value  023d04b0:NotInit  024169d0:06903850 <<
    648ef118  400192a      750 ...ropertyCollection  0   shared   static _properties
        >> Domain:Value  023d04b0:NotInit  024169d0:06904714 <<
    648ef05c  400192b      754 ...igurationProperty  0   shared   static _propEnable
        >> Domain:Value  023d04b0:NotInit  024169d0:06903db4 <<
    648ef05c  400192c      758 ...igurationProperty  0   shared   static _propTimeout
        >> Domain:Value  023d04b0:NotInit  024169d0:06903df4 <<
    648ef05c  400192d      75c ...igurationProperty  0   shared   static _propIdleTimeout
        >> Domain:Value  023d04b0:NotInit  024169d0:06903e38 <<
    648ef05c  400192e      760 ...igurationProperty  0   shared   static _propShutdownTimeout
        >> Domain:Value  023d04b0:NotInit  024169d0:06903e7c <<
    648ef05c  400192f      764 ...igurationProperty  0   shared   static _propRequestLimit
        >> Domain:Value  023d04b0:NotInit  024169d0:06903ed8 <<
    648ef05c  4001930      768 ...igurationProperty  0   shared   static _propRequestQueueLimit
        >> Domain:Value  023d04b0:NotInit  024169d0:06903f74 <<
    648ef05c  4001931      76c ...igurationProperty  0   shared   static _propRestartQueueLimit
        >> Domain:Value  023d04b0:NotInit  024169d0:06903ff8 <<
    648ef05c  4001932      770 ...igurationProperty  0   shared   static _propMemoryLimit
        >> Domain:Value  023d04b0:NotInit  024169d0:06904064 <<
    648ef05c  4001933      774 ...igurationProperty  0   shared   static _propWebGarden
        >> Domain:Value  023d04b0:NotInit  024169d0:069040a4 <<
    648ef05c  4001934      778 ...igurationProperty  0   shared   static _propCpuMask
        >> Domain:Value  023d04b0:NotInit  024169d0:069040e4 <<
    648ef05c  4001935      77c ...igurationProperty  0   shared   static _propUserName
        >> Domain:Value  023d04b0:NotInit  024169d0:06904118 <<
    648ef05c  4001936      780 ...igurationProperty  0   shared   static _propPassword
        >> Domain:Value  023d04b0:NotInit  024169d0:0690414c <<
    648ef05c  4001937      784 ...igurationProperty  0   shared   static _propLogLevel
        >> Domain:Value  023d04b0:NotInit  024169d0:06904180 <<
    648ef05c  4001938      788 ...igurationProperty  0   shared   static _propClientConnectedCheck
        >> Domain:Value  023d04b0:NotInit  024169d0:069041d4 <<
    648ef05c  4001939      78c ...igurationProperty  0   shared   static _propComAuthenticationLevel
        >> Domain:Value  023d04b0:NotInit  024169d0:06904218 <<
    648ef05c  400193a      790 ...igurationProperty  0   shared   static _propComImpersonationLevel
        >> Domain:Value  023d04b0:NotInit  024169d0:0690426c <<
    648ef05c  400193b      794 ...igurationProperty  0   shared   static _propResponseDeadlockInterval
        >> Domain:Value  023d04b0:NotInit  024169d0:069042c0 <<
    648ef05c  400193c      798 ...igurationProperty  0   shared   static _propResponseRestartDeadlockInterval
        >> Domain:Value  023d04b0:NotInit  024169d0:06904304 <<
    648ef05c  400193d      79c ...igurationProperty  0   shared   static _propAutoConfig
        >> Domain:Value  023d04b0:NotInit  024169d0:06904348 <<
    648ef05c  400193e      7a0 ...igurationProperty  0   shared   static _propMaxWorkerThreads
        >> Domain:Value  023d04b0:NotInit  024169d0:069043ac <<
    648ef05c  400193f      7a4 ...igurationProperty  0   shared   static _propMaxIOThreads
        >> Domain:Value  023d04b0:NotInit  024169d0:0690443c <<
    648ef05c  4001940      7a8 ...igurationProperty  0   shared   static _propMinWorkerThreads
        >> Domain:Value  023d04b0:NotInit  024169d0:069044cc <<
    648ef05c  4001941      7ac ...igurationProperty  0   shared   static _propMinIOThreads
        >> Domain:Value  023d04b0:NotInit  024169d0:0690455c <<
    648ef05c  4001942      7b0 ...igurationProperty  0   shared   static _propServerErrorMessageFile
        >> Domain:Value  023d04b0:NotInit  024169d0:069045c8 <<
    648ef05c  4001943      7b4 ...igurationProperty  0   shared   static _propPingFrequency
        >> Domain:Value  023d04b0:NotInit  024169d0:069045fc <<
    648ef05c  4001944      7b8 ...igurationProperty  0   shared   static _propPingTimeout
        >> Domain:Value  023d04b0:NotInit  024169d0:06904640 <<
    648ef05c  4001945      7bc ...igurationProperty  0   shared   static _propMaxAppDomains
        >> Domain:Value  023d04b0:NotInit  024169d0:069046a8 <<
    79102290  4001946      938         System.Int32  1   shared   static cpuCount
        >> Domain:Value  023d04b0:NotInit  024169d0:2 <<
    0:000> !do 0690443c 
    Name: System.Configuration.ConfigurationProperty
    MethodTable: 648ef05c
    EEClass: 648eefec
    Size: 52(0x34) bytes
          MT    Field   Offset                 Type VT     Attr    Value Name
    790fd8c4  4000277        4        System.String  0 instance 06903c08 _name
    790fd8c4  4000278        8        System.String  0 instance 06903c08 _providedName
    790fd8c4  4000279        c        System.String  0 instance 00000000 _description
    79106894  400027a       10          System.Type  0 instance 068b7554 _type
    790fd0f0  400027b       14        System.Object  0 instance 06904430 _defaultValue
    7a7588dc  400027c       18 ...del.TypeConverter  0 instance 00000000 _converter
    648ef1a0  400027d       2c         System.Int32  1 instance        0 _options
    648f022c  400027e       1c ...tionValidatorBase  0 instance 06904418 _validator
    790fd8c4  400027f       20        System.String  0 instance 00000000 _addElementName
    790fd8c4  4000280       24        System.String  0 instance 00000000 _removeElementName
    790fd8c4  4000281       28        System.String  0 instance 00000000 _clearElementName
    648f022c  4000274       68 ...tionValidatorBase  0   shared   static NonEmptyStringValidator
        >> Domain:Value  023d04b0:NotInit  024169d0:068fcc00 <<
    648f022c  4000275       6c ...tionValidatorBase  0   shared   static DefaultValidatorInstance
        >> Domain:Value  023d04b0:NotInit  024169d0:068fcc14 <<
    790fd8c4  4000276       70        System.String  0   shared   static DefaultCollectionPropertyName
        >> Domain:Value  023d04b0:NotInit  024169d0:068fcbec <<
    0:000> !do 06904430 
    Name: System.Int32
    MethodTable: 79102290
    EEClass: 79102218
    Size: 12(0xc) bytes
          MT    Field   Offset                 Type VT     Attr    Value Name
    79102290  40003e9        4         System.Int32  1 instance       20 m_value
        enable = "true" [true|false]
        timeout = "10675199.02:48:05.4775807" [HH:MM:SS]
        idleTimeout = "10675199.02:48:05.4775807" [HH:MM:SS]
        shutdownTimeout = "00:00:05" [HH:MM:SS]
        requestLimit = "2147483647" [number]
        requestQueueLimit = "5000" [number]
        restartQueueLimit = "10" [number]
        memoryLimit = "60" [number]
        webGarden = "false" [true|false]
        cpuMask = "0xffffffff" [number]
        userName = "machine" [String]
        password = "AutoGenerate" [String]
        logLevel = "Errors" [None | All | Errors]
        clientConnectedCheck = "00:00:05" [HH:MM:SS]
        comAuthenticationLevel = "Connect" [None | Call | Connect | Default | Pkt | PktIntegrity | PktPrivacy]
        comImpersonationLevel = "Impersonate" [Default | Anonymous | Delegate | Identify | Impersonate]
        responseDeadlockInterval = "00:03:00" [HH:MM:SS]
        responseRestartDeadlockInterval = "00:03:00" [HH:MM:SS]
        autoConfig = "false" [true|false]
        maxWorkerThreads = "20" [number]
        maxIoThreads = "20" [number]
        minWorkerThreads = "1" [number]
        minIoThreads = "1" [number]
        serverErrorMessageFile = "" [String]
        pingFrequency = "10675199.02:48:05.4775807" [HH:MM:SS]
        pingTimeout = "10675199.02:48:05.4775807" [HH:MM:SS]
        maxAppDomains = "2000" [number]

    Internally to debug managed code we use a Windbg extension which works well with private symbols and has some additional commands the public sos.dll does not have and one of those shows the ThreadPool configuration, very useful to have a quick report of those configuration values.

    After the discussion I found myself thinking how to extract those values from a dump using only public symbols and the public sos extension everyone has access to; two of them are quite easy to find out (you can get a full list of members of ThreadpoolMgr class running mscorwks!ThreadpoolMgr*):

    0:000> ?poi(mscorwks!ThreadpoolMgr::NumberOfProcessors)
    Evaluate expression: 2 = 00000002
    0:000> ?poi(mscorwks!ThreadpoolMgr::MaxLimitTotalWorkerThreads) 
    Evaluate expression: 200 = 000000c8

    From System.Web.RequestQueue we can get _minLocalFreeThreads, _minExternFreeThreads and _queueLimit (RequestQueueLimit):

    Name: System.Web.RequestQueue
    MethodTable: 663aebf0
    EEClass: 663aeb80
    Size: 64(0x40) bytes
          MT    Field   Offset                 Type VT     Attr    Value Name
    79102290  4001203       14         System.Int32  1 instance      176 _minExternFreeThreads
    79102290  4001204       18         System.Int32  1 instance      152 _minLocalFreeThreads
    79102290  4001205       1c         System.Int32  1 instance     5000 _queueLimit
    7911228c  4001206       2c      System.TimeSpan  1 instance 0691da2c _clientConnectedTime
    7910be50  4001207       28       System.Boolean  1 instance        1 _iis6
    791087cc  4001208        4 ...Collections.Queue  0 instance 0691da40 _localQueue
    791087cc  4001209        8 ...Collections.Queue  0 instance 0691daf4 _externQueue
    79102290  400120a       20         System.Int32  1 instance        0 _count
    791186fc  400120b        c ...ding.WaitCallback  0 instance 0691dba8 _workItemCallback
    79102290  400120c       24         System.Int32  1 instance        0 _workItemCount
    7910be50  400120d       29       System.Boolean  1 instance        0 _draining
    7911228c  400120e       34      System.TimeSpan  1 instance 0691da34 _timerPeriod
    791127fc  400120f       10 ...m.Threading.Timer  0 instance 0691dbe8 _timer

    I was trying to find a convenient way to get the remaining three values (autoConfig, maxConnection and Number of timers) when I found this post from Tom, and now my quest is over: !DumpThreadConfig is the same command we have in the internal Windbg extension I mentioned and also the output is the same… so apparently you just have to be patient and wait for the next Debugging Tools release smile_regular


    Quote of the day:
    There is nothing new under the sun but there are lots of old things we don't know. - Ambrose Bierce
  • Never doubt thy debugger

    Ups and downs of ASP.NET Performance Counters


    Perfmon is likely the first tool you think to when it comes to monitor some internals and performance of your application and it’s relatively easy to find out information and resources on the Internet; unfortunately this seems to also be a fragile component and every now and then we receive new calls about it: event log spammed with warnings about missing or corrupted counters, values returned are inconsistent or clearly wrong etc…

    Messages like the following can appear after for example installing the .NET Framework 2.0 on your machine:

    The Open Procedure for service "ASP.NET" in DLL "C:\Windows\Microsoft.NET\Framework\v2.0.50727\aspnet_perf.dll" failed. Performance data for this service will not be available. Status code returned is data DWORD 0

    To note that such warnings are logged even if your application is not running.

    Some troubleshooting

    First of all you can try to disable any third party services and processes (for example using msconfig.exe or Autoruns) and see if the problem still reproduces. If it does, you can use How to manually rebuild Performance Counter Library values to try to fix it manually, or try the following:

    • Uninstall the  .NET Framework 2.0
    • If available for Perfc009.dat, Perfh009.dat and Perfh007.dat , Perfh007.dat, backup the registry keys
      • HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services
      • HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\Current Version\Perflib
    • Open a command prompt to your System32 folder
      • REN perf?009.dat *.bak
      • Expand <CD_drive_letter:>\i386\perfc009.DA_ %systemroot%\system32\perfc009.DAT
      • Expand <CD_drive_letter:>\i386\perfd009.DA_ %systemroot%\system32\perfd009.DAT
      • Expand <CD_drive_letter:>\i386\perfh009.DA_ %systemroot%\system32\perfh009.DAT
      • Expand <CD_drive_letter:>\i386\perfi009.DA_ %systemroot%\system32\perfi009.DAT
      • REN perf?007.dat *.bak
      • Expand <CD_drive_letter:>\i386\perfc007.DA_ %systemroot%\system32\perfc007.DAT
      • Expand <CD_drive_letter:>\i386\perfd007.DA_ %systemroot%\system32\perfd007.DAT
      • Expand <CD_drive_letter:>\i386\perfh007.DA_ %systemroot%\system32\perfh007.DAT
      • Expand <CD_drive_letter:>\i386\perfi007.DA_ %systemroot%\system32\perfi007.DAT
    • Open your system registry (be very careful!) to HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Perflib and set
      • Last Counter = 1846 (decimal) or 736 (hex)
      • Last Help = 1847 (decimal) or 737 (hex)
    • Now move to HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services
    • Search for the subkey Performance  in services and set the values
      • First Counter = 0
      • First Help = 0
      • Last Counter = 0
      • Last Help = 0
    • Back to the command prompt (still in System32)
    • findstr drivername *.ini > Perfini.txt
    • open  Perfini.txt  and note  drivername=nnn

    ================  sample  Perfini.txt :==================


    ================  Sample Perfini.txt :==================

    Using this information  unload und load the counter in  System32:

    unlodctr <drivername>
    lodctr <ini file>
    <drivername>  driver's name from  Perfini.txt
    <ini file>  listed  “.ini” files  in  Perfini.txt  

    Example:  ASP Drivername: axperf.ini:drivername=ASP 

    unlodctr ASP
    lodctr axperf.ini

    Or using the following command you can transfer settings from a  working machine to a not working one: 

    lodctr /s:Perf_all.ini
    lodctr Perf_all.ini
    Check Perfdiag 

    Disable not needed counters (as a last resource)

    If nothing else works as you wish or you do not want to waste time troubleshooting and just need the machine up and running without much annoyances you can disable the counters for the problematic service(s), of course at the cost of not having those data if you’ll ever need them.

    For this task you can use Exctrlst.exe utility from the OS Resource Kit (see Exctrlst Overview and Exctrlst UI); it disables the specific counter by adding a registry key similar to the example below. You can add these registry entries under the specific counter entry manually if Exctrlst.exe is not available or desirable

    Value name: Disable Performance Counters
    Value type: REG_DWORD
    Value data: 1

    A few fixes

    Some How To and general articles



    Quote of the day:
    The only time people dislike gossip is when you gossip about them. - Will Rogers
  • Never doubt thy debugger

    LogParser & event logs on Vista/Windows 2008


    I was discussing this morning with a customer, here’s a useful reminder to this post if you need to deal with “legacy” .evt logs on Vista/Win2008.

          wevtutil epl application.evt application.evtx /lf:true


    Quote of the day:
    That's the secret to life... replace one worry with another.... - Charles M. Schulz
Page 4 of 13 (187 items) «23456»