• #### How to automate Process Monitor

The other day I needed to capture a Process Monitor trace on a machine to troubleshoot a problem where the entire OS GUI was “broken” after the resume from screensaver; with “GUI broken” I mean that clicking on any icon on the desktop has no effect, the right click menu does not work etc…, but we are still able to access the Task Manager and run tasks (command line etc…) from the File > New Task (Run…) menu. This suggests the user’s registry is for some reason unloaded during when the screensaver is running, but the point is: if I run Procmon before the screensaver starts and wait for the problem to reproduce (to have a complete trace) then I’m unable to save the trace because I cannot access any running task which a GUI unless I start a new one or reboot the machine. Either way the trace is lost.

Coincidentally also my colleague Stefano had a similar problem and we (he, actually) found a couple of command line switches that can be used to control Procmon for such situations, in particular “/BackingFile” and “/Terminate”: /backingfile tells Procmon where to automatically save the trace, while /terminate actually starts a new instance of Procmon, terminates all other Procmon instances and exits.

So now I can start Procmon like:

procmon /quiet /minimized /backingfile c:\temp\trace.pml

before the screensaver starts, wait for the problem to reproduce, resume the OS and even without access to the GUI and from the Task Manager run:

procmon /terminate

Now I can safely reboot the machine and have my trace to analyze; of course you can also use those (and other) command line switches in a batch file or scripts if you need to automate the tool.

Carlo

Quote of the day:
The first question I ask myself when something doesn't seem to be beautiful is why do I think it's not beautiful. And very shortly you discover that there is no reason. - John Cage
• #### HashTable insert failed. Load factor too high

I already encountered a few recurrences of this errors and every time it has been a torture to capture a correct set of data (especially memory dumps) to review because this usually happens sporadically and you could have to wait weeks if not months before getting it again. There are a couple of hotfix available for this error message (see http://support.microsoft.com/default.aspx?scid=kb;EN-US;831730 and http://support.microsoft.com/default.aspx?scid=kb;EN-US;893314) but you already have them if you’re on the latest Service Pack for the .NET Framework.

Unless you’re using HashTable objects yourself… If that’s the case you might be running into a race condition which leads to the error, so you might want to consider to use the following code to synchronize the HashTable using SyncLock:

Dim MyHashTable as new Hashtable()
SyncLock MyHashtable.Syncroot
End SyncLock

As an alternative you can use the Hashtable.Synchronized() method to return a synchronized (thread-safe) wrapper for the HashTable object.

Carlo

Quote of the day:
Everyone is a genius at least once a year. The real geniuses simply have their bright ideas closer together. - Georg Christoph Lichtenberg
• #### /bin is not for native assemblies

After 4 years as a Support Engineer for ASP.NET, when I get a new case I usually can make a decently reliable idea pretty soon (often from the problem description the customer gives us, if it has enough details) about the problem complexity, how long it will take to resolve etc…, but every now and then (luckily not too often) there are occasions, as the one I’m going to talk about, where a fairly common problem as a “File not found” issue turns into something able to leave you completely puzzled and after you have considered almost every possibility you are left without any clue or idea about how to continue with troubleshooting. And again most of the times in those situations the solution comes from a fresh pair of eyes (and a new, well skilled brain) which looking at the problem from a different perspective can spot that small clue which makes the rest appear as obvious as the easiest of the problems…

### The problem

This time the problem description was well written: the customer developed a custom authentication mechanism for web applications, defined an authentication interface which can be implemented by other developers etc… The product exposes a “DoWorkProc” interface developed with Visual Studio 2003; they also developed an HttpModule “My Authentication” filter that allows customers to setup authentication by defining an assembly that implements an IAuth interface, and the customer shipped his product with a sample authentication assembly named MyAuthenticationSample.dll. My Authentication.dll and MyAuthenticationSample.dll assemblies are copied into the bin folder of the DoWorkProc application. When DoWorkProc is first referenced, the ASP.NET worker process binds the assemblies from the DoWorkProc /bin folder along with various system assemblies into the C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\Temporary ASP.NET Files folder. This all works properly and an intermediate binding assembly is created. Now the problem: if I create a new project under Visual Studio to implement the IAuth interface and I change the assembly name and project namespace to MyAuthenticationSample2 but making sure that everything else in the project is the same, the binding process fails with message “File or assembly name masDotNet, or one of its dependencies not found” and it highlights the “<add assembly="*"/>” line in machine.config; it appears that not all assemblies get copied to the “Temporary ASP.NET Files” folder as well.

### Troubleshooting

This smelt of a versioning problem, partially supported by the fact that using the compiled sample assembly resulted in the error above, while if the customer added the source project for the control to the new solution and compiled the them altogether, it worked…

The customer sent a repro I was able to run on my machine (it is always easier when we can repro, it’s much easier to debug and have a close look at what’s going wrong), and at first I actually found a versioning problem which fitted quite well with the error we got. The runtime was not able to find the appropriate version of a file it was looking for, hence the error: in the failing sample we were trying to load masDotNet.dll version 10.6.2719.37534, while the file version I had on disk (and the same loaded in the working sample) was version 10.6.0.5.

But… why does the error message mentions masDotNet.dll while to repro I had to use MyAuthenticationSample.dll and MyAuthenticationSample2.dll? Well, checking with the customer it turned out that masDotNet.dll is a mixed mode (C++ .NET) dll so we actually have two version numbers for that file: the version number in the PE header (what WinDBG shows, or what you can see from Windows Explorer if you look at the dll properties) and the version number in the assembly manifest have nothing to do with each other, it is the one in the manifest that is important for .NET assembly loading (binding), and at a more careful look that version was correct.

As often in this kind of cases I asked Doug to have a look at the problem with me and after some time spent working on the repro (basically to run again through the troubleshooting I did until that moment to double check) we confirmed in neither the non-working case nor the working case, neither MyAuthentication.dll nor MyAuthenticationSample.dll (nor MyAuthenticationSample2.dll in the non-working case) reference any mas* assembly in their manifest, neither directly nor indirectly. So why having differing versions of MyAuthentication.dll and MyAuthenticationSample.dll affects the version of masDotNet.dll that we look for? Very odd…

We then had a look at masDotNet.dll (which is managed C++) and we saw that from a native DLL dependency point of view it depends two third party native dlls we had in the /bin folder. Well, what are they doing in the BIN folder? ASP.NET’s BIN folder is for .NET assemblies, not native DLLs and the native OS loader will not find the native DLLs after they get shadow copied. So we did what should be done with native DLLs: put them in a location that was discoverable on the system PATH and removed them from the BIN folder.

I then checked and everything worked (also clearing out “temporary asp.net files” in between each test just to be sure), in both the previously working and non-working cases!

### Conclusion

There would be some additional questions to answer, such as why under some circumstances the native dlls in /bin where not a problem, while in other cases it was and consistently failed; unfortunately we did not had enough time left to continue our investigations (the customer needed urgently a working solution and preferred to not spend more time on the theory behind the issue), but here is a good bet: of the two native third party dlls, one (let’s call it A.dll) has a static dependency on the other one (call it B.dll). In the working case, B.dll was loaded in memory before A.dll, so when it was its turn it was able to find all the needed dependencies already in memory. But in the failing case for some reason (maybe because of timing, but I’m not really sure) A.dll was loaded first and then load of B.dll failed because it was not discoverable on the normal loader DLL search path.

Bottom line: the /bin folder is a sensitive place, for the above and other ones (check here for another example), be careful what you store in there!

Carlo

Quote of the day:
The real problem is not whether machines think but whether men do. - B. F. Skinner
• #### DropDownList with tooltip

I recently got this question from a customer whom is binding some data to a DropDownList control inside a GridView; the various items inside the DropDownList has variable length while the control itself per their page resign requirements must be fixed, making it difficult to fully read and understand the text associated with the selected item, so the customer asked for some hints to add a tooptip functionality to the DropDownList. Before raising the call he had already searched the Internet but without much luck; I did the same myself and apparently there are plenty of samples showing how to display a tooltip when the mouse is over an item already selected in a DropDown control, but the customer wanted something different: he needed to show the tooltip over each item in the DropDownList before actually selecting them to help users make the appropriate selection.

Well, for Internet Explorer 7 (al later) it is possible to use the “Title” attribute for the <option> tag:

for (int i = 0; i < GridView1.Rows.Count; i++)
{
//Add logic to fill the GridView

//Now let's iterate through the DropDown controls to add a "Title" to the
//resulting HTML "<option>" elements: this will add the toolip to highlighted element
for (int i2 = 0; i2 < dropDownList.Items.Count; i2++)
{
}
}

Here is how it looks like in IE7:

Even if Firefox (at least the version I’ve tried) does not respect the control width we have set through CSS and the DropDown for this sample is large enough to read the complete text, the tooltip is displayed:

Unfortunately this is not supported by previous versions if IE, though… if anyone has a suggestion, feel free to add a comment!

Carlo

Quote of the day:
Calamities are of two kinds: misfortunes to ourselves, and good fortune to others. - Ambrose Bierce
• #### The show goes on even if I’m on vacation

… so while I was taking a break from computers to move to a new house (I’m still packing and unpacking things), a few notable things happened: Internet Explorer 8 beta 2 has been released, Google released its new browser Chrome and my colleagues from the Italian Enterprise Support Group have started a team blog! So if you can at least read Italian I suggest you to keep an eye on it, knowing that smart people you can expect a lot of good content from them

Carlo

Quote of the day:
I was always taught to respect my elders and I've now reached the age when I don't have anybody to respect. - George Burns
• #### How many application pools can you cope with?

This is an interesting question I have to admit I had not thought to until we got a case from a customer whom was in trouble with his production servers: how many application pools can you run on IIS? Event better, how many w3wp.exe instances can the OS cope with? As quite often happens in our job, the correct answer is: it depends…

First, the problem. The customer was a hosting company so they had a large numbers of sites on each web server, around 450, each of them with its own application pool, each application pool run under its own specific account and aspExecuteInMTA was set to 1 on each server. Under some circumstances asp applications from some web sites are failing with the following message (roughly translated from Spanish):

Server error 'ASP 0177 : 80070057'
Error in Server.CreateObject
/site/page.asp, line 338
80070057

When the problem occurred the customer recycled the two or three top consuming web sites in terms of memory, threads and handles; then the failing web site was usually back online. Note that the failing web site was always only affected by other problematic ones, it was almost never the culprit.

At first we were distracted by the fact that the failing web sites where using Access as the database and we also got some errors like:

Server error 'ASP 0177 : 80070583'
Error in Server.CreateObject
/database.asp, line 9 -- "Provider=Microsoft.Jet.OLEDB.4.0;Data source=" & server.MapPath("../data/bd.mdb")
80070583

This is a deprecated configuration, see Using Microsoft JET with IIS:

The Access ODBC Driver and Microsoft OLE DB Provider for Microsoft Jet are not intended to be used with high-stress, high-concurrency, 24x7 server applications, such as web, commerce, transactional, messaging servers, and so on.

Anyway we soon found that was not the right direction. 450 application pools are a lot, plus there was the thing about recycling the most handles and threads consuming pools… this suggests desktop heap exhaustion.

### A bit of theory

Desktop Heap overview has a good explanation of what Desktop Heap is and how things might start to go wrong when the system runs low in Desktop Heap storage; it also has a few interesting bits about IIS:

Regarding non-ASP.NET applications, IIS 6.0 has been tested on a well-configured mainframe server running up to 2,000 concurrent worker processes, each serving one application pool, but not using unique identities. In practice, a design of up to 500 simultaneous application pools is achievable, depending on the application requirements and assuming hardware resources are not a significant constraint. It is important to configure the UseSharedWPDesktop registry setting mentioned above when using more than 60 application pools with unique identities

My colleague Nicolas also has a post about Desktop Heap and Windows Stations here. Having one shared desktop can theoretically bring some security issues, but David Wang explains well this scenario:

"we should use a common desktop when the number of application pools with unique identities configured on a server is greater than 60". In other words, IIS does not have a problem running >60 application pools which uses <60 unique identities. The limitation in question is the number of user desktops associated with each unique application pool identity that Windows simultaneously supports.

The following are the interesting combinations and consequences:

1. If UseSharedWPDesktop=0 and all application pools launch as Network Service, then there is exactly one Desktop (of Network Service) shared by all the application pools. You have no isolation of desktop nor process identity between application pools.
2. If UseSharedWPDesktop=0 and all application pools launch as unique user identity, then there is a unique Desktop for each user identity, which is obviously not shared. This is limited to a number around 60 due to Desktop heap limit in Windows Server 2003. This means that you can only have ~60 application pools perfectly isolated by desktop and process identity simultaneously running.
3. If UseSharedWPDesktop=1 and all application pools launch as unique user identity, then there is one Desktop that is shared by all the user identities. This removes the limit of 60 at the cost of not perfectly isolated application pools. You have no desktop isolation but retain process identity isolation between application pools.
4. UseSharedWPDesktop=1 and all application pools launch as Network Service is pretty much a degenerate case that is not interesting. :-)

### Back to the problem

The customer was already aware of those blog posts so they were already using UseSharedWPDesktop=1 (otherwise they were not able to run more than 50 concurrent application pools), set sessionViewSize to 64 Mb and the default value for “HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\SubSystems\Windows”.

To know more about Desktop Heap status on the server we used Desktop Heap Monitor: this comes with a comprehensive usage guide, anyway here are some quick steps to use it:

1. Log in with an account that has Administrator privilege. Note that you must either do this at the physical interactive console (i.e. not through Terminal Services) OR connect to the console session using mstsc /console (only possible on XP or Server 2003)
2. Open a command prompt and change to "c:\kktools\dheapmon8.0"
3. For uni-processor systems run
• DHEAPINST -f
4. For multi-processor systems (including single processor hyper-threaded systems) run
• DHEAPINST
• You will be prompted to agree to the symbol server terms of use. You must accept these for the tool to work
The server in question must be able to make an outgoing connection to the Internet for this to work
5. For either 3. or 4. you should see
• dheapinst - Desktop Heap Monitor installed successfully
6. DHEAPMON uses a driver in order to read information from the kernel memory. This must be loaded by the following command:
• DHEAPMON –l
• You should see this confirmation message: Dheapmon - Driver loaded successfully
7. You are now ready to run the tool and capture the desktop heap information. Run the following command:
• DHEAPMON -v > desktopheap.txt
• Important: if the output text file does not contain a line like this: “Windowstation: (Service-0x0-3e7\$) SessionID: 0” (without quotation marks) then it is almost certain that you did not connect to the server using Terminal Services Client using the /console switch. You must connect using /console to get the full information required

To automate the process of gathering Desktop Heap information, create a batch file called HeapMon.bat containing the following:

@echo off
rem vars C B A are day month year in the for line below, change them to change date
for /F "tokens=2,3,4 delims=/- " %%A in ('date /T') do set DT=%%C_%%B_%%A
for /F "tokens=1,2 delims=/: " %%A in ('time /T') do set TM=%%A-%%B
echo %DT%
echo %TM%
Dheapmon -v > dheapmon_out_%DT%_%TM%.txt

This will create an output file uniquely named with the date and time included; you can use task scheduler to run this batch file at regular intervals, e.g. once per hour.

Once finished all diagnostic activity with DHeapMon, unload the device driver with:

DHEAPMON -u

To uninstall dheapmon, run:

DHEAPMON -r

From the report I got is was clear that the customer was really using only 1 desktop within IIS WinSTA (remember he was using a shared desktop across his application pools); as IIS WinSTA is created as non interactive, this single desktop is 512 Kb but it was exhausted at 99.2% when then problem appeared:

Windowstation: (WP_WINSTA-054f77ac-6ece-4080-b3a2-b0bf08dacaab) SessionID: 0
Desktop Heap          524288 (0x   80000) Bytes
Committed             520192 (0x   7f000) Bytes
Uncommitted             4096 (0x    1000) Bytes
Allocated             520072 (0x   7ef88) Bytes
Total Freed              120 (0x      78) Bytes
Unused                  4216 (0x    1078) Bytes
Used Rate                    (      99.2) %

To calculate SessionViewState size we should do something like:

312Kb (Winsta0/Default) +
64Kb (Winsta0/Disconnect) +
128Kb (Winsta0/Winlogon) +
7 * 512Kb (all other non interactive winsta) +
512Kb (IIS non interactive winsta WP_WINSTA-xxx)
== 4600 Kb  (< 5 Mb)

So apparently the problem is not on SessionViewSize bur rather just on the non interactive Desktop Heap size; moreover setting this value to 64 Mb as the customer did is not necessary because they were using a bit less than 4 Mb for that value so also the default (20 Mb) were far enough.

The problem of increasing this one too much is that not only IIS owns one, so we first tried using 1024 Kb:

312 + 64 + 128 + 7 * 1024 = 7672 Kb (< 7 Mb)

So we tried with the following (not the customer was not using /3gb):

• UseSharedWPDesktop enabled
• Delete the SessionViewSize registry key to get the default 20 Mb
• SharedSection=1024,3072,1024

This first try allowed the server to run fine a bit longer than before but at the end the problem was there again, and another heapmon trace showed we were still exhausting Desktop Heap… we gave another try with SharedSection=1024,3072,2048 (312 + 64 + 128 + 7 * 1024 = 14840 Kb (< 15 Mb).

Bingo, this time we did it!

Of course be very careful when you manipulate the Desktop Heap configuration and if you’re not sure how to do (or if you’re doing it right, or if you need help to fine tune your settings etc…) give Microsoft Support a call!

### Conclusions

To close with some general recommendations:

1. Any kind of platform that has to support more than concurrent 450 processes should be considered to migrate under x64, and to really care about context switching
2. Windows 2008 Server with Dynamic Session Space removes the SessionViewSize limits, but desktop heap limits would still apply
3. 450 processes identity is also a very big because logon session & token cache might duplicate kernel memory
4. Always be very careful when changing those values since this might have effect on other kernel quota

The point is we will always have inherent x32 bit limitations that we cannot workaround and 450 application pools is a pretty close “practically tested” limit…

Carlo

Quote of the Day:
Women like silent men. They think they're listening. - Marcel Archard
• #### Visual Studio 2008 SP1 and .NET 3.5 SP1 available for download

Microsoft Visual Studio 2008 Service Pack 1 (exe)

This download installs Visual Studio 2008 Service Pack 1 (SP1) and the .NET Framework 3.5 SP1. SP1 addresses issues that were found through a combination of customer and partner feedback, as well as internal testing. These service packs offer Visual Studio and .NET Framework users improvements in responsiveness, stability and performance.

Microsoft .NET Framework 3.5 Service Pack 1

.NET Framework version 3.5 Service Pack 1 provides the following new features and improvements:

• ASP.NET Dynamic Data, which provides a rich scaffolding framework that enables rapid data driven development without writing code, and a new addition to ASP.NET AJAX that provides support for managing browser history (back button support). For more information, see What’s New in ASP.NET and Web Development.
• Core improvements to the CLR (common language runtime) that include better layout of .NET Framework native images, opting out of strong-name verification for fully trusted assemblies, improved application startup performance, better generated code that improves end-to-end application execution time, and opting managed code to run in ASLR (Address Space Layout Randomization) mode if supported by the operating system. Additionally, managed applications that are opened from network shares have the same behavior as native applications by running with full trust.
• Performance improvements to WPF (Windows Presentation Foundation), including a faster startup time and improved performance for Bitmap effects. Additional functionality for WPF includes better support for line of business applications, native splash screen support, DirectX pixel shader support, and the new WebBrowser control.
• ClickOnce application publishers can decide to opt out of signing and hashing as appropriate for their scenarios, developers can programmatically install ClickOnce applications that display a customized branding, and ClickOnce error dialog boxes support links to application-specific support sites on the Web.
• The Entity Framework is an evolution of the existing suite of ADO.NET data access technologies. The Entity Framework enables developers to program against relational databases in according to application-specific domain models instead of the underlying database models. For more information, see Getting Started with the Entity Framework. The Entity Framework introduces some additional features, including support for new SQL Server 2008 types, default graph serialization of Entities, and the Entity Data Source. This release of the Entity Framework supports the new date and file stream capabilities in SQL Server 2008. The graph serialization work helps developers who want to build Windows Communication Foundation (WCF) services that model full graphs as data contracts. The Entity Data Source provides a traditional data source experience for ASP.NET Web application builders who want to work with the Entity Framework.
• LINQ to SQL includes new support for the new date and file stream capabilities in SQL Server 2008.
• The ADO.NET Data Services Framework consists of a combination of patterns and libraries, which enable data to be exposed as a flexible REST (Representational State Transfer)-based data service that can be consumed by Web clients in a corporate network or across the Internet. The ADO.NET Data Services Framework makes data service creation over any data source. A conceptual view model of the underlying storage schema can easily be exposed through rich integration with the ADO.NET Entity Framework. Services created by using the ADO.NET Data Services Framework, and also compatible Windows Live (dev.live.com) services, can be easily accessed from any platform. For client applications that are running on Microsoft platforms, a set of client libraries are provided to make interaction with data services simple. For example, .NET Framework-based clients can use LINQ to query data services and a simple .NET Framework object layer to update data in the service.
• Windows Communication Foundation now makes the DataContract Serializer easier to use by providing improved interoperability support, enhancing the debugging experience in partial trust scenarios, and extending syndication protocol support for wider usage in Web 2.0 applications.
• The .NET Framework Data Provider for SQL Server (SqlClient) adds new support for file stream and sparse column capabilities in SQL Server 2008.

Carlo

Quote of the Day:
Resentment is like taking poison and hoping the other person dies. - St. Augustine

• #### Charting with LogParser

A picture worth 1000 words and this is true also when you have to analyze IIS logs to get some statistics on your site; as you already know, LogParser is extremely flexible and very useful in such situation and its chart output format is what it’s needed.

The first thing you need is a copy of Office Web Components installed on the machine; if you don’t, LogParser will complain and return this error:

Error creating output format "chart": This output format requires a licensed Microsoft Office Chart Web Component to be installed on the local machine

From LogParser help:

chartType

Values:
name of chart type

Default:
Line

Description:
Chart type

Details:
The set of available chart types depends on the version of the Microsoft Office Web Components installed on the local computer.
For a list of the available chart types, type the following
help command from the command-line shell:

LogParser -h -o:CHART

Currently the latest Office Web Components version available (the newest I’ve been able to find on the Internet) is version 11 which has been released for Office 2003.

Changes in the 2007 Office System

OWC11
Description: OWC11 is removed from the 2007 Office system and now ships only with Microsoft Office Project 2007. The DataFinder functionality in OWC11 is also removed. This affects solutions that use the spreadsheet, chart, pivot table, and data source control functionality of OWC11 in a Web page or client program that requires installation of this ActiveX control on the user's computer. The user experience varies depending upon the host container of the OWC11 ActiveX control.

Reason for change: New technologies such as Windows SharePoint Services 3.0 with Excel server side capabilities and the new charting component in the 2007 Office system replace some features of OWC11.

And if you have Office 2007 like me? I installed OWC11 anyway and everything is still working fine so you should be fine; just in case you need the download links, here they are: Office 2003 Add-in: Office Web Components and Microsoft Office 2003 Web Components Service Pack 1 (SP1) for the 2007 Microsoft Office System.

### A few samples

Bytes per data type as exploded pie

LogParser
"SELECT TO_UPPERCASE(EXTRACT_EXTENSION(cs-uri-stem)) AS PageType, MUL(PROPSUM(sc-bytes),100.0) AS Bytes
INTO Pie.gif FROM *.log
GROUP BY PageType ORDER BY Bytes DESC"
-chartType:PieExploded -chartTitle:"Bytes per data type" -categories:off -o:chart -i:iisw3c

Calculating percentages per data type

logparser
"select extract_extension(cs-uri-stem) as Resource, mul(propcount(*),100.0) as ResourceHits
into percentage.jpg from *.log
group by Resource order by ResourceHits desc"
-chartType:PieExploded3D -chartTitle:"Percentages per data type" -categories:off -o:chart -i:iisw3c

.aspx page hits per minute

logparser
"select quantize(time, 60) as TimeGenerated, count(*) as Hits
into qnt.gif from *.log
where to_lowercase(extract_extension(cs-uri-stem))='aspx' group by TimeGenerated"
-i:iisw3c -o:chart -chartType:Line

Getting the number of unique visitors (actually the number of unique client IP addresses) is a bit more complex. We could try the following:

select date, count(distinct c-ip) into UniqueIPs.gif from *.log group by date" -i:iisw3c -o:chart -chartType:Line

but we get this error:

Error: Semantic Error: aggregate functions with DISTINCT arguments are not supported with GROUP BY clauses

The solution in this case is to split the query into two separate commands; for example you could create a .bat file (or script, or whatever method you prefer) and run the following:

@echo off

logparser
"select distinct date, c-ip into dates.txt from *.log" -i:iisw3c -o:w3c

logparser
"select date, count(c-ip) as UniqueIPs into UniqueIPs.gif
from dates.txt group by date order by date"
-i:iisw3c -o:chart -chartType:SmoothLine -view:on

### Advanced carting with configuration scripts

Using an external script file (JScript or VBScript) it is possible to have a deeper control over the produced chart; this is based on two global objects which expose methods and properties that can be used to modify parameters such as the chart colors, the chart fonts, and many other attributes. Those two global objects are instances of the chartSpace and chart objects of the Microsoft Office Web Components ChartSpace object model, and they are named "chartSpace" and "chart"; for more information on those objects you can have a look at the MSDN ChartSpace Object Model documentation.

Taken from the LogParser documentation, we can use the following sample script to add a caption to the previous UniqueIPs chant and make the background color transparent:

// Add a caption
chartSpace.HasChartSpaceTitle = true;
chartSpace.ChartSpaceTitle.Caption = "Generated by Log Parser 2.2";
chartSpace.ChartSpaceTitle.Font.Size = 8;
chartSpace.ChartSpaceTitle.Position = chartSpace.Constants.chTitlePositionBottom;

// Change the background color
chart.PlotArea.Interior.Color = chartSpace.Constants.chColorNone;
@echo off

logparser
"select distinct date, c-ip into dates.txt from *.log" -i:iisw3c -o:w3c

logparser
"select date, count(c-ip) as UniqueIPs into UniqueIPs.gif
from dates.txt group by date order by date"
-i:iisw3c -o:chart -chartType:SmoothLine -config:MyScript.js -groupSize: 800x600 -view:on

From here you can experiment using the ChartSpace object model to further customize the appearance of your charts: the limit is your imagination

Carlo

Quote of the day:
Ability will never catch up with the demand for it. - Malcolm Forbe
• #### LogParser did it again: application pool recycle

This started as a “standard” w3wp.exe crash, but this time IIS was also randomly disabling one of the application pools on the server. This is a protection mechanism we have since IIS 6 called Rapid Fail Protection that disables ad application pool (for security and stability reasons) if it fails (encounters severe errors) too often: how often and how much time it’s configurable (see Configuring Rapid-Fail Protection in IIS 6.0), the default is 5 failures in 5 minutes.

Looking at the System event log there were plenty of messages like:

A worker process with process id of 'xxx' serving application pool 'MyAppPool' has requested a recycle because it reached its virtual memory limit.

And also:

A process serving application pool 'MyAppPool' exceeded time limits during shut down. The process id was 'xxx'.

How often? As usual, LogParser comes at hand.

logparser
"select quantize(TimeGenerated, 300) as TimeGen, count(*) as NumberOfrecycles
into memory_limit_recycles.log from system.evt
where EventID = 1077 and Message like '%\'MyAppPool%' group by TimeGen order by TimeGen"
-i:evt -o:nat -rtp:-1

Note the “quantize” function: LogParser help states “Truncates a value to the nearest multiple of another value”; what it actually does in the command above is count the number of recurrences of a certain event in “blocks” of 5 minutes (300 seconds in my sample).

The output is something like:

Generated           NumberOfRecycles
------------------- ----------------
2008-07-06 21:55:00 5
2008-07-06 22:00:00 5
2008-07-06 22:05:00 5
2008-07-06 22:10:00 4
2008-07-06 22:15:00 4
2008-07-06 22:20:00 3
2008-07-06 22:25:00 1
2008-07-06 22:30:00 2
2008-07-06 22:35:00 1
2008-07-06 22:40:00 2
2008-07-06 22:45:00 1
2008-07-06 22:50:00 2
2008-07-06 22:55:00 1
2008-07-06 23:00:00 2
2008-07-06 23:05:00 1
2008-07-06 23:10:00 2
2008-07-06 23:15:00 1
2008-07-06 23:20:00 2
2008-07-06 23:25:00 1
2008-07-06 23:30:00 2
2008-07-06 23:35:00 3
[...]

As you can see the pool is recycled quite often, event 4-5 times every 5 minutes, exactly the default limit configured for pool heal checking in IIS… Moreover we don’t have to forget about the second error message:

logparser
"select quantize(TimeGenerated, 300) as TimeGen, count(*) as ShutDownLimit into ShutDownLimit.log
from system.evt where Message like '%\'MyAppPool%' and Message like '%exceeded time limits during shut down%'
group by TimeGen order by TimeGen" -i:evt -o:nat -rtp:-1

And the output is:

TimeGen             ShutDownLimit
------------------- -----
2008-07-09 09:05:00 3
2008-07-09 10:00:00 5
2008-07-11 14:40:00 3
2008-07-15 10:50:00 5
2008-07-15 11:30:00 3
2008-07-15 13:00:00 3
2008-07-15 14:45:00 3
2008-07-15 14:50:00 4
2008-07-15 14:55:00 3
2008-07-15 16:15:00 3
2008-07-15 16:20:00 4
2008-07-15 16:40:00 3
2008-07-15 16:45:00 3
2008-07-15 17:00:00 3
2008-07-15 17:25:00 3
2008-07-15 17:30:00 3
2008-07-15 17:40:00 3
2008-07-15 17:45:00 3
2008-07-15 17:55:00 3
2008-07-15 18:00:00 4

Sometimes we’re clearly exceeding the configured pool health checking so really no surprise IIS disables it.

But why the pool is recycled so often? Well, we found a mistake in customer’s configuration: the application pool was configured to be recycled when the virtual memory reached 50 Mb, definitely a value too low. Increasing it to resolved also this problem: no more pool recycles and as obvious side effect the users noticed an increased stability and performance when working with the application.

Carlo

Quote of the day:
A wise man gets more use from his enemies than a fool from his friends. - Baltasar Gracian
• #### Time-taken and LogParser for web site statistics

The time-taken field is quite handy if your web application or IIS web server is performing poorly (or slower that you expect) or even for simple statistics and monitoring purposes, if you want to keep under how long it takes every web request to be processed and the page sent back to the client.

In earlier versions of IIS the time-taken field was not selected by default so I often had to request customers to first enable it, wait some more time for the problem to reoccur (to give IIS the change to create a significant amount of logs with the new value) before being able to start troubleshooting. Well, at least before being able to extract some statistics involving the duration of web requests.

Luckily in IIS 7 time-taken is now selected by default so we already it logged and ready to use.

Taken from the IIS online docs, here is the meaning each field in IISW3C logs:

What does time-taken actually measures? It measures the length of time that it takes for a request to be processed. The client-request time stamp is initialized when HTTP.sys receives the first byte of the request. HTTP.sys is the kernel-mode component that is responsible for HTTP logging for IIS activity. The client-request time stamp is initialized before HTTP.sys begins parsing the request. The client-request time stamp is stopped when the last IIS response send completion occurs. Beginning in IIS 6.0, the time-taken field typically includes network time. Before HTTP.sys logs the value in the time-taken field, HTTP.sys usually waits for the client to acknowledge the last response packet send operation or HTTP.sys waits for the client to reset the underlying TCP connection. Therefore, when a large response or large responses are sent to a client over a slow network connection, the value of the time-taken field may be more than expected.

### A few samples

If you are either troubleshooting or just monitoring your site/IIS, LogParser can be extremely useful to elaborate statistics and charts out of your IIS logs, find average and peak execution time for your pages and web services etc…

For example if you want to get the average execution time for your .aspx (or .asmx) pages/services you can run something like:

logparser
"SELECT AVG(time-taken) As AverageTimeTaken, MAX(time-taken) As MaxTimeTaken, COUNT(*) As Hits, TO_LOWERCASE(cs-uri-stem) As Uri
INTO average_uri_execution_time.log FROM ex*.log
WHERE EXTRACT_EXTENSION(TO_LOWERCASE(cs-uri-stem)) = %1
GROUP BY TO_LOWERCASE(cs-uri-stem)
ORDER BY AverageTimeTaken DESC"
-i:IISW3C -o:TSV

Note: the command must be placed on a single line, I split it here just to be easier to read

Using the IISW3C input format, if you have your logs organized in subfolders you can parse them all at once appending “-recurse:-1” to the command above.

Here you can get the top n slowest pages to complete:

logparser
"SELECT TOP 100 time-taken, cs-uri-stem
INTO top_time_taken.log FROM ex*.log
WHERE extract_extension(to_lowercase(cs-uri-stem)) = 'aspx'
ORDER BY time-taken DESC"
-i:iisw3c -o:tsv

If you are interested only on a specific timeframe (say you had a load pick and want to know more) you can take the cue from here:

logparser
"SELECT AVG(time-taken) As AverageTimeTaken, MAX(time-taken) As MaxTimeTaken, COUNT(*) As Hits, TO_LOWERCASE(cs-uri-stem) As Uri
INTO date_range.txt FROM *.log
WHERE TO_TIMESTAMP(date,time) BETWEEN TIMESTAMP('2008/05/27 10:11:00', 'yyyy/MM/dd hh:mm:ss') AND TIMESTAMP('2008/05/27 10:13:00', 'yyyy/MM/dd hh:mm:ss')
GROUP BY TO_LOWERCASE(cs-uri-stem)"
-i:IISW3C -RECURSE:-1 -o:NAT -RTP:-1

If you want an overview of resource types served by your IIS (with resource I mean every file extension IIS logs) you can use something like this:

logparser
"select to_lowercase(extract_extension(cs-uri-stem))as ResourceType, MIN(time-taken) as MinTimeTaken, MAX(time-taken) as MaxTimeTaken, AVG(time-taken) as AvgTimeTaken, COUNT(*) as Hits
INTO timers.txt FROM time-taken.log GROUP BY ResourceType"
-i:IISW3C -o:NAT -RTP:-1

A few words about the load you add on the serve enabling additional fields for IIS logs: the default files are essentially the same in IIS 6 and 7 except for time-taken which is now enabled by default on IIS 7. This suggests the load you’ll add to the server is negligible, but as always is really depends on your server and application; to stay on the safe side you can add fixed-size fields, for example handle with case the cookie field, enable it if you need to troubleshoot specific problems involving cookies (session mixed-up etc…), otherwise just leave it off.

Carlo

Quote of the day:
Art is science made clear. - Jean Cocteau
• #### Visual Studio 2008 crashes in “split view”

“Split view” is one of the new features in Visual Studio 2008 web designer: this is the possibility to have Design View and Source View of your page at the same time (see What's New in ASP.NET and Web Development, “Visual Web Developer Enhancements” paragraph). A few days ago a customer called in to report a problem with his Visual Studio and split view: when trying to view a file (even a new one) in split view or design view the IDE either crashed or frozen.

As you can imagine we took a crash dump with adplus, but a first look at the threads, stack, exceptions etc… (the usual stuff) did not show anything interesting. Then I thought to some external process (like an antivirus, third parties add-ons etc…) that might meddle in and had a look at the list of modules loaded within the process with lmf (list loaded modules with full path). There where only a couple of odd dlls loaded, from Office 2003:

33f20000 34118000   FPCUTL     (deferred)
Image path: C:\Program Files\Microsoft Office\OFFICE11\FPCUTL.DLL
Image name: FPCUTL.DLL
Timestamp:        Wed Jun 06 19:44:55 2007 (4666F297)
CheckSum:         001EC356
ImageSize:        001F8000
File version:     11.0.8170.0
Product version:  11.0.8170.0
File OS:          40004 NT Win32
File type:        2.0 Dll
File date:        00000000.00000000
Translations:     0409.04e4
CompanyName:      Microsoft Corporation
ProductName:      Microsoft Office 2003
InternalName:     FP40CUTL
OriginalFilename: FP40CUTL.DLL
ProductVersion:   11.0.8170
FileVersion:      11.0.8170
FileDescription:  Microsoft Office FrontPage Client Utility Library

37050000 37157000   OMFC       (deferred)
Image path: C:\Program Files\Microsoft Office\OFFICE11\OMFC.DLL
Image name: OMFC.DLL
Timestamp:        Fri Apr 13 21:43:21 2007 (461FDD59)
CheckSum:         0010A869
ImageSize:        00107000
File version:     11.0.8164.0
Product version:  11.0.8164.0
File OS:          40004 NT Win32
File type:        2.0 Dll
File date:        00000000.00000000
Translations:     0000.04e4
CompanyName:      Microsoft Corporation
ProductName:      Microsoft Office 2003
InternalName:     OMFC.DLL
OriginalFilename: OMFC.DLL
ProductVersion:   11.0.8164
FileVersion:      11.0.8164
FileDescription:  Microsoft Office MFC
LegalCopyright:   Copyright © 1993-2003 Microsoft Corporation.  All rights reserved. 

That rang a bell, especially about fpcutl.dll… This dll comes with both Visual Studio and Office 2003 but the two versions are incompatible: Visual Studio in this case was loading the wrong one. Why?

Well, it happens that Visual Studio looks at the system PATH to load this (and other) component, while Office doesn’t… Here is the PATH value from the dump (use !procinfo and then check the path value):

C:\WINDOWS\system32;
C:\WINDOWS;
C:\WINDOWS\System32\Wbem;
C:\Program Files\Microsoft SQL Server\90\Tools\binn\;
C:\Program Files\Microsoft SQL Server\90\DTS\Binn\;
C:\Program Files\Microsoft SQL Server\90\Tools\Binn\VSShell\Common7\IDE\;
C:\Program Files\Microsoft Visual Studio 8\Common7\IDE\PrivateAssemblies\;
C:\PROGRA~1\MICROS~2\OFFICE11

The Office 11 entry is the last one… but there is something wrong anyway, who spots it?

We have an entry for Visual Studio 8 (Visual Studio 2005), but where is the entry for Visual Studio 9 (VS 2008)?

For some reason the PATH value has not been updated (or maybe it has modified after the setup) and we don’t know exactly why; the point is that we need to fix it. A quick search on the machine found three copies of fpcutl.dll:

We added the missing folders at the beginning of the PATH string which now reads as:

C:\Program Files\Microsoft Web Designer Tools\VWD;
C:\Program Files\Microsoft Visual Studio 9.0\Common7\IDE;
C:\Program Files\Microsoft Visual Studio 9.0\VC\BIN;
C:\Program Files\Microsoft Visual Studio 9.0\Common7\Tools;
C:\Windows\Microsoft.NET\Framework\v3.5;
C:\Windows\Microsoft.NET\Framework\v2.0.50727;
C:\Program Files\Microsoft Visual Studio 9.0\VC\VCPackages;
C:\WINDOWS\system32;
C:\WINDOWS;
C:\WINDOWS\System32\Wbem;
C:\Program Files\Microsoft SQL Server\90\Tools\binn\;
C:\Program Files\Microsoft SQL Server\90\DTS\Binn\;
C:\Program Files\Microsoft SQL Server\90\Tools\Binn\VSShell\Common7\IDE\;
C:\Program Files\Microsoft Visual Studio 8\Common7\IDE\PrivateAssemblies\;
C:\PROGRA~1\MICROS~2\OFFICE11

The most important one on this case is “C:\Program Files\Microsoft Web Designer Tools\VWD”: Visual Studio will look into this folder to load the right version of fpcutl.dll (PATH is a first in – first serve list of folders, the first match will be the one we use) and Office continues working happily anyway.

Carlo

Quote of the day:
Any sufficiently advanced technology is indistinguishable from magic. - Arthur C. Clarke
• #### An interesting fusion cache lock: that’s what the GAC is meant for

This has been an interesting case where we had an ASP.NET 2.0 application which under load was completely blocked after a few minutes. Since we were talking about a hang/deadlock (as reported from the customer), the !critlist command (you can find the command within the SieExtPub.dll extension) is a good start:

0:021> !critlist
CritSec at 7a393800.  Owned by thread 21.
Waiting Threads: 6 7 8 9 10 11 12 13 42 72 104 107 108 109 110 111 112 113 114 115 116
CritSec at    e194c.  Owned by thread 32.
Waiting Threads: 21

Thread 32 holds a critical section and thread 21 is waiting on it, but thread 21 is also owning another critical section, and about 20 other threads are waiting there… so apparently this is not a real deadlock (21 is waiting on 32 but 32 is not waiting on 21); but what is thread 32 doing?

0:032> kpL2000
0327fd30 7c822124 ntdll!KiFastSystemCallRet
0327fd34 77e6baa8 ntdll!NtWaitForSingleObject+0xc
0327fda4 79e718fd kernel32!WaitForSingleObjectEx+0xac
0327fe38 79e7187c mscorwks!CLREvent::WaitEx+0x117
0327fe48 7a0e288e mscorwks!CLREvent::Wait+0x17
0327ff88 7a0d867b mscorwks!SVR::GCHeap::SuspendEE+0x16c
0327ffec 00000000 kernel32!BaseThreadStart+0x34

It is waiting to start GC but it cannot, because thread 95 has PreEmptive disabled:

0:032> !threads
Hosted Runtime: yes
PreEmptive   GC Alloc           Lock
ID OSID ThreadOBJ    State     GC       Context       Domain   Count APT Exception
23    1 1288 0012efb8   1808220 Enabled  1048ebfc:1048efe8 0011f578     0 MTA (Threadpool Worker)
[...]
95   35 1114 05424c30   180b222 Disabled 16577334:16579008 00162f38     1 MTA (Threadpool Worker)
96   36 1b54 05407bf8   180b220 Enabled  145124ec:14512fe8 0011f578     0 MTA (Threadpool Worker)
[...]

As you might know, if a thread has PreEmptive GC disabled is because it does not want to be interrupted by the GC; loading a file is a good reason to disable PreEmptive and that’s exactly the case:

0:095> !clrstack
ESP       EIP
05e5ecd0 7c82ed54 [HelperMethodFrame_PROTECTOBJ: 05e5ecd0] System.Reflection.Assembly.nLoadFile(System.String, System.Security.Policy.Evidence)
05e5ef9c 04674f82 MyApp.Core.Lib.Managers.LangMan.ResourceFile(System.String, System.String, MyApp.Core.Lib.Managers.ResourceMode)
05e5efd8 04674f0c MyApp.Core.Lib.Managers.LangMan.ResourceFile(System.String, System.String)
05e5efe0 04674ee5 MyApp.FPCore.Pages.Page.ResourceFile(System.String, System.String)
05e5f028 031bb2b5 [MulticastFrame: 05e5f028] System.EventHandler.Invoke(System.Object, System.EventArgs)
[...]

!showstringw from SieExtPub on the first parameter passed to mscorwks!ExplicitBind we can get the name of the file we’re trying to load:

0:095> kb
05e5e860 7c822124 77e6baa8 00001348 00000000 ntdll!KiFastSystemCallRet
05e5e864 77e6baa8 00001348 00000000 00000000 ntdll!NtWaitForSingleObject+0xc
05e5e8d4 79e718fd 00001348 ffffffff 00000000 kernel32!WaitForSingleObjectEx+0xac
05e5e918 79e718c6 00001348 ffffffff 00000000 mscorwks!PEImage::LoadImage+0x199
05e5e968 79e7187c ffffffff 00000000 00000000 mscorwks!CLREvent::WaitEx+0x117
05e5e978 79f90f81 ffffffff 00000000 00000000 mscorwks!CLREvent::Wait+0x17
05e5e988 79f917ba 7a390970 03c55358 ffffffff mscorwks!CExecutionEngine::WaitForEvent+0x37
05e5e99c 7a15cfce 03c55358 ffffffff 00000000 mscorwks!ClrWaitEvent+0x17
05e5e9ac 7a153339 05e5ed7c 00000000 00000000 mscorwks!CSimpleFusionBindSink::Wait+0xe
05e5e9c8 7a1545d2 03cecd70 0016e3d0 00000200 mscorwks!BindHelper+0x7d
05e5ec64 7a2d0c7c 05e5ed7c 0016e3d0 00000000 mscorwks!ExplicitBind+0x1a6
05e5ef80 793f583f 00000000 16577088 1657703c mscorwks!AssemblyNative::LoadFile+0x1de
[...]

0:095> !showstringw 05e5ed7c
D:\MyApp\App\Members\bin\DescMembers.dll

Looking around in the stack it’s interesting to note that also thread 81 was loading the same file from from ASP.NET temporary folder:

0:081> kb
0573cf58 7c822124 77e6baa8 000004f4 00000000 ntdll!KiFastSystemCallRet
0573cf5c 77e6baa8 000004f4 00000000 00000000 ntdll!NtWaitForSingleObject+0xc
0573cfcc 79e718fd 000004f4 ffffffff 00000000 kernel32!WaitForSingleObjectEx+0xac
0573d010 79e718c6 000004f4 ffffffff 00000000 mscorwks!PEImage::LoadImage+0x199
0573d060 79e7187c ffffffff 00000000 00000000 mscorwks!CLREvent::WaitEx+0x117
0573d070 7a0851cb ffffffff 00000000 00000000 mscorwks!CLREvent::Wait+0x17
0573d080 79f40e96 00000000 5ed28252 79eaebea mscorwks!SVR::GCHeap::WaitUntilGCComplete+0x32
0573d0bc 79e797c6 5ed2822a 79eaebea 053ee59a mscorwks!Thread::RareDisablePreemptiveGC+0x1a1
0573d0e8 79ea91b2 7a3879e0 5ed283e6 79eaebea mscorwks!CrstBase::AcquirePreempLock+0x2e
0573d124 79eb4132 0573d154 00000000 0573d1b4 mscorwks!PEImage::OpenImage+0x92
0573d1c0 79eb3d78 053ee598 0573d1f8 0573d1f4 mscorwks!RuntimeOpenImageInternal+0xcd
0573d208 79eb3cee 053ee598 79eaebb0 00000000 mscorwks!GetAssemblyMDInternalImportEx+0x9d
0573d21c 79ecf4b9 053ee598 00000000 0573d238 mscorwks!CreateMetaDataImport+0x16
0573d23c 79ecf462 053ee598 5ed280a6 00000000 mscorwks!CAssemblyManifestImport::Init+0x35
0573d264 79f2e637 053ee598 0573d27c 000f2a20 mscorwks!CreateAssemblyManifestImport+0x53
0573d280 7a161a1d 053ee598 03d09c00 0573d9a8 mscorwks!CreateAssemblyFromManifestFile+0x48

0:081> !showstringw 053ee598
C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\_mem_bin\bf10f877\41cee659\assembly\dl3\9cf10f5d\0090cb26_8229c601\DescMembers.DLL

0:081> !showstringw 0573dcc4
D:\MyApp\App\Members\bin\DescMembers.dll

Another set of dumps helped to further clarify the situation: thread 94 holds the lock and is trying to load a previously compiled assembly from disk. It is trying to load this in relation to page D:\MyApp\App\Secure\Root\default.aspx and this this is leading to the attempt to load the assembly for global.asax: C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\App_global.asax.ylfot7zw.dll. As part of this assembly load, thread 94 is blocked waiting to acquire a critical section owned by thread 123:

0:094> !syncblk
Index SyncBlock MonitorHeld Recursion Owning Thread Info  SyncBlock Owner
140 03f05a84           13         1 03ebd208   714  94   1225a688 System.Web.Compilation.BuildManager
-----------------------------
Total           219
CCW             11
RCW             0
ComClassFactory 0
Free            0

0:094> !dso
ESP/REG  Object   Name
04f7e9e0 790d6654 System.String
04f7eb70 1c571e2c System.Byte[]
04f7ecb8 1c571df0 System.Reflection.AssemblyName
04f7eccc 1c571df0 System.Reflection.AssemblyName
04f7edac 1c571df0 System.Reflection.AssemblyName
04f7edc4 1c571df0 System.Reflection.AssemblyName
04f7edf8 1c571ad0 System.String    C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\App_global.asax.ylfot7zw.dll.delete
04f7ee00 1c47a640 System.Object[]    (System.Reflection.AssemblyName[])
04f7ee04 1c5715e8 System.String    C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\App_global.asax.ylfot7zw.dll
04f7ee24 1c4775e0 System.Reflection.Assembly
04f7ee28 1c468e58 System.String    App_Web_gf9hosur
04f7ee30 121a2004 System.Globalization.CultureInfo
04f7ee34 1c468dbc System.String    110000
04f7ee44 1c1f7e58 System.Web.VirtualPath
04f7ee4c 1c4693bc System.Web.Compilation.BuildResultCompiledTemplateType
04f7ee58 1c1f7e58 System.Web.VirtualPath
04f7ee60 1c4693bc System.Web.Compilation.BuildResultCompiledTemplateType
04f7ee7c 1c468cbc System.String    fffffff8697ef330
04f7ee84 1c428a68 System.String    C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\default.aspx.8cd6fbf4.compiled
04f7ee88 1c428a68 System.String    C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\default.aspx.8cd6fbf4.compiled
04f7ee8c 1c1f7e58 System.Web.VirtualPath
04f7ee9c 1c428a68 System.String    C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\default.aspx.8cd6fbf4.compiled
04f7eea8 1a200b5c System.String    .compiled
04f7eeac 1c428970 System.String    C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\default.aspx.8cd6fbf4
04f7eeb8 1a200b5c System.String    .compiled
04f7eebc 1c428a68 System.String    C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\default.aspx.8cd6fbf4.compiled
04f7eec4 1225b41c System.Web.Compilation.StandardDiskBuildResultCache
04f7eec8 1c1f7e58 System.Web.VirtualPath
04f7eed8 1c428a68 System.String    C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\default.aspx.8cd6fbf4.compiled
04f7eee8 1c1f7e58 System.Web.VirtualPath
04f7eef8 1c1f7e58 System.Web.VirtualPath
04f7ef00 1225a688 System.Web.Compilation.BuildManager
04f7ef08 1c427124 System.String    default.aspx.8cd6fbf4
04f7ef0c 1225a7c4 System.Web.Util.SimpleRecyclingCache
04f7ef20 1c427100 System.String    8cd6fbf4
04f7ef2c 1c1f7e58 System.Web.VirtualPath
04f7ef30 1225a688 System.Web.Compilation.BuildManager
04f7ef44 1c1f7e58 System.Web.VirtualPath
04f7ef50 1225a688 System.Web.Compilation.BuildManager
04f7ef54 1c1f7e58 System.Web.VirtualPath
04f7ef70 1c418fb8 System.Web.ApplicationImpersonationContext
04f7ef74 1c418fb8 System.Web.ApplicationImpersonationContext
04f7ef78 1c418fb8 System.Web.ApplicationImpersonationContext
04f7ef88 1c1f7e58 System.Web.VirtualPath
04f7ef9c 1c418fb8 System.Web.ApplicationImpersonationContext
04f7efa8 1c418fb8 System.Web.ApplicationImpersonationContext
04f7efac 1c1f7e58 System.Web.VirtualPath
04f7efc4 1c418fb8 System.Web.ApplicationImpersonationContext
04f7efcc 1c418f88 System.Object[]    (System.Type[])
04f7efd0 1c418fac System.Web.UI.PageHandlerFactory
04f7efe4 1c1f7e58 System.Web.VirtualPath
04f7efe8 1c1f7c2c System.Web.HttpContext
04f7f000 1c1f7c2c System.Web.HttpContext
04f7f004 1444b800 System.RuntimeType
04f7f024 1239e718 System.String    System.Web.UI.PageHandlerFactory
04f7f02c 1c393abc System.Collections.Hashtable
04f7f034 1c1f73a4 System.String    D:\MyApp\App\Secure\Root\default.aspx
04f7f038 1c1f7c2c System.Web.HttpContext
04f7f03c 1c1f7e58 System.Web.VirtualPath
04f7f050 1c1f7c2c System.Web.HttpContext
04f7f054 1c418fac System.Web.UI.PageHandlerFactory
04f7f058 1c1f7c2c System.Web.HttpContext
04f7f060 1c1f73a4 System.String    D:\MyApp\App\Secure\Root\default.aspx
04f7f064 1c1f7e58 System.Web.VirtualPath
04f7f068 1c1f7338 System.String    POST
04f7f06c 1c418fac System.Web.UI.PageHandlerFactory
04f7f070 1c4187b0 System.Web.ApplicationImpersonationContext
04f7f074 1c1f87b8 ASP.global_asax
04f7f07c 1c1f73a4 System.String    D:\MyApp\App\Secure\Root\default.aspx
04f7f080 1c1f7e58 System.Web.VirtualPath
04f7f084 1c1f7338 System.String    POST
04f7f088 1c4187b0 System.Web.ApplicationImpersonationContext
04f7f090 1c1f7c2c System.Web.HttpContext
04f7f0a4 1c1f87b8 ASP.global_asax
04f7f0a8 1c1f87b8 ASP.global_asax
04f7f0ac 1c1f7cd4 System.Web.HttpRequest
04f7f0b0 1c1f7c2c System.Web.HttpContext
04f7f0c0 1c1f73a4 System.String    D:\MyApp\App\Secure\Root\default.aspx
04f7f0c4 1c1f7e58 System.Web.VirtualPath
04f7f0c8 1c1f7338 System.String    POST
04f7f0cc 1c1f7c2c System.Web.HttpContext
04f7f0d0 1c1f87b8 ASP.global_asax
04f7f0e8 1c411f58 System.Web.HttpApplication+MapHandlerExecutionStep
04f7f0ec 1c1f87b8 ASP.global_asax
04f7f0f0 1c1f7c2c System.Web.HttpContext
04f7f104 1c1f7c2c System.Web.HttpContext
04f7f108 1c1f87b8 ASP.global_asax
04f7f11c 1c1f87b8 ASP.global_asax
04f7f124 1c1f7c2c System.Web.HttpContext
04f7f128 1c1f87b8 ASP.global_asax
04f7f12c 1c412238 System.Web.AspNetSynchronizationContext
04f7f134 1c411cdc System.Web.HttpApplication+ApplicationStepManager
04f7f158 1c1f7c2c System.Web.HttpContext
04f7f15c 1c412218 System.Web.HttpAsyncResult
04f7f160 1c1f87b8 ASP.global_asax
04f7f164 121c489c System.AsyncCallback
04f7f170 1c1f4ff4 System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6
04f7f174 1c1f87b8 ASP.global_asax
04f7f178 1c1f7e6c System.Web.HttpWriter
04f7f180 1c1f7c2c System.Web.HttpContext
04f7f184 121c489c System.AsyncCallback
04f7f188 1c1f7c2c System.Web.HttpContext
04f7f18c 1c1f4ff4 System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6
04f7f190 121c4138 System.Web.HttpRuntime
04f7f1a8 1c1f72dc System.String    D:\MyApp\App\Secure\
04f7f1ac 1c1f4ff4 System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6
04f7f1bc 121a1038 System.String    D:\MyApp\App\Secure\
04f7f1c0 1c1f4ff4 System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6
04f7f1f8 122f2880 System.Web.Hosting.ISAPIRuntime

0:094> !do 1c1f4ff4
Name: System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6
MethodTable: 663a6f84
EEClass: 663a6f04
Size: 256(0x100) bytes
(C:\WINDOWS\assembly\GAC_32\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll)
Fields:
MT    Field   Offset                 Type VT     Attr    Value Name
79105fac  400133a        4      System.DateTime  1 instance 1c1f4ff8 _startTime
7910ead0  400133b        c          System.Guid  1 instance 1c1f5000 _traceId
7912ab5c  400133c      194    System.String[][]  0   shared   static s_HTTPStatusDescriptions
>> Domain:Value  001171e8:NotInit  00159f68:1a227064 03ea3b78:122f6384 <<
79124304  400133d      198      System.Object[]  0   shared   static s_serverVarFromRequestHeaderNames
>> Domain:Value  001171e8:NotInit  00159f68:1a2271ac 03ea3b78:122f64cc <<
79124304  400133e      19c      System.Object[]  0   shared   static s_requestHeaderNames
>> Domain:Value  001171e8:NotInit  00159f68:1a22725c 03ea3b78:122f657c <<
79124304  400133f      1a0      System.Object[]  0   shared   static s_responseHeaderNames
>> Domain:Value  001171e8:NotInit  00159f68:1a22730c 03ea3b78:122f662c <<
790feb48  4001340      1a4 ...ections.Hashtable  0   shared   static s_requestHeadersLoookupTable
>> Domain:Value  001171e8:NotInit  00159f68:1a227394 03ea3b78:122f66b4 <<
790feb48  4001341      1a8 ...ections.Hashtable  0   shared   static s_responseHeadersLoookupTable
>> Domain:Value  001171e8:NotInit  00159f68:1a22745c 03ea3b78:122f677c <<
790fe238  4001b3a       70        System.IntPtr  1 instance  3066598 _ecb
790fe238  4001b3b       74        System.IntPtr  1 instance      2d0 _token
7910ead0  4001b3c       a8          System.Guid  1 instance 1c1f509c _traceId
790fa4b0  4001b3d       1c        System.String  0 instance 1c1f7338 _method
790fa4b0  4001b3e       20        System.String  0 instance 1c1f7354 _path
790fa4b0  4001b3f       24        System.String  0 instance 1c1f7354 _filePath
790fa4b0  4001b40       28        System.String  0 instance 790d6654 _pathInfo
790fa4b0  4001b41       2c        System.String  0 instance 1c1f73a4 _pathTranslated
790fa4b0  4001b42       30        System.String  0 instance 141a4b28 _appPath
790fa4b0  4001b43       34        System.String  0 instance 1c1f72dc _appPathTranslated
[...]

0:094> !do 1c1f73a4
Name: System.String
MethodTable: 790fa4b0
EEClass: 790fa410
Size: 150(0x96) bytes
(C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
String: D:\MyApp\App\Secure\Root\default.aspx
Fields:
MT    Field   Offset                 Type VT     Attr    Value Name
790fedf4  4000096        4         System.Int32  1 instance       67 m_arrayLength
790fedf4  4000097        8         System.Int32  1 instance       66 m_stringLength
790fbfcc  4000098        c          System.Char  1 instance       44 m_firstChar
790fa4b0  4000099       10        System.String  0   shared   static Empty
>> Domain:Value  001171e8:790d6654 00159f68:790d6654 03ea3b78:790d6654 <<
7912474c  400009a       14        System.Char[]  0   shared   static WhitespaceChars
>> Domain:Value  001171e8:141a0624 00159f68:141a77f8 03ea3b78:121a2374 <<

0:094> !do 1c1f87b8
Name: ASP.global_asax
EEClass: 04ff1dd8
Size: 140(0x8c) bytes
(C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\App_global.asax.ylfot7zw.dll)
Fields:
MT    Field   Offset                 Type VT     Attr    Value Name
663a2450  4000dc8        4 ...pApplicationState  0 instance 122e14dc _state
663a2750  4000dc9        8 ...m.Web.HttpContext  0 instance 00000000 _initContext
663a359c  4000dca        c ...b.HttpAsyncResult  0 instance 1c412218 _ar
663a3394  4000dcb       10 ...pModuleCollection  0 instance 1c3a648c _moduleCollection
79118214  4000dcc       14 ...ncipal.IPrincipal  0 instance 1c413d68 _savedPrincipal
79105020  4000dcd       78       System.Boolean  1 instance        1 _restorePrincipal
7a74da98  4000de8       18 ....EventHandlerList  0 instance 1c3f7408 _events
663a2aec  4000de9       1c ...ventHandlersTable  0 instance 1c401980 _asyncEvents
[...] 0:094> !critlist CritSec at 7a38ee88. Owned by thread 123.   Waiting Threads: 26 66 94 

!dso output already shows objects and strings in the same order as they are stored on the stack and executed, so it’s quite easy to spot interesting file paths, URLs etc… and often that is enough to suggest which files are involved, which page or web service you’re accessing and so on; of course you can always use !do <address> to dig into the stack to be 100% sure you’re on the right path.

Using the same commands above we found that thread 123 is trying to explicitly load "D:\MyApp\App\Secure\bin\secure.dll" whilst executing "ASP.logout_aspx"; after transitioning from managed to unmanaged code, this thread then gets blocked waiting for garbage collection to complete (because it needs to disable pre-emptive mode in order to acquire a particular lock).

Finally, the resolution: the customer confirmed the involved assemblies are “resource only” files which they access from everywhere in their quite complex application (basically those were localization assemblies they were explicitly loading when needed to localize their interface, error messages etc…); we strong-named those “sensitive” assemblies and installed them in the GAC: easy solution, when you know which is the problem…

By the way, there are a couple hotfixes available for this situation (see http://support.microsoft.com/kb/936757/en-us and 946644, KB not yet available but you can ask for the fix to CSS), but did not help in this case.

Carlo

Quote of the day:
Tact is the ability to describe others as they see themselves. - Abraham Lincoln
• #### autoConfig=”false” effects

My post on autoConfig triggered and interesting discussion with my colleague Cenk about which is the real effect of this setting; one of his customers asked what to do if they needed to adjust only one or two values and not all the ones discussed in Contention, poor performance, and deadlocks when you make Web service requests from ASP.NET applications, they wanted to be sure they could change only what they needed to fine tune and forget about the rest. The short answer is: you can change only what you need to and leave he rest as is, since we have default values hardcoded to improve performance so you’re safe (provided the values you’re changing make sense in the context, of course).

If we have a look at System.Web.Configuration.ProcessModelSection with Windbg, we can see all the properties we have in <processModel> and _defaultValue corresponds to the defaults we have in machine.config.comments:

0:000> !do 06904868
Name: System.Web.Configuration.ProcessModelSection
MethodTable: 6638dcb0
EEClass: 663a0604
Size: 64(0x40) bytes
(C:\Windows\assembly\GAC_32\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll)
Fields:
MT    Field   Offset                 Type VT     Attr    Value Name
7910be50  400014d       30       System.Boolean  1 instance        0 _bDataToWrite
7910be50  400014e       31       System.Boolean  1 instance        0 _bModified
7910be50  400014f       32       System.Boolean  1 instance        1 _bReadOnly
7910be50  4000150       33       System.Boolean  1 instance        0 _bElementPresent
7910be50  4000151       34       System.Boolean  1 instance        1 _bInited
648eff64  4000152        4 ...ionLockCollection  0 instance 00000000 _lockedAttributesList
648eff64  4000153        8 ...ionLockCollection  0 instance 00000000 _lockedAllExceptAttributesList
648eff64  4000154        c ...ionLockCollection  0 instance 00000000 _lockedElementsList
648eff64  4000155       10 ...ionLockCollection  0 instance 00000000 _lockedAllExceptElementsList
648ea704  4000156       14 ...nfigurationValues  0 instance 069048a8 _values
790fd8c4  4000157       18        System.String  0 instance 00000000 _elementTagName
648f032c  4000158       1c ...lementInformation  0 instance 00000000 _evaluationElement
648eee30  4000159       20 ...onElementProperty  0 instance 068fce20 _elementProperty
648f53d0  400015a       2c         System.Int32  1 instance        0 _fItemLocked
648f0a48  400015b       24 ...ontextInformation  0 instance 00000000 _evalContext
648eda8c  400015c       28 ...nfigurationRecord  0 instance 028bea78 _configRecord
7912d8f8  4000148        8      System.Object[]  0   shared   static s_lockAttributeNames
>> Domain:Value  023d04b0:NotInit  024169d0:068fcd1c <<
79101fe4  4000149        c ...ections.Hashtable  0   shared   static s_propertyBags
>> Domain:Value  023d04b0:NotInit  024169d0:068fcd40 <<
00000000  400014a       10                       0   shared   static s_perTypeValidators
>> Domain:Value  023d04b0:NotInit  024169d0:00000000 <<
790fd0f0  400014b       14        System.Object  0   shared   static s_nullPropertyValue
>> Domain:Value  023d04b0:NotInit  024169d0:068fce08 <<
648eee30  400014c       18 ...onElementProperty  0   shared   static s_ElementProperty
>> Domain:Value  023d04b0:NotInit  024169d0:068fce20 <<
648eee84  400015d       38 ...ectionInformation  0 instance 069049b8 _section
648eee30  4001928      74c ...onElementProperty  0   shared   static s_elemProperty
>> Domain:Value  023d04b0:NotInit  024169d0:06903da8 <<
7911228c  4001929      748      System.TimeSpan  1   shared   static DefaultClientConnectedCheck
>> Domain:Value  023d04b0:NotInit  024169d0:06903850 <<
648ef118  400192a      750 ...ropertyCollection  0   shared   static _properties
>> Domain:Value  023d04b0:NotInit  024169d0:06904714 <<
648ef05c  400192b      754 ...igurationProperty  0   shared   static _propEnable
>> Domain:Value  023d04b0:NotInit  024169d0:06903db4 <<
648ef05c  400192c      758 ...igurationProperty  0   shared   static _propTimeout
>> Domain:Value  023d04b0:NotInit  024169d0:06903df4 <<
648ef05c  400192d      75c ...igurationProperty  0   shared   static _propIdleTimeout
>> Domain:Value  023d04b0:NotInit  024169d0:06903e38 <<
648ef05c  400192e      760 ...igurationProperty  0   shared   static _propShutdownTimeout
>> Domain:Value  023d04b0:NotInit  024169d0:06903e7c <<
648ef05c  400192f      764 ...igurationProperty  0   shared   static _propRequestLimit
>> Domain:Value  023d04b0:NotInit  024169d0:06903ed8 <<
648ef05c  4001930      768 ...igurationProperty  0   shared   static _propRequestQueueLimit
>> Domain:Value  023d04b0:NotInit  024169d0:06903f74 <<
648ef05c  4001931      76c ...igurationProperty  0   shared   static _propRestartQueueLimit
>> Domain:Value  023d04b0:NotInit  024169d0:06903ff8 <<
648ef05c  4001932      770 ...igurationProperty  0   shared   static _propMemoryLimit
>> Domain:Value  023d04b0:NotInit  024169d0:06904064 <<
648ef05c  4001933      774 ...igurationProperty  0   shared   static _propWebGarden
>> Domain:Value  023d04b0:NotInit  024169d0:069040a4 <<
648ef05c  4001934      778 ...igurationProperty  0   shared   static _propCpuMask
>> Domain:Value  023d04b0:NotInit  024169d0:069040e4 <<
648ef05c  4001935      77c ...igurationProperty  0   shared   static _propUserName
>> Domain:Value  023d04b0:NotInit  024169d0:06904118 <<
648ef05c  4001936      780 ...igurationProperty  0   shared   static _propPassword
>> Domain:Value  023d04b0:NotInit  024169d0:0690414c <<
648ef05c  4001937      784 ...igurationProperty  0   shared   static _propLogLevel
>> Domain:Value  023d04b0:NotInit  024169d0:06904180 <<
648ef05c  4001938      788 ...igurationProperty  0   shared   static _propClientConnectedCheck
>> Domain:Value  023d04b0:NotInit  024169d0:069041d4 <<
648ef05c  4001939      78c ...igurationProperty  0   shared   static _propComAuthenticationLevel
>> Domain:Value  023d04b0:NotInit  024169d0:06904218 <<
648ef05c  400193a      790 ...igurationProperty  0   shared   static _propComImpersonationLevel
>> Domain:Value  023d04b0:NotInit  024169d0:0690426c <<
648ef05c  400193b      794 ...igurationProperty  0   shared   static _propResponseDeadlockInterval
>> Domain:Value  023d04b0:NotInit  024169d0:069042c0 <<
648ef05c  400193c      798 ...igurationProperty  0   shared   static _propResponseRestartDeadlockInterval
>> Domain:Value  023d04b0:NotInit  024169d0:06904304 <<
648ef05c  400193d      79c ...igurationProperty  0   shared   static _propAutoConfig
>> Domain:Value  023d04b0:NotInit  024169d0:06904348 <<
648ef05c  400193e      7a0 ...igurationProperty  0   shared   static _propMaxWorkerThreads
>> Domain:Value  023d04b0:NotInit  024169d0:069043ac <<
648ef05c  400193f      7a4 ...igurationProperty  0   shared   static _propMaxIOThreads
>> Domain:Value  023d04b0:NotInit  024169d0:0690443c <<
648ef05c  4001940      7a8 ...igurationProperty  0   shared   static _propMinWorkerThreads
>> Domain:Value  023d04b0:NotInit  024169d0:069044cc <<
648ef05c  4001941      7ac ...igurationProperty  0   shared   static _propMinIOThreads
>> Domain:Value  023d04b0:NotInit  024169d0:0690455c <<
648ef05c  4001942      7b0 ...igurationProperty  0   shared   static _propServerErrorMessageFile
>> Domain:Value  023d04b0:NotInit  024169d0:069045c8 <<
648ef05c  4001943      7b4 ...igurationProperty  0   shared   static _propPingFrequency
>> Domain:Value  023d04b0:NotInit  024169d0:069045fc <<
648ef05c  4001944      7b8 ...igurationProperty  0   shared   static _propPingTimeout
>> Domain:Value  023d04b0:NotInit  024169d0:06904640 <<
648ef05c  4001945      7bc ...igurationProperty  0   shared   static _propMaxAppDomains
>> Domain:Value  023d04b0:NotInit  024169d0:069046a8 <<
79102290  4001946      938         System.Int32  1   shared   static cpuCount
>> Domain:Value  023d04b0:NotInit  024169d0:2 <<

0:000> !do 0690443c
Name: System.Configuration.ConfigurationProperty
MethodTable: 648ef05c
EEClass: 648eefec
Size: 52(0x34) bytes
(C:\Windows\assembly\GAC_MSIL\System.Configuration\2.0.0.0__b03f5f7f11d50a3a\System.Configuration.dll)
Fields:
MT    Field   Offset                 Type VT     Attr    Value Name
790fd8c4  4000277        4        System.String  0 instance 06903c08 _name
790fd8c4  4000278        8        System.String  0 instance 06903c08 _providedName
790fd8c4  4000279        c        System.String  0 instance 00000000 _description
79106894  400027a       10          System.Type  0 instance 068b7554 _type
790fd0f0  400027b       14        System.Object  0 instance 06904430 _defaultValue
7a7588dc  400027c       18 ...del.TypeConverter  0 instance 00000000 _converter
648ef1a0  400027d       2c         System.Int32  1 instance        0 _options
648f022c  400027e       1c ...tionValidatorBase  0 instance 06904418 _validator
790fd8c4  400027f       20        System.String  0 instance 00000000 _addElementName
790fd8c4  4000280       24        System.String  0 instance 00000000 _removeElementName
790fd8c4  4000281       28        System.String  0 instance 00000000 _clearElementName
648f022c  4000274       68 ...tionValidatorBase  0   shared   static NonEmptyStringValidator
>> Domain:Value  023d04b0:NotInit  024169d0:068fcc00 <<
648f022c  4000275       6c ...tionValidatorBase  0   shared   static DefaultValidatorInstance
>> Domain:Value  023d04b0:NotInit  024169d0:068fcc14 <<
790fd8c4  4000276       70        System.String  0   shared   static DefaultCollectionPropertyName
>> Domain:Value  023d04b0:NotInit  024169d0:068fcbec <<

0:000> !do 06904430
Name: System.Int32
MethodTable: 79102290
EEClass: 79102218
Size: 12(0xc) bytes
(C:\Windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
Fields:
MT    Field   Offset                 Type VT     Attr    Value Name
79102290  40003e9        4         System.Int32  1 instance       20 m_value
<processModel
enable = "true" [true|false]
timeout = "10675199.02:48:05.4775807" [HH:MM:SS]
idleTimeout = "10675199.02:48:05.4775807" [HH:MM:SS]
shutdownTimeout = "00:00:05" [HH:MM:SS]
requestLimit = "2147483647" [number]
requestQueueLimit = "5000" [number]
restartQueueLimit = "10" [number]
memoryLimit = "60" [number]
webGarden = "false" [true|false]
logLevel = "Errors" [None | All | Errors]
clientConnectedCheck = "00:00:05" [HH:MM:SS]
comAuthenticationLevel = "Connect" [None | Call | Connect | Default | Pkt | PktIntegrity | PktPrivacy]
comImpersonationLevel = "Impersonate" [Default | Anonymous | Delegate | Identify | Impersonate]
autoConfig = "false" [true|false]
serverErrorMessageFile = "" [String]
pingFrequency = "10675199.02:48:05.4775807" [HH:MM:SS]
pingTimeout = "10675199.02:48:05.4775807" [HH:MM:SS]
maxAppDomains = "2000" [number]
/>

Internally to debug managed code we use a Windbg extension which works well with private symbols and has some additional commands the public sos.dll does not have and one of those shows the ThreadPool configuration, very useful to have a quick report of those configuration values.

After the discussion I found myself thinking how to extract those values from a dump using only public symbols and the public sos extension everyone has access to; two of them are quite easy to find out (you can get a full list of members of ThreadpoolMgr class running mscorwks!ThreadpoolMgr*):

0:000> ?poi(mscorwks!ThreadpoolMgr::NumberOfProcessors)
Evaluate expression: 2 = 00000002
Evaluate expression: 200 = 000000c8

Name: System.Web.RequestQueue
MethodTable: 663aebf0
EEClass: 663aeb80
Size: 64(0x40) bytes
(C:\Windows\assembly\GAC_32\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll)
Fields:
MT    Field   Offset                 Type VT     Attr    Value Name
79102290  4001203       14         System.Int32  1 instance      176 _minExternFreeThreads
79102290  4001204       18         System.Int32  1 instance      152 _minLocalFreeThreads
79102290  4001205       1c         System.Int32  1 instance     5000 _queueLimit
7911228c  4001206       2c      System.TimeSpan  1 instance 0691da2c _clientConnectedTime
7910be50  4001207       28       System.Boolean  1 instance        1 _iis6
791087cc  4001208        4 ...Collections.Queue  0 instance 0691da40 _localQueue
791087cc  4001209        8 ...Collections.Queue  0 instance 0691daf4 _externQueue
79102290  400120a       20         System.Int32  1 instance        0 _count
791186fc  400120b        c ...ding.WaitCallback  0 instance 0691dba8 _workItemCallback
79102290  400120c       24         System.Int32  1 instance        0 _workItemCount
7910be50  400120d       29       System.Boolean  1 instance        0 _draining
7911228c  400120e       34      System.TimeSpan  1 instance 0691da34 _timerPeriod
791127fc  400120f       10 ...m.Threading.Timer  0 instance 0691dbe8 _timer

I was trying to find a convenient way to get the remaining three values (autoConfig, maxConnection and Number of timers) when I found this post from Tom, and now my quest is over: !DumpThreadConfig is the same command we have in the internal Windbg extension I mentioned and also the output is the same… so apparently you just have to be patient and wait for the next Debugging Tools release

Carlo

Quote of the day:
There is nothing new under the sun but there are lots of old things we don't know. - Ambrose Bierce
• #### Ups and downs of ASP.NET Performance Counters

Perfmon is likely the first tool you think to when it comes to monitor some internals and performance of your application and it’s relatively easy to find out information and resources on the Internet; unfortunately this seems to also be a fragile component and every now and then we receive new calls about it: event log spammed with warnings about missing or corrupted counters, values returned are inconsistent or clearly wrong etc…

Messages like the following can appear after for example installing the .NET Framework 2.0 on your machine:

The Open Procedure for service "ASP.NET" in DLL "C:\Windows\Microsoft.NET\Framework\v2.0.50727\aspnet_perf.dll" failed. Performance data for this service will not be available. Status code returned is data DWORD 0

To note that such warnings are logged even if your application is not running.

### Some troubleshooting

First of all you can try to disable any third party services and processes (for example using msconfig.exe or Autoruns) and see if the problem still reproduces. If it does, you can use How to manually rebuild Performance Counter Library values to try to fix it manually, or try the following:

• Uninstall the  .NET Framework 2.0
• If available for Perfc009.dat, Perfh009.dat and Perfh007.dat , Perfh007.dat, backup the registry keys
• HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services
• HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\Current Version\Perflib
• Open a command prompt to your System32 folder
• REN perf?009.dat *.bak
• Expand <CD_drive_letter:>\i386\perfc009.DA_ %systemroot%\system32\perfc009.DAT
• Expand <CD_drive_letter:>\i386\perfd009.DA_ %systemroot%\system32\perfd009.DAT
• Expand <CD_drive_letter:>\i386\perfh009.DA_ %systemroot%\system32\perfh009.DAT
• Expand <CD_drive_letter:>\i386\perfi009.DA_ %systemroot%\system32\perfi009.DAT
• REN perf?007.dat *.bak
• Expand <CD_drive_letter:>\i386\perfc007.DA_ %systemroot%\system32\perfc007.DAT
• Expand <CD_drive_letter:>\i386\perfd007.DA_ %systemroot%\system32\perfd007.DAT
• Expand <CD_drive_letter:>\i386\perfh007.DA_ %systemroot%\system32\perfh007.DAT
• Expand <CD_drive_letter:>\i386\perfi007.DA_ %systemroot%\system32\perfi007.DAT
• Open your system registry (be very careful!) to HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Perflib and set
• Last Counter = 1846 (decimal) or 736 (hex)
• Last Help = 1847 (decimal) or 737 (hex)
• Now move to HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services
• Search for the subkey Performance  in services and set the values
• First Counter = 0
• First Help = 0
• Last Counter = 0
• Last Help = 0
• Back to the command prompt (still in System32)
• findstr drivername *.ini > Perfini.txt
• open  Perfini.txt  and note  drivername=nnn

================  sample  Perfini.txt :==================

C:\WINDOWS\system32\axperf.ini:drivername=ASP
C:\WINDOWS\system32\esentprf.ini:drivername=ESENT
C:\WINDOWS\system32\ftpctrs.ini:drivername=MSFTPSVC
C:\WINDOWS\system32\iasperf.ini:drivername=IAS
C:\WINDOWS\system32\infoctrs.ini:drivername=inetinfo
C:\WINDOWS\system32\mqperf.ini:drivername=MSMQ
C:\WINDOWS\system32\msdtcprf.ini:drivername=MSDTC
C:\WINDOWS\system32\ntdsctrs.ini:drivername=NTDS
C:\WINDOWS\system32\ntfrscon.ini:drivername=FileReplicaConn
C:\WINDOWS\system32\ntfrsrep.ini:drivername=FileReplicaSet
C:\WINDOWS\system32\perfci.ini:drivername=ContentIndex
C:\WINDOWS\system32\perffilt.ini:drivername=ContentFilter
C:\WINDOWS\system32\perfwci.ini:drivername=ISAPISearch
C:\WINDOWS\system32\pschdprf.ini:drivername=PSched
C:\WINDOWS\system32\rasctrs.ini:drivername=RemoteAccess
C:\WINDOWS\system32\tslabels.ini:drivername=TermService
C:\WINDOWS\system32\w3ctrs.ini:drivername=W3SVC

================  Sample Perfini.txt :==================

unlodctr <drivername>
lodctr <ini file>
<drivername>  driver's name from  Perfini.txt
<ini file>  listed  “.ini” files  in  Perfini.txt

Example:  ASP Drivername: axperf.ini:drivername=ASP

unlodctr ASP
lodctr axperf.ini

Or using the following command you can transfer settings from a  working machine to a not working one:

lodctr /s:Perf_all.ini
lodctr Perf_all.ini
Reboot
Check Perfdiag

### Disable not needed counters (as a last resource)

If nothing else works as you wish or you do not want to waste time troubleshooting and just need the machine up and running without much annoyances you can disable the counters for the problematic service(s), of course at the cost of not having those data if you’ll ever need them.

For this task you can use Exctrlst.exe utility from the OS Resource Kit (see Exctrlst Overview and Exctrlst UI); it disables the specific counter by adding a registry key similar to the example below. You can add these registry entries under the specific counter entry manually if Exctrlst.exe is not available or desirable

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Spooler\Performance
Value name: Disable Performance Counters
Value type: REG_DWORD
Value data: 1

### Some How To and general articles

Carlo

Quote of the day:
The only time people dislike gossip is when you gossip about them. - Will Rogers