This is an interesting question I have to admit I had not thought to until we got a case from a customer whom was in trouble with his production servers: how many application pools can you run on IIS? Event better, how many w3wp.exe instances can the OS cope with? As quite often happens in our job, the correct answer is: it depends…
First, the problem. The customer was a hosting company so they had a large numbers of sites on each web server, around 450, each of them with its own application pool, each application pool run under its own specific account and aspExecuteInMTA was set to 1 on each server. Under some circumstances asp applications from some web sites are failing with the following message (roughly translated from Spanish):
Server error 'ASP 0177 : 80070057' Error in Server.CreateObject /site/page.asp, line 338 80070057
When the problem occurred the customer recycled the two or three top consuming web sites in terms of memory, threads and handles; then the failing web site was usually back online. Note that the failing web site was always only affected by other problematic ones, it was almost never the culprit.
At first we were distracted by the fact that the failing web sites where using Access as the database and we also got some errors like:
Server error 'ASP 0177 : 80070583' Error in Server.CreateObject /database.asp, line 9 -- "Provider=Microsoft.Jet.OLEDB.4.0;Data source=" & server.MapPath("../data/bd.mdb") 80070583
This is a deprecated configuration, see Using Microsoft JET with IIS:
The Access ODBC Driver and Microsoft OLE DB Provider for Microsoft Jet are not intended to be used with high-stress, high-concurrency, 24x7 server applications, such as web, commerce, transactional, messaging servers, and so on.
Anyway we soon found that was not the right direction. 450 application pools are a lot, plus there was the thing about recycling the most handles and threads consuming pools… this suggests desktop heap exhaustion.
Desktop Heap overview has a good explanation of what Desktop Heap is and how things might start to go wrong when the system runs low in Desktop Heap storage; it also has a few interesting bits about IIS:
Regarding non-ASP.NET applications, IIS 6.0 has been tested on a well-configured mainframe server running up to 2,000 concurrent worker processes, each serving one application pool, but not using unique identities. In practice, a design of up to 500 simultaneous application pools is achievable, depending on the application requirements and assuming hardware resources are not a significant constraint. It is important to configure the UseSharedWPDesktop registry setting mentioned above when using more than 60 application pools with unique identities
My colleague Nicolas also has a post about Desktop Heap and Windows Stations here. Having one shared desktop can theoretically bring some security issues, but David Wang explains well this scenario:
"we should use a common desktop when the number of application pools with unique identities configured on a server is greater than 60". In other words, IIS does not have a problem running >60 application pools which uses <60 unique identities. The limitation in question is the number of user desktops associated with each unique application pool identity that Windows simultaneously supports. The following are the interesting combinations and consequences: If UseSharedWPDesktop=0 and all application pools launch as Network Service, then there is exactly one Desktop (of Network Service) shared by all the application pools. You have no isolation of desktop nor process identity between application pools. If UseSharedWPDesktop=0 and all application pools launch as unique user identity, then there is a unique Desktop for each user identity, which is obviously not shared. This is limited to a number around 60 due to Desktop heap limit in Windows Server 2003. This means that you can only have ~60 application pools perfectly isolated by desktop and process identity simultaneously running. If UseSharedWPDesktop=1 and all application pools launch as unique user identity, then there is one Desktop that is shared by all the user identities. This removes the limit of 60 at the cost of not perfectly isolated application pools. You have no desktop isolation but retain process identity isolation between application pools. UseSharedWPDesktop=1 and all application pools launch as Network Service is pretty much a degenerate case that is not interesting. :-)
"we should use a common desktop when the number of application pools with unique identities configured on a server is greater than 60". In other words, IIS does not have a problem running >60 application pools which uses <60 unique identities. The limitation in question is the number of user desktops associated with each unique application pool identity that Windows simultaneously supports.
The following are the interesting combinations and consequences:
The customer was already aware of those blog posts so they were already using UseSharedWPDesktop=1 (otherwise they were not able to run more than 50 concurrent application pools), set sessionViewSize to 64 Mb and the default value for “HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\SubSystems\Windows”.
To know more about Desktop Heap status on the server we used Desktop Heap Monitor: this comes with a comprehensive usage guide, anyway here are some quick steps to use it:
To automate the process of gathering Desktop Heap information, create a batch file called HeapMon.bat containing the following:
@echo off rem vars C B A are day month year in the for line below, change them to change date for /F "tokens=2,3,4 delims=/- " %%A in ('date /T') do set DT=%%C_%%B_%%A for /F "tokens=1,2 delims=/: " %%A in ('time /T') do set TM=%%A-%%B echo %DT% echo %TM% Dheapmon -v > dheapmon_out_%DT%_%TM%.txt
This will create an output file uniquely named with the date and time included; you can use task scheduler to run this batch file at regular intervals, e.g. once per hour.
Once finished all diagnostic activity with DHeapMon, unload the device driver with:
DHEAPMON -u
To uninstall dheapmon, run:
DHEAPMON -r
From the report I got is was clear that the customer was really using only 1 desktop within IIS WinSTA (remember he was using a shared desktop across his application pools); as IIS WinSTA is created as non interactive, this single desktop is 512 Kb but it was exhausted at 99.2% when then problem appeared:
Windowstation: (WP_WINSTA-054f77ac-6ece-4080-b3a2-b0bf08dacaab) SessionID: 0 Desktop: (Default) Addr: bbc30000 Desktop Heap 524288 (0x 80000) Bytes Committed 520192 (0x 7f000) Bytes Uncommitted 4096 (0x 1000) Bytes Allocated 520072 (0x 7ef88) Bytes Total Freed 120 (0x 78) Bytes Unused 4216 (0x 1078) Bytes Used Rate ( 99.2) %
To calculate SessionViewState size we should do something like:
312Kb (Winsta0/Default) + 64Kb (Winsta0/Disconnect) + 128Kb (Winsta0/Winlogon) + 7 * 512Kb (all other non interactive winsta) + 512Kb (IIS non interactive winsta WP_WINSTA-xxx) == 4600 Kb (< 5 Mb)
So apparently the problem is not on SessionViewSize bur rather just on the non interactive Desktop Heap size; moreover setting this value to 64 Mb as the customer did is not necessary because they were using a bit less than 4 Mb for that value so also the default (20 Mb) were far enough.
The problem of increasing this one too much is that not only IIS owns one, so we first tried using 1024 Kb:
312 + 64 + 128 + 7 * 1024 = 7672 Kb (< 7 Mb)
So we tried with the following (not the customer was not using /3gb):
This first try allowed the server to run fine a bit longer than before but at the end the problem was there again, and another heapmon trace showed we were still exhausting Desktop Heap… we gave another try with SharedSection=1024,3072,2048 (312 + 64 + 128 + 7 * 1024 = 14840 Kb (< 15 Mb).
Bingo, this time we did it!
Of course be very careful when you manipulate the Desktop Heap configuration and if you’re not sure how to do (or if you’re doing it right, or if you need help to fine tune your settings etc…) give Microsoft Support a call!
To close with some general recommendations:
The point is we will always have inherent x32 bit limitations that we cannot workaround and 450 application pools is a pretty close “practically tested” limit…
Carlo
Microsoft Visual Studio 2008 Service Pack 1 (exe)
This download installs Visual Studio 2008 Service Pack 1 (SP1) and the .NET Framework 3.5 SP1. SP1 addresses issues that were found through a combination of customer and partner feedback, as well as internal testing. These service packs offer Visual Studio and .NET Framework users improvements in responsiveness, stability and performance. Click here for more information regarding about these service packs.
Microsoft .NET Framework 3.5 Service Pack 1
.NET Framework version 3.5 Service Pack 1 provides the following new features and improvements: ASP.NET Dynamic Data, which provides a rich scaffolding framework that enables rapid data driven development without writing code, and a new addition to ASP.NET AJAX that provides support for managing browser history (back button support). For more information, see What’s New in ASP.NET and Web Development. Core improvements to the CLR (common language runtime) that include better layout of .NET Framework native images, opting out of strong-name verification for fully trusted assemblies, improved application startup performance, better generated code that improves end-to-end application execution time, and opting managed code to run in ASLR (Address Space Layout Randomization) mode if supported by the operating system. Additionally, managed applications that are opened from network shares have the same behavior as native applications by running with full trust. Performance improvements to WPF (Windows Presentation Foundation), including a faster startup time and improved performance for Bitmap effects. Additional functionality for WPF includes better support for line of business applications, native splash screen support, DirectX pixel shader support, and the new WebBrowser control. ClickOnce application publishers can decide to opt out of signing and hashing as appropriate for their scenarios, developers can programmatically install ClickOnce applications that display a customized branding, and ClickOnce error dialog boxes support links to application-specific support sites on the Web. The Entity Framework is an evolution of the existing suite of ADO.NET data access technologies. The Entity Framework enables developers to program against relational databases in according to application-specific domain models instead of the underlying database models. For more information, see Getting Started with the Entity Framework. The Entity Framework introduces some additional features, including support for new SQL Server 2008 types, default graph serialization of Entities, and the Entity Data Source. This release of the Entity Framework supports the new date and file stream capabilities in SQL Server 2008. The graph serialization work helps developers who want to build Windows Communication Foundation (WCF) services that model full graphs as data contracts. The Entity Data Source provides a traditional data source experience for ASP.NET Web application builders who want to work with the Entity Framework. LINQ to SQL includes new support for the new date and file stream capabilities in SQL Server 2008. The ADO.NET Data Services Framework consists of a combination of patterns and libraries, which enable data to be exposed as a flexible REST (Representational State Transfer)-based data service that can be consumed by Web clients in a corporate network or across the Internet. The ADO.NET Data Services Framework makes data service creation over any data source. A conceptual view model of the underlying storage schema can easily be exposed through rich integration with the ADO.NET Entity Framework. Services created by using the ADO.NET Data Services Framework, and also compatible Windows Live (dev.live.com) services, can be easily accessed from any platform. For client applications that are running on Microsoft platforms, a set of client libraries are provided to make interaction with data services simple. For example, .NET Framework-based clients can use LINQ to query data services and a simple .NET Framework object layer to update data in the service. Windows Communication Foundation now makes the DataContract Serializer easier to use by providing improved interoperability support, enhancing the debugging experience in partial trust scenarios, and extending syndication protocol support for wider usage in Web 2.0 applications. The .NET Framework Data Provider for SQL Server (SqlClient) adds new support for file stream and sparse column capabilities in SQL Server 2008.
.NET Framework version 3.5 Service Pack 1 provides the following new features and improvements:
Here is the link to download the full package: Microsoft .NET Framework 3.5 Service Pack 1
A picture worth 1000 words and this is true also when you have to analyze IIS logs to get some statistics on your site; as you already know, LogParser is extremely flexible and very useful in such situation and its chart output format is what it’s needed.
The first thing you need is a copy of Office Web Components installed on the machine; if you don’t, LogParser will complain and return this error:
Error creating output format "chart": This output format requires a licensed Microsoft Office Chart Web Component to be installed on the local machine
From LogParser help:
chartType Values: name of chart type Default: Line Description: Chart type Details: The set of available chart types depends on the version of the Microsoft Office Web Components installed on the local computer. For a list of the available chart types, type the following help command from the command-line shell: LogParser -h -o:CHART
Currently the latest Office Web Components version available (the newest I’ve been able to find on the Internet) is version 11 which has been released for Office 2003.
Changes in the 2007 Office System OWC11 Description: OWC11 is removed from the 2007 Office system and now ships only with Microsoft Office Project 2007. The DataFinder functionality in OWC11 is also removed. This affects solutions that use the spreadsheet, chart, pivot table, and data source control functionality of OWC11 in a Web page or client program that requires installation of this ActiveX control on the user's computer. The user experience varies depending upon the host container of the OWC11 ActiveX control. For Web pages, users might see a broken icon or a notification that OWC is required. For client programs, users might receive a runtime error or other notification. In these situations, users download OWC11 from the Microsoft Download Center. The notification message generated by Office programs provides a link to the download location for OWC11. Neither Microsoft Office Excel 2007 nor SharePoint Designer 2007 now generate Web pages that use OWC. Reason for change: New technologies such as Windows SharePoint Services 3.0 with Excel server side capabilities and the new charting component in the 2007 Office system replace some features of OWC11.
And if you have Office 2007 like me? I installed OWC11 anyway and everything is still working fine so you should be fine; just in case you need the download links, here they are: Office 2003 Add-in: Office Web Components and Microsoft Office 2003 Web Components Service Pack 1 (SP1) for the 2007 Microsoft Office System.
Bytes per data type as exploded pie
LogParser "SELECT TO_UPPERCASE(EXTRACT_EXTENSION(cs-uri-stem)) AS PageType, MUL(PROPSUM(sc-bytes),100.0) AS Bytes INTO Pie.gif FROM *.log GROUP BY PageType ORDER BY Bytes DESC" -chartType:PieExploded -chartTitle:"Bytes per data type" -categories:off -o:chart -i:iisw3c
Calculating percentages per data type
logparser "select extract_extension(cs-uri-stem) as Resource, mul(propcount(*),100.0) as ResourceHits into percentage.jpg from *.log group by Resource order by ResourceHits desc" -chartType:PieExploded3D -chartTitle:"Percentages per data type" -categories:off -o:chart -i:iisw3c
.aspx page hits per minute
logparser "select quantize(time, 60) as TimeGenerated, count(*) as Hits into qnt.gif from *.log where to_lowercase(extract_extension(cs-uri-stem))='aspx' group by TimeGenerated" -i:iisw3c -o:chart -chartType:Line
Getting the number of unique visitors (actually the number of unique client IP addresses) is a bit more complex. We could try the following:
select date, count(distinct c-ip) into UniqueIPs.gif from *.log group by date" -i:iisw3c -o:chart -chartType:Line
but we get this error:
Error: Semantic Error: aggregate functions with DISTINCT arguments are not supported with GROUP BY clauses
The solution in this case is to split the query into two separate commands; for example you could create a .bat file (or script, or whatever method you prefer) and run the following:
@echo off logparser "select distinct date, c-ip into dates.txt from *.log" -i:iisw3c -o:w3c logparser "select date, count(c-ip) as UniqueIPs into UniqueIPs.gif from dates.txt group by date order by date" -i:iisw3c -o:chart -chartType:SmoothLine -view:on
Using an external script file (JScript or VBScript) it is possible to have a deeper control over the produced chart; this is based on two global objects which expose methods and properties that can be used to modify parameters such as the chart colors, the chart fonts, and many other attributes. Those two global objects are instances of the chartSpace and chart objects of the Microsoft Office Web Components ChartSpace object model, and they are named "chartSpace" and "chart"; for more information on those objects you can have a look at the MSDN ChartSpace Object Model documentation.
Taken from the LogParser documentation, we can use the following sample script to add a caption to the previous UniqueIPs chant and make the background color transparent:
// Add a caption chartSpace.HasChartSpaceTitle = true; chartSpace.ChartSpaceTitle.Caption = "Generated by Log Parser 2.2"; chartSpace.ChartSpaceTitle.Font.Size = 8; chartSpace.ChartSpaceTitle.Position = chartSpace.Constants.chTitlePositionBottom; // Change the background color chart.PlotArea.Interior.Color = chartSpace.Constants.chColorNone;
@echo off logparser "select distinct date, c-ip into dates.txt from *.log" -i:iisw3c -o:w3c logparser "select date, count(c-ip) as UniqueIPs into UniqueIPs.gif from dates.txt group by date order by date" -i:iisw3c -o:chart -chartType:SmoothLine -config:MyScript.js -groupSize: 800x600 -view:on
From here you can experiment using the ChartSpace object model to further customize the appearance of your charts: the limit is your imagination