How many application pools can you cope with?

This is an interesting question I have to admit I had not thought to until we got a case from a customer whom was in trouble with his production servers: how many application pools can you run on IIS? Event better, how many w3wp.exe instances can the OS cope with? As quite often happens in our job, the correct answer is: it depends…

First, the problem. The customer was a hosting company so they had a large numbers of sites on each web server, around 450, each of them with its own application pool, each application pool run under its own specific account and aspExecuteInMTA was set to 1 on each server. Under some circumstances asp applications from some web sites are failing with the following message (roughly translated from Spanish):

Server error 'ASP 0177 : 80070057'
Error in Server.CreateObject
/site/page.asp, line 338
80070057

When the problem occurred the customer recycled the two or three top consuming web sites in terms of memory, threads and handles; then the failing web site was usually back online. Note that the failing web site was always only affected by other problematic ones, it was almost never the culprit.

At first we were distracted by the fact that the failing web sites where using Access as the database and we also got some errors like:

Server error 'ASP 0177 : 80070583'
Error in Server.CreateObject
/database.asp, line 9 -- "Provider=Microsoft.Jet.OLEDB.4.0;Data source=" & server.MapPath("../data/bd.mdb")
80070583

This is a deprecated configuration, see Using Microsoft JET with IIS:

The Access ODBC Driver and Microsoft OLE DB Provider for Microsoft Jet are not intended to be used with high-stress, high-concurrency, 24x7 server applications, such as web, commerce, transactional, messaging servers, and so on.

Anyway we soon found that was not the right direction. 450 application pools are a lot, plus there was the thing about recycling the most handles and threads consuming pools… this suggests desktop heap exhaustion.

A bit of theory

Desktop Heap overview has a good explanation of what Desktop Heap is and how things might start to go wrong when the system runs low in Desktop Heap storage; it also has a few interesting bits about IIS:

Regarding non-ASP.NET applications, IIS 6.0 has been tested on a well-configured mainframe server running up to 2,000 concurrent worker processes, each serving one application pool, but not using unique identities. In practice, a design of up to 500 simultaneous application pools is achievable, depending on the application requirements and assuming hardware resources are not a significant constraint. It is important to configure the UseSharedWPDesktop registry setting mentioned above when using more than 60 application pools with unique identities

My colleague Nicolas also has a post about Desktop Heap and Windows Stations here. Having one shared desktop can theoretically bring some security issues, but David Wang explains well this scenario:

"we should use a common desktop when the number of application pools with unique identities configured on a server is greater than 60". In other words, IIS does not have a problem running >60 application pools which uses <60 unique identities. The limitation in question is the number of user desktops associated with each unique application pool identity that Windows simultaneously supports.

The following are the interesting combinations and consequences:

1. If UseSharedWPDesktop=0 and all application pools launch as Network Service, then there is exactly one Desktop (of Network Service) shared by all the application pools. You have no isolation of desktop nor process identity between application pools.
2. If UseSharedWPDesktop=0 and all application pools launch as unique user identity, then there is a unique Desktop for each user identity, which is obviously not shared. This is limited to a number around 60 due to Desktop heap limit in Windows Server 2003. This means that you can only have ~60 application pools perfectly isolated by desktop and process identity simultaneously running.
3. If UseSharedWPDesktop=1 and all application pools launch as unique user identity, then there is one Desktop that is shared by all the user identities. This removes the limit of 60 at the cost of not perfectly isolated application pools. You have no desktop isolation but retain process identity isolation between application pools.
4. UseSharedWPDesktop=1 and all application pools launch as Network Service is pretty much a degenerate case that is not interesting. :-)

Back to the problem

The customer was already aware of those blog posts so they were already using UseSharedWPDesktop=1 (otherwise they were not able to run more than 50 concurrent application pools), set sessionViewSize to 64 Mb and the default value for “HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\SubSystems\Windows”.

To know more about Desktop Heap status on the server we used Desktop Heap Monitor: this comes with a comprehensive usage guide, anyway here are some quick steps to use it:

1. Log in with an account that has Administrator privilege. Note that you must either do this at the physical interactive console (i.e. not through Terminal Services) OR connect to the console session using mstsc /console (only possible on XP or Server 2003)
2. Open a command prompt and change to "c:\kktools\dheapmon8.0"
3. For uni-processor systems run
   • DHEAPINST -f
4. For multi-processor systems (including single processor hyper-threaded systems) run
   • SET _NT_SYMBOL_PATH=symsrv*symsrv.dll*C:\Symbols*http://msdl.microsoft.com/download/symbols
   • DHEAPINST
   • You will be prompted to agree to the symbol server terms of use. You must accept these for the tool to work
     The server in question must be able to make an outgoing connection to the Internet for this to work
5. For either 3. or 4. you should see
   • dheapinst - Desktop Heap Monitor installed successfully
6. DHEAPMON uses a driver in order to read information from the kernel memory. This must be loaded by the following command:
   • DHEAPMON –l
   • You should see this confirmation message: Dheapmon - Driver loaded successfully
7. You are now ready to run the tool and capture the desktop heap information. Run the following command:
   • DHEAPMON -v > desktopheap.txt
   • Important: if the output text file does not contain a line like this: “Windowstation: (Service-0x0-3e7$) SessionID: 0” (without quotation marks) then it is almost certain that you did not connect to the server using Terminal Services Client using the /console switch. You must connect using /console to get the full information required

To automate the process of gathering Desktop Heap information, create a batch file called HeapMon.bat containing the following:

@echo off
rem vars C B A are day month year in the for line below, change them to change date
for /F "tokens=2,3,4 delims=/- " %%A in ('date /T') do set DT=%%C_%%B_%%A
for /F "tokens=1,2 delims=/: " %%A in ('time /T') do set TM=%%A-%%B
echo %DT%
echo %TM%
Dheapmon -v > dheapmon_out_%DT%_%TM%.txt

This will create an output file uniquely named with the date and time included; you can use task scheduler to run this batch file at regular intervals, e.g. once per hour.

Once finished all diagnostic activity with DHeapMon, unload the device driver with:

DHEAPMON -u

To uninstall dheapmon, run:

DHEAPMON -r

From the report I got is was clear that the customer was really using only 1 desktop within IIS WinSTA (remember he was using a shared desktop across his application pools); as IIS WinSTA is created as non interactive, this single desktop is 512 Kb but it was exhausted at 99.2% when then problem appeared:

Windowstation: (WP_WINSTA-054f77ac-6ece-4080-b3a2-b0bf08dacaab) SessionID: 0
Desktop: (Default) Addr: bbc30000
Desktop Heap          524288 (0x   80000) Bytes
Committed             520192 (0x   7f000) Bytes
Uncommitted             4096 (0x    1000) Bytes
Allocated             520072 (0x   7ef88) Bytes
Total Freed              120 (0x      78) Bytes
Unused                  4216 (0x    1078) Bytes
Used Rate                    (      99.2) %

To calculate SessionViewState size we should do something like:

312Kb (Winsta0/Default) +
64Kb (Winsta0/Disconnect) +
128Kb (Winsta0/Winlogon) + 
7 * 512Kb (all other non interactive winsta) +
512Kb (IIS non interactive winsta WP_WINSTA-xxx)
== 4600 Kb  (< 5 Mb)

So apparently the problem is not on SessionViewSize bur rather just on the non interactive Desktop Heap size; moreover setting this value to 64 Mb as the customer did is not necessary because they were using a bit less than 4 Mb for that value so also the default (20 Mb) were far enough.

The problem of increasing this one too much is that not only IIS owns one, so we first tried using 1024 Kb:

                312 + 64 + 128 + 7 * 1024 = 7672 Kb (< 7 Mb)

So we tried with the following (not the customer was not using /3gb):

• UseSharedWPDesktop enabled
• Delete the SessionViewSize registry key to get the default 20 Mb
• SharedSection=1024,3072,1024

This first try allowed the server to run fine a bit longer than before but at the end the problem was there again, and another heapmon trace showed we were still exhausting Desktop Heap… we gave another try with SharedSection=1024,3072,2048 (312 + 64 + 128 + 7 * 1024 = 14840 Kb (< 15 Mb).

Bingo, this time we did it!  

Of course be very careful when you manipulate the Desktop Heap configuration and if you’re not sure how to do (or if you’re doing it right, or if you need help to fine tune your settings etc…) give Microsoft Support a call!

Conclusions

To close with some general recommendations:

1. Any kind of platform that has to support more than concurrent 450 processes should be considered to migrate under x64, and to really care about context switching
2. Windows 2008 Server with Dynamic Session Space removes the SessionViewSize limits, but desktop heap limits would still apply
3. 450 processes identity is also a very big because logon session & token cache might duplicate kernel memory
4. Always be very careful when changing those values since this might have effect on other kernel quota

The point is we will always have inherent x32 bit limitations that we cannot workaround and 450 application pools is a pretty close “practically tested” limit…

 

Carlo

 

Quote of the Day:
Women like silent men. They think they're listening. - Marcel Archard