Troubleshots

IIS7 Error: cannot add duplicate collection entry of type ‘mimeMap’ with unique key attribute ‘fileExtension’ . . .

chaun@microsoft.com — Fri, 04 Dec 2009 15:42:00 GMT

Problem:

For the sake of comparison, in IIS6 (Windows 2003) if you add a mime type (example: .xyz - xml/text) at the virtual directory level and then also attempt to adds the same mime type at the web site level, IIS6 will alert you that a child entity has an override for the mime type. If I accept the override, all entities under the web site will inherit the mime type, but the one with an explicit value will override the inherited one.

By contrast, in IIS7 (Windows 2008 and Vista), adding the same mime types at the application level (which IIS will place into a web.config file) and the site level (which IIS will place into the applicationHost.config file) may cause a conflict. No warning is given except for an error in the client browser that goes something like this:

Server Error in Application
Internet Information Services 7.0
HTTP Error 500.19 - Internal Server Error

MIME Types
There was an error while performing this operation.
Details
Filename
\\?\c:\something\something\.....\web.config
Line number: 25
Error: cannot add duplicate collection entry of type ‘mimeMap’ with unique key attribute ‘fileExtension’ set to ‘.xyz’
OK

Solution:

One solution would be to simply remove one of the two conflicting mimeMap extensions from one of the config files. However, in some rare scenarios the IIS administator may need to not remove one of the two conflicting mime mappings. If both mimeMaps are needed, the way to get around the conflict is to make one simple addition to the web.config which will do a REMOVE immediately prior to doing an ADD.

So if your application installation adds the following line to the web.config. . .

           <mimeMap fileExtension=".xyz" mimeType="xml/text" />

… And this line creates a conflict with the same mime map in the applicationhost.config file, all you have to do is add the following line ABOVE that line so that it will look like the following:

           <remove fileExtension=".xyz" />
           <mimeMap fileExtension=".xyz" mimeType="xml/text" />

Doing the remove before the add should be fine even when the .xyz mime map doesn't exist at a higher level.

IIS compression overwrites the VARY: header

chaun@microsoft.com — Fri, 02 Oct 2009 03:51:00 GMT

Symptoms

IIS compression overwrites the VARY header. If for example an ASP.NET Web Service adds the HTTP header "Vary: Accept-Language, Accept-Encoding” to every response, it may work fine until IIS compression is enabled. The vary header may still contain: “Vary: Accept-Encoding” but it no longer in this example contains the "Vary: Accept-Language, Accept-Encoding” that the web service had added.

Cause

This is by design.

Resolution

The choice of workarounds seems to be to between writing an ISAPI filter that writes the header in SF_NOTIFY_SEND_RESPONSE or disabling compression.

The IIS compression code is overwritting the header after SF_NOTIFY_SEND_RESPONSE. The only way around that would be to use an ISAPI filter. But the filter would have to rewrite the header after SF_NOTIFY_SEND_RESPONSE. It may be possible to have a filter rewrite the header in SF_NOTIFY_SEND_RAW_DATA. But this has not been tested out yet to know for sure.

Products Affected

Internet Information Services (IIS) 5.0, 6.0 and IIS 7.0.

More Information

If you use the ISAPI, then IIS6 would need to be run in IIS 5.0 isolation mode.
http://support.microsoft.com/kb/311852
Information about ISAPI filters that register the SF_NOTIFY_READ_RAW_DATA event in Internet Information Services version 6.0

Also, if you use the ISAPI filter, you can expect a significant performance hit.
http://blogs.msdn.com/david.wang/archive/2005/12/14/How-IIS6-Compression-Schemes-interact-with-ISAPI-Filters.aspx
“SF_NOTIFY_SEND_RAW_DATA is an absolute/performance optimization killer.”

IE 8 seems unable to display some Wikipedia Pages - application/x-gzip-compressed MIME type

chaun@microsoft.com — Sun, 07 Jun 2009 03:12:00 GMT

Problem:

IE8 is unable to display some pages from http://en.wikipedia.org when IE 7 and other browsers can open the same pages fine. IE8 is able to open most wikipedia pages. The pages it cannot open seem to be very random and fit no pattern.

Environment:

IE 8.0.6001.18702 on XP SP3 32-bit OS

Error Message:

File download - Security Warning

Do you want to save this file, or find a program online to open it?

Unknown file type

En.wikipedia.org

Find save cancel

(And after selecting the FIND option…)

Windows has the following information about this MIME type. This page will help you find software needed to open your file.

Windows File Association

MIME Type: application/x-gzip-compressed

Description: UnKnown

Windows does not recognize this MIME type.

Possible Solutions:

1. Simply register with Wikipedia (create an account) and log into Wikipedia. This seems to work for most people experiencing this error.

2. Uninstalling the google desktop supposedly has worked for others. Updating the google desktop may have worked for some. Stopping the google desktop supposedly worked for some. Installing the google desktop supposedly worked for one.

3. Setting IE as the "Default program" has supposedly worked for some.

Click the Windows Start Button, type "Default program" (without quotation marks) in the Start Search box and press Enter.

Click Set default program.

Click Internet Explorer

Click Set this program as Default and click OK.

Click Set program access and computer defaults

Expand Custom

Besides IE, check Enable access to this program.

Click OK.

D. Downloading the IE8 addon for Wikipedia Visual Search helped one person. (http://www.ieaddons.com/en/details/searchhelpers/Wikipedia_Visual_Search/)

E. Clearing the IE cache (Tools menu > Internet Options > General Tab > Delete button (to delete temp files, history, cookies, saved passwords, and web form information) has supposedly helped one person.

F. Running IE8 in compatibility mode and restarting IE supposedly helped one person

Root Cause:

Unknown. I took Fiddler2 traces of a failing attempt and a succeeding attempt. The main differences I see between the failure trace and the success trace is that in the success trace we get a 200 rather than a 304. We also have a cookie that has more information. I’m tempted to speculate that the problem may be more with some nuance of the X-Vary-Option rather than with the application/x-gzip-compressed MIME type. But this is only speculation at this point.

A Slippery System.IO.FileLoadException

chaun@microsoft.com — Thu, 02 Apr 2009 21:47:00 GMT

After installing the framework 3.5 SP1 (kb 951847) (which also updates framework 2.0) an asp.net web app occasionally began throwing the following exception, as reported by their custom error page:

An error occurred attempting to process your request. Loading this assembly would produce a different grant set from other instances. (Exception from HRESULT: 0x80131401) MyCustomDll.MyCustomDllat MyCustomDll.MyCustomDll.

This should be a System.IO.FileLoadException, but there never seemed to be any record of the System.IO.FileLoadException. It didn't even show up in the application event log as expected:

Event Type:     Warning
Event Source: ASP.NET 2.0.50727.0
Event Category:          (3)
Event ID:          1309
Description:
Event code: 3005
Event message: An unhandled exception has occurred.
Process name: w3wp.exe
Exception type: Exception
Exception message: An error occurred attempting to process your request.
Loading this assembly would produce a different grant set from other instances. (Exception from HRESULT: 0x80131401)

Request information:
    Request URL: <https://www.mysite.com:443/Application/Jobs/CustomStuff.aspx?
    Request path: /APPLICATION/Jobs/CustomStuff.aspx
Thread information:
   Stack trace:    at MyCustomDll.MyCustomDll.CustomStuff.updateButton_Click(Object sender, EventArgs e)
   at System.Web.UI.WebControls.Button.OnClick(EventArgs e)
   at System.Web.UI.WebControls.Button.RaisePostBackEvent(String eventArgument)
   at System.Web.UI.WebControls.Button.System.Web.UI.IPostBackEventHandler.RaisePostBackEvent(String eventArgument)
   at System.Web.UI.Page.RaisePostBackEvent(IPostBackEventHandler sourceControl, String eventArgument)
   at System.Web.UI.Page.RaisePostBackEvent(NameValueCollection postData)
   at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)

Getting the dump of the w3wp.exe at the right time was impossible at first. We couldn't dump on CLR exception with a DebugDiag 1.1 crash rule because instead of the expected System.IO.FileLoadException was showing up as a ',' rather than a 'System.IO.FileLoadExeception' according to the debugdiag crash rule log file. It continued to show up as a “blank exception” in the debugdiag log.txt files (made by a crash rule) until the customer installed the updates-to-3.5-sp1 (KB959209 = KB958481/KB958483/KB958484). After those updates Debugdiag was able to trip a dump when the System.IO.FileLoadException was thrown:

Exception type: System.IO.FileLoadException
Message: Loading this assembly would produce a different grant set from other instances. (Exception from HRESULT: 0x80131401)
InnerException: <none>

The dump showed that the web application was initializing a class in a GAC'd assembly ("C:\WINDOWS\assembly\GAC\MyCustomDLL\1.0.530.2__26fddaf73a962ee0\MyCustomDLL.Integration.Export.dll") and also showed that somehow this assembly has a different "SecurityDescriptor" in the "Shared Domain." The calling assembly MyCustomDLL.Base.DLL is not present in GAC and so it gets loaded in LoadFrom loader context. Now when calling assembly tries to load the assembly MyCustomDLL.Integration.Export.dll, fusion sees that it is GACed and already loaded in Load Context as a domain neutral assembly.

So far I know of two workarounds to this issue:

1) Remove the called assembly MyCustomDLL.Integration.Export.dll and its dependency MyCustomDll.Data.dll from GAC so that it does not get loaded as domain-neutral in load context

2) Move the calling assembly to GAC so that it also gets loaded in Load Context.

When VS2005 won’t Break on Breakpoints

chaun@microsoft.com — Wed, 25 Mar 2009 21:41:00 GMT

On a fully updated Vista workstation running Visual Studio 2005 SP1 a breakpoint would be set anywhere in a simple asp.net web application, debugging would begin, and the breakpoint would be missed. It would not break on the breakpoint. During troubleshooting it became clear that Visual Studio was installed onto an external USB drive and the web application project was also saved to the external USB drive. The problem disappeared when we saved the web application to the local system partition--debugging on the same project saw the expected breakpoint breaking. The only difference was the location of the project. When we saved the project to a thumb drive (from a different vendor than the first usb drive) as a second test of a location on a USB driven drive the problem returned—the breakpoint began to be missed again.

Recommendations:

1. In the context of Vista the first caveat is to watch out for UAC related issues. Reference: http://blogs.msdn.com/mubashirk/archive/2009/02/12/not-able-to-debug-an-asp-net-application-in-visual-studio-2005-on-windows-vista-operating-system-having-ie7.aspx

2. Try debugging the project on an internal IDE or SCSI drive rather than an external USB-driven HDD

3. Try installing Visual Studio 2005 and SP1 to an internal drive rather than an external USB-drive HDD

4. Also consider moving the project locally if the project is saved on a remote share and accessed using SMB

So far this problem isn't something I can reproduce in my test environment so I'm not able to speculate much about root cause on this one. It does seem like this problem is extremely rare. Hopefully more details will emerge on this one.

Additional Search Words:

"WebDev.WebServer.EXE" Managed Loaded "Skipped loading symbols" "Module is optimized and the debugger option" enabled

'WebDev.WebServer.EXE' (Managed): Loaded 'C:\Windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.

'WebDev.WebServer.EXE' (Managed): Loaded 'C:\Windows\Microsoft.NET\Framework\v2.0.50727\WebDev.WebServer.EXE', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.

'WebDev.WebServer.EXE' (Managed): Loaded 'C:\Windows\assembly\GAC_32\WebDev.WebHost\8.0.0.0__b03f5f7f11d50a3a\WebDev.WebHost.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.

'WebDev.WebServer.EXE' (Managed): Loaded 'C:\Windows\assembly\GAC_MSIL\System.Windows.Forms\2.0.0.0__b77a5c561934e089\System.Windows.Forms.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.

'WebDev.WebServer.EXE' (Managed): Loaded 'C:\Windows\assembly\GAC_MSIL\System\2.0.0.0__b77a5c561934e089\System.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.

'WebDev.WebServer.EXE' (Managed): Loaded 'C:\Windows\assembly\GAC_MSIL\System.Drawing\2.0.0.0__b03f5f7f11d50a3a\System.Drawing.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.

'WebDev.WebServer.EXE' (Managed): Loaded 'C:\Windows\assembly\GAC_32\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.

'WebDev.WebServer.EXE' (Managed): Loaded 'C:\Windows\assembly\GAC_MSIL\System.Configuration\2.0.0.0__b03f5f7f11d50a3a\System.Configuration.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.

'WebDev.WebServer.EXE' (Managed): Loaded 'C:\Windows\assembly\GAC_MSIL\System.Xml\2.0.0.0__b77a5c561934e089\System.Xml.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.

'WebDev.WebServer.EXE' (Managed): Loaded 'C:\Windows\assembly\GAC_32\System.Data\2.0.0.0__b77a5c561934e089\System.Data.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.

'WebDev.WebServer.EXE' (Managed): Loaded 'C:\Windows\assembly\GAC_32\System.Transactions\2.0.0.0__b77a5c561934e089\System.Transactions.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.

'WebDev.WebServer.EXE' (Managed): Loaded 'C:\Windows\assembly\GAC_32\System.EnterpriseServices\2.0.0.0__b03f5f7f11d50a3a\System.EnterpriseServices.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.

'WebDev.WebServer.EXE' (Managed): Loaded 'C:\Windows\assembly\GAC_MSIL\System.Web.RegularExpressions\2.0.0.0__b03f5f7f11d50a3a\System.Web.RegularExpressions.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.

'WebDev.WebServer.EXE' (Managed): Loaded 'C:\Windows\assembly\GAC_MSIL\System.Design\2.0.0.0__b03f5f7f11d50a3a\System.Design.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.

'WebDev.WebServer.EXE' (Managed): Loaded 'C:\Windows\assembly\GAC_MSIL\Microsoft.JScript\8.0.0.0__b03f5f7f11d50a3a\Microsoft.JScript.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.

'WebDev.WebServer.EXE' (Managed): Loaded 'C:\Windows\assembly\GAC_32\VJSharpCodeProvider\2.0.0.0__b03f5f7f11d50a3a\VJSharpCodeProvider.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.

'WebDev.WebServer.EXE' (Managed): Loaded 'C:\Windows\assembly\GAC_MSIL\CppCodeProvider\8.0.0.0__b03f5f7f11d50a3a\CppCodeProvider.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.

'WebDev.WebServer.EXE' (Managed): Loaded 'C:\Users\tatchison\AppData\Local\Temp\Temporary ASP.NET Files\website1\605567ab\d84dc142\App_Code._u4goukp.dll', Symbols loaded.

'WebDev.WebServer.EXE' (Managed): Loaded 'C:\Users\tatchison\AppData\Local\Temp\Temporary ASP.NET Files\website1\605567ab\d84dc142\App_global.asax.fmh52wto.dll', Symbols loaded.

'WebDev.WebServer.EXE' (Managed): Loaded 'C:\Windows\assembly\GAC_32\System.EnterpriseServices\2.0.0.0__b03f5f7f11d50a3a\System.EnterpriseServices.Wrapper.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.

'WebDev.WebServer.EXE' (Managed): Loaded 'C:\Windows\assembly\GAC_MSIL\System.Web.Mobile\2.0.0.0__b03f5f7f11d50a3a\System.Web.Mobile.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.

'WebDev.WebServer.EXE' (Managed): Loaded 'C:\Windows\assembly\GAC_MSIL\System.ServiceModel\3.0.0.0__b77a5c561934e089\System.ServiceModel.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.

'WebDev.WebServer.EXE' (Managed): Loaded 'C:\Windows\assembly\GAC_MSIL\SMDiagnostics\3.0.0.0__b77a5c561934e089\SMDiagnostics.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.

'WebDev.WebServer.EXE' (Managed): Loaded 'C:\Windows\assembly\GAC_MSIL\System.Web.Services\2.0.0.0__b03f5f7f11d50a3a\System.Web.Services.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.

'WebDev.WebServer.EXE' (Managed): Loaded 'C:\Windows\assembly\GAC_MSIL\System.Runtime.Serialization\3.0.0.0__b77a5c561934e089\System.Runtime.Serialization.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.

'WebDev.WebServer.EXE' (Managed): Loaded 'C:\Windows\assembly\GAC_MSIL\System.IdentityModel\3.0.0.0__b77a5c561934e089\System.IdentityModel.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.

'WebDev.WebServer.EXE' (Managed): Loaded 'C:\Windows\assembly\GAC_MSIL\System.ServiceModel.Web\3.5.0.0__31bf3856ad364e35\System.ServiceModel.Web.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.

'WebDev.WebServer.EXE' (Managed): Loaded 'C:\Windows\assembly\GAC_MSIL\System.WorkflowServices\3.5.0.0__31bf3856ad364e35\System.WorkflowServices.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.

'WebDev.WebServer.EXE' (Managed): Loaded 'C:\Users\tatchison\AppData\Local\Temp\Temporary ASP.NET Files\website1\605567ab\d84dc142\App_Web_3e6yucwc.dll', Symbols loaded.

'WebDev.WebServer.EXE' (Managed): Loaded 'A_70bac5d9_755a_4557_802d_853bcd696012', No symbols loaded.

'WebDev.WebServer.EXE' (Managed): Loaded 'M_70bac5d9_755a_4557_802d_853bcd696012', No symbols loaded.

'WebDev.WebServer.EXE' (Managed): Loaded 'C:\Users\tatchison\AppData\Local\Temp\Temporary ASP.NET Files\website1\605567ab\d84dc142\App_Web_lnzudcag.dll', Symbols loaded.

'WebDev.WebServer.EXE' (Managed): Loaded 'C:\Users\tatchison\AppData\Local\Temp\Temporary ASP.NET Files\website1\605567ab\d84dc142\App_Theme_White.sbtkniwu.dll', Symbols loaded.

'WebDev.WebServer.EXE' (Managed): Loaded 'C:\Windows\assembly\GAC_MSIL\Microsoft.VisualBasic\8.0.0.0__b03f5f7f11d50a3a\Microsoft.VisualBasic.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.

The program '[4984] WebDev.WebServer.EXE: Managed' has exited with code 0 (0x0).

ISS can lock the ASP.net Temp folder files

chaun@microsoft.com — Wed, 02 Apr 2008 22:42:00 GMT

Problem:

If you're browsing to an asp.net page and your browser reports "Sever Error in '/' Application." and "The current identity... ASPNET does not have write access to 'c:Windows\microsoft.net\framework\v2.0.50727\Temporary ASP.net files', you might naturally think of adjusting NTFS permissions for the ASPNet account or running aspnet_regiis -ga in the attempt to correct the problem.

Possible solution:

Try stopping any and every service that has anything to do with either Blackice and/or ISS (that's ISS, not IIS). Particulary look for the "ISS Buffer Overflow Exploit Prevention" service. Stop it. Restart IIS. Test asp.net functionality.

The application event log may have an event resembling this logged:

Event Type: Warning

Event Source: ASP.NET 2.0.50727.0

Event Category: Web Event

Event ID: 1309

Event code: 3005

Event message: An unhandled exception has occurred.

Event ID: 0746e871023341b099dd3393e82e464d

Application domain: /LM/W3SVC/1/Root/Blahblahblah

Process name: aspnet_wp.exe

Account name: COMPUTERNAME\ASPNET

Exception information:

Exception type: HttpException

Exception message: The current identity (Computername\ASPNET) does not have write access to 'c:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files'.

Request information:

Stack trace: at System.Web.HttpRuntime.SetUpCodegenDirectory(CompilationSection compilationSection)

at System.Web.HttpRuntime.HostingInit(HostingEnvironmentFlags hostingFlags)

"Page cannot be displayed" due to Non-Paged Pooled Memory Depletion

chaun@microsoft.com — Thu, 21 Feb 2008 04:03:00 GMT

Overview

It’s not unnatural to assume an IIS process hang when web client browsers begin reporting either “Page cannot be displayed . . . cannot find server or dns error” (IE6) or "Internet Explorer cannot display the webpage. . .” (IE7). But when an IISRESET proves insufficient to provide temporary relief, it is time to begin investigating possibilities outside of IIS. When it is clear that a reboot of the server is needed for temporary respite, troubleshooting should be sensitive to the possibility of a NPP (Non-Paged Pooled) memory leak in the kernel. Checking the httperror log for evidence of “connections_refused” should verify whether or not NPP is depleted. Poolmon and Perfmon can be used to confirm the NPP leak and determine root cause. Root cause often points to an outdated third-party driver which, when updated or uninstalled, solves the NPP leak.

Troubleshooting

The best clue needs to be seen in the little-known httperr log. While there are many possible causes for the “Page cannot be displayed” error, there is only root cause which causes the http.sys driver to begin refusing client connections--a depletion of non-paged pooled memory, an NPP leak. The HTTP.sys driver was new with Windows 2003, is a kernel mode driver, and, at the risk of splitting hairs, is technically not part of IIS 6.0. This distinction is important in troubleshooting. When http.sys refuses to hand connections to IIS a “Connection_refused” or “Connections_refused” will be logged in the httperr log (C:\WINDOWS\system32\Logfiles\HTTPERR) rather than the IIS logs. Also, if adplus.vbs or debugdiag is used to make a memory dump of IIS processes at this time, dump analysis will show no problems (and probably no connections either) going on in the IIS processes.

KB 820729 offers only one possible explanation for a “Connections_refused” error: “The kernel NonPagedPool memory has dropped below 20MB and http.sys has stopped receiving new connections.” I wouldn’t get too set on the 20MB threshold as there seem to be exceptions. Sometimes connections get refused when it appears like there is more than 20MB of NPP left. So it may be more like 20% of the total NPP memory rather than 20MB. Perhaps when NPP has become 80% depletion, http.sys will begin to refuse client connections. But whatever the official answer is there, suffice it to say that if you see “connections_refused” in the httperr log, you can safely assume this is an NPP leak. With its default settings, HTTP.sys is the ‘canary in the coal mine’ in an NPP leak scenario. On an Exchange server, for instance, the OWA website becomes unavailable to clients long before SMTP stops delivering mail. But give the leak enough time to leak further and mail will stop flowing. Allow the NPP leak to go further and the administrator probably won’t be able to connect via RDP to the server.

So once evidence of http.sys refusing client connections in the httperr log, what comes next? Some decisions have to be made.

Solution Options

If immediate relief is needed, perhaps the server needs to be rebooted and troubleshooting can begin after the reboot with perfmon and poolmon utilities. The poolmon and perfmon tools are the best way to find which driver is causing the leak. (I’d say more about this but the Microsoft Product Support Platforms Performance team usually handles this for me.) But if root cause analysis is needed more immediately than relief of the symptoms, the reboot needs to be avoided. Perfmon and Poolmon can be used to track the leak further. No IIS websites will be reachable of course, but other services may continue to function a little longer.

If the server is on SP2 for Windows 2003, it may be a very good idea to try disabling the TCPChimney. This can be done without a reboot per kb 945977. Try disabling the TCPChimney, run an IISRESET, wait a few seconds, and test the websites. If this ‘trick’ works, perhaps your NIC drivers need to be updated and the TCPChimney can be enabled later.

Open Taskmanager, checkmark “Show processes from all users,” expand the View menu, select Show Columns, and add checkmarks beside Non-paged Pool, Username, and PID. This may show you if there are any processes in particular which seem gluttonous with the NPP memory. Such information may or may not provide a good clue. Regardless, it will help you know where your NPP is allocated.

Since the usual cause of an NPP leak is an outdated driver, try stopping various services one at a time (such as antivirus services or backup software services), run IISRESET, wait a few seconds, and test the websites. This method might give you a clue as to what is leaking. If, for example, you stop antivirus services and NPP levels increase and remain stable, perhaps the ultimate solution is simply to contact the Antivirus vendor and ask for recommended updates to the product installed on the server. It may be that simple.

You also might consider whether or not you want to enable aggressive memory usage in the registry (EnableAggressiveMemoryUsage). This allows http.sys to keep accepting and handing off client connections to IIS well below 20 MB. This can give you some added relief but doesn’t solve the NPP leak.

If you have the /3GB switch set, you might want to consider whether it is truly needed or not. Removing it may give you more NPP. But it also may cause other problems.

References and Resources

933844 Error message when you try to view a Web page that is hosted on IIS 6.0: "Page cannot be displayed"

http://support.microsoft.com/default.aspx?scid=kb;EN-US;933844

934878 Users receive a "The page cannot be displayed" error message, and "Connections_refused" entries are logged in the Httperr.log file on a server that is running Windows Server 2003, Exchange 2003, and IIS 6.0

http://support.microsoft.com/default.aspx?scid=kb;EN-US;934878

http://blogs.msdn.com/david.wang/archive/2005/09/21/HOWTO-Diagnose-IIS6-failing-to-accept-connections-due-to-Connections-Refused.aspx

http://support.microsoft.com/?id=945977 - Some problems occur after installing Windows Server 2003 SP2

912376 How to monitor and troubleshoot the use of paged pool memory in Exchange Server 2003 or in Exchange 2000 Server

http://support.microsoft.com/default.aspx?scid=kb;EN-US;912376

177415 How to Use Memory Pool Monitor (Poolmon.exe) to Troubleshoot Kernel Mode Memory Leaks

http://support.microsoft.com/default.aspx?scid=kb;EN-US;177415

248345 How to Create a Log Using System Monitor in Windows 2000

http://support.microsoft.com/?id=248345

244139 Windows Feature Allows a Memory.dmp File to Be Generated with Keyboard

http://support.microsoft.com/?id=244139

Event ID 2019 or 2020 or "Insufficient System Resources" error returned when logging on

http://support.microsoft.com/default.aspx?scid=kb;EN-US;272568

You experience difficulties when you use an Outlook client computer to connect to a front-end server that is running Exchange Server 2003

http://support.microsoft.com/default.aspx?scid=kb;EN-US;924047

820129 Http.sys registry settings for IIS

http://support.microsoft.com/default.aspx?scid=kb;EN-US;820129

820729 Error logging in HTTP API

http://support.microsoft.com/kb/820729

http://technet2.microsoft.com/windowsserver/en/library/0d302498-c947-4655-95af-719ae75acfb51033.mspx

Poolmon Overview

http://blogs.technet.com/askperf/archive/2007/03/07/memory-management-understanding-pool-resources.aspx

Memory Management - Understanding Pool Resources

http://blogs.technet.com/askperf/archive/2007/03/23/memory-management-demystifying-3gb.aspx

Memory Management - Demystifying /3GB

401.2 Troubleshooting for Intranet Websites using Integrated Authentication

chaun@microsoft.com — Thu, 21 Feb 2008 01:25:00 GMT

Although there are many reasons why IIS might return the http response code of 401.2 ("Unauthorized: Logon Failed due to server configuration") there is one scenario I keep running into which must be solved on the client side rather than the server. An IIS administrator visits the properties sheet of an internal intranet website, unchecks Anonymous authentication, and confirms that Integrated Windows authentication is selected. Testing with Internet Explorer, he or she is surprised to see the prompt for username and password. If the administrator types in valid credentials, the page is given to the client. If the Cancel button is chosen, however, when prompted for credentials, the browser displays the 401.2 response from IIS. The reflex reaction of the Administrator might go something like this, "Ohh, it's not the well known 401.3 repsonse, so it must not be an NTFS permission problem on web content. The lines are kind of blurry between the 401.1 and 401.2 in my mind so maybe it's some complicated kerberos problem."

When using integrated authentication we would expect the IIS logs to show a 401.2 (or 401 2 rather) for the initial client request. The client is going to attempt to authenticate anonymously at first. IIS will respond to the client saying something like, "Sorry but I'm not configured to allow anonymous requests. Try either NTLM or Kerberos next time." The client tries again using NTLM or Kerberos (it's the clients choice at this point) and if the 401.2 is still being issued, one piece of low hanging fruit to reach for involves not the server but the client.

First I'd focusing on which zone IE says the site is in--Internet zone? Local Intranet zone? Trusted sites? Presumably you'll want an intranet site in the Local Intranet zone. But if there are "dots" in the address (example: http://accounting.intranet.local) then perhaps you may see that IE is thinking it is part of the Intranet zone instead. Expand the Tools menu of IE, select Internet options, and settle on the Security tab.

Highlight the icon for Local Intranet and click the Custom Level button. When the window entitled "Security Settings - Local Intranet Zone" opens, scroll to the bottom of the window and consider the four options for "Logon." IS the bullet beside "Automatic logon only in Intranet Zone?" Perhaps it should be. Or is it beside "Automatic logon with current user name and password?" That should work well too. Either of the other two options may not be a good idea for an intranet site using integrated authentication. Adjust if desired.

While the Local Intranet icon is still highlighted, click the Sites button. Click the Advanced button. Consider typing in the address of the intranet site into the field labeled "Add this website to the zone:" and click the Add button. If you're unable to do this, your workstation may have these settings dictated by group policy.

When the website is added to the local intranet zone list and when the client is set to automatically provide credentials when browsing sites found in the local intranet list, the 401.2 often goes away. The client simply wasn't set to present the credentials to IIS.

How to Make an IIS 6.0 Metabase 'Exportable'

chaun@microsoft.com — Thu, 21 Feb 2008 00:44:00 GMT

When a %windir%/system32/inetsrv/metabase.xml file is transplanted from an IIS 6.0 server to the inetsrv directory of another IIS 6.0 sever, the IISAdmin service on the target server won't run. This is by design. This happens because important portions of the metadata are encrypted by a "c23 key" that is unique to the source IIS server. The c23 key on the target server is also unique and cannot decrypt that which was encrypted by another c23 key. Even if you save the metabase.xml file to a safe location, uninstall and reinstall IIS, and attempt to use the saved metabase.xml file, you run into the same problem. The c23 key has been recreated. And if the metabase.xml file can't be decrypted, the IIS servcies aren't going to be starting.

One little-known way around this obstacle is to take advantage of the "Encrypt backup using password" option given in the IIS metabase backup wizard. Even among those who know the how and why of making metabase backups it seems rare that administrators understand the significance of this encryption option. In my early days with IIS 6.0 I assumed that the wizard was only offering me the option to protect the metabase backup such that no other administrators could restore to it unless they knew the password. KB article 324277 sings this same song. But somewhere in the past years the helpful tip was pointed out to me that this option is also enables an administrator to encrypt the metabase backup with their own password rather than with the unique c23 machinekey. If you use your own password to do the encryption, you can then move that backup from the %windir%/system32/inetsrv/MetaBack folder on the source server to the same folder on the target server. On the source server you can run the metabase backup/restore wizard, select the imported backup, begin the restore, supply your password, and watch the IIS services start. The IIS configuration on the target server should then be practically identical to the source server. (But you may have to edit IP bindings.)

I'm a big fan of metabase backups. I like to back the metabase up before any major changes to IIS configuration (changes to sites, adding application pools, removing isapi filters, etc.), test my changes, and make a new metabase backup when I've become comfortable with the changes. If there is the slightest idea in my mind that I might end up reinstalling IIS I'm especially quick to make the backup using my own password for encryption. When filling the field for the name of the backup I even type the password as part of the name. (So I don't have to remember it.)

Other ways to accomplish similar migrations would include Microsoft's IIS 6 Migration tool, Microsoft's Web Deployment Tool, or ADSonline's IISExport tool. But those may have to wait for another blog.

References:

http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/d40b56ee-90d4-45e1-9b82-4aaea90eb02e.mspx?mfr=true
Backing Up and Restoring the Metabase in IIS 6.0 (IIS 6.0) > To create a portable backup (password required)

http://support.microsoft.com/default.aspx?scid=kb;EN-US;324277
324277 How To Create a Metabase Backup by Using IIS 6.0 in Windows Server 2003

http://support.microsoft.com/default.aspx?scid=kb;EN-US;300672
300672 How to create a metabase backup in IIS 5

http://support.microsoft.com/default.aspx?scid=kb;EN-US;302573
302573 How To Back Up and Restore IIS

http://support.microsoft.com/default.aspx?scid=kb;EN-US;302888
302888    How To Schedule Metabase Backups Using WSH

http://www.microsoft.com/technet/scriptcenter/scripts/default.mspx?mfr=true
http://www.microsoft.com/technet/scriptcenter/scripts/iis/default.mspx
http://www.microsoft.com/technet/scriptcenter/scripts/iis/iis6/default.mspx
    Search for: Back Up the Metabase Using a Password {Scripts for iis 5, 5.1 and 6.0}

814866    Common tasks for administration of IIS 6.0
http://support.microsoft.com/default.aspx?scid=kb;EN-US;814866

http://sanal.org/story/discover-the-iis-metabase-unlock-the-details-of-your-web-server-configuration-2/
Discover the IIS Metabase: Unlock the Details of Your Web Server Configuration

http://www.iis-aid.com/articles/how_to_guides/backing_up_and_restoring_your_iis_6_metabase
Backing up and Restoring your IIS 6 Metabase

http://www.ms2.cn/downloads/details.aspx?FamilyId=2AEFC3E4-CE97-4F25-ACE6-127F933A6CD2&displaylang=en
Internet Information Services 6.0 Migration Tool

http://www.iis.net/downloads/default.aspx?tabid=34&g=6&i=1602
Microsoft Web Deployment Tool - Tech Preview 1 - x86

http://www.adsonline.co.uk/IISExport/ - IISExport

http://support.microsoft.com/default.aspx?scid=kb;EN-US;887417 - {discusses iissync.vbs and iiscnfg.vbs for IIS in a clustered scenario}

http://msevents.microsoft.com/CUI/WebCastEventDetails.aspx?EventID=1032268486&EventCategory=3&culture=en-US&CountryCode=US

TechNet Webcast: IIS 6.0: IISCNFG.vbs versus IISBack.vbs

http://blogs.iis.net/lprete/archive/2008/01/20/iis6-restoring-metabase-configuration-to-a-new-server.aspx

IIS6: Restoring Metabase Configuration to a New Server

Disabling the TCPChimney During IIS 6.0 Troubleshooting

chaun@microsoft.com — Wed, 20 Feb 2008 21:39:00 GMT

I've become quick to consider disabling the TCP off-loading engine features (or “TOE”) when trouble-shooting a problem involving IIS 6.0 and detecting the slightest hint of a "networking problem" or "communication problem." With the right combination of NIC, NIC driver level, and OS level, TOE is a good thing which significantly improves the performance of TCP processing. Windows can offload the TCP processing of some network streams to the network controller. But under some circumstances (especially with outdated NIC drivers) strange networking problems can result.

THREE CHECKS

The TOE features are enabled in Windows when either the earlier Scalable Networking Pack (kb 912222) or Service Pack 2 for Win2003 or are installed on a Windows 2003 server. The TOE features are disabled whenever update 948496 is applied. Here are three things you might consider checking:

(1) If the file version of %windir%\system32\netsh.exe is 5.2.3790.2670 or higher it is possible that TOE is enabled. If the version number is 5.2.3790.1860 (SP1) or lower the TOE features need not be worried about.

(2) Open the registry editor, navigate to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters and focus on the following values: EnableTCPChimney, EnableTCPA, and EnableRSS. If all three are set to a value of Zero, consider TOE disabled. If they’re all set to 1, I’d recommend you consider disabling them during troubleshooting.

(3) Try running “netstat –t” from the command prompt to attempt to see if any client connections are in the offloaded state

TWO METHODS of DISABLING

One of the three TOE features can be disabled ‘on the fly’ and may be the first place to start. Disabling the TCPChimney (but not RSS or TCPA) is often sufficient to deal with problems related to TOE. But not always. To disable the TCPChimney, open a command prompt and run the following command on the server:

C:\>Netsh int ip set chimney DISABLED

No reboot is needed. No IISRESET is needed either.

A more thorough (and much more preferable) alternative to using the netsh command would be to open the registry, navigate to. . .

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

… and change the dword values to zero for the following three keys from 1 to 0:

o "EnableTCPChimney"=dword:00000000

o "EnableTCPA"=dword:00000000

o "EnableRSS"=dword:00000000

A reboot is needed for these registry value changes to take effect.

When troubleshooting what seems like a networking problem, the disabling of TOE features should be done not just on the front end web servers but also on ISA servers, any middle tier web service servers, the back-end SQL servers, and perhaps even the domain controllers—any machine that is running Win2003 with SP2 might be a good candidate. If you need to disable the TOE features on multiple servers, consider deploying update 948496.

If disabling the TCPChimney seems to fix the problem, leave it disabled at least long enough to update your NIC drivers and firmware. If your NIC is a Broadcom NIC, I’d definitely recommend considering the updating drivers from http://www.broadcom.com/support/ethernet_nic/downloaddrivers.php. The NIC driver version numbers need to be version 3.7.19 or higher.

References:

http://support.microsoft.com/default.aspx/kb/948496 - An update to turn off default SNP features is available
http://support.microsoft.com/default.aspx?scid=kb;EN-US;912222 - The Microsoft Windows Server 2003 Scalable Networking Pack release
http://support.microsoft.com/?id=945977 - Some problems occur after installing Windows Server 2003 SP2
http://support.microsoft.com/?id=942861 - Error message when an application connects to SQL Server on a server that is running Windows Server 2003: "General Network error," "Communication link failure," or "A transport-level error"
http://www.microsoft.com/whdc/device/network/TCP_Chimney.mspx - Scalable Networking: Network Protocol Offload - Introducing TCP Chimney
http://blogs.msdn.com/jamesche/ - Having Network Problems on Win2003 SP2?
http://msexchangeteam.com/archive/2007/07/18/446400.aspx - Windows 2003 Scalable Networking pack and its possible effects on Exchange (part 1)
http://msexchangeteam.com/archive/2008/03/12/448421.aspx - Windows 2003 Scalable Networking pack and its possible effects on Exchange - Part 2
http://www.broadcom.com/press/release.php?id=522952 - Broadcom Demonstrates Industry's First Fully Integrated TCP/IP Offload Engine (TOE) that Supports Microsoft's TCP Chimney Architecture
http://www.broadcom.com/press/release.php?id=409783 - Broadcom to Deliver TCP/IP Offload Technology
http://blogs.technet.com/networking/archive/2007/11/04/sp2-scalable-networking-pack-connectivity-issues.aspx