HTTP/1.1 200 OK - Jimmie's IIS/ASP.NET Chatter

How to capture memory dumps when a specific event log occurs using new Event Log feature.

I came across a new feature (well new to me) in the Event Viewer in Windows 7 (also in Vista and Windows Server 2008) recently that can be useful in troubleshooting various issues.  The new Event Viewer now allows you to attach a task to a particular event which will be launched automatically when a matching event is logged.  As you are probably already thinking, there are a multitude of ways in which this could be used to aid in troubleshooting problems and monitoring server health.  For this article I will cover one real–world scenario where this feature could be used.

Something I have run into on several occasions while working in IIS/ASP.NET support is the desire to capture memory dumps of the IIS process(s) when a particular Event occurs on the system.  Before discovering this feature there wasn’t a really robust and easy way of doing this.  We used to have a tool called EventMon which was an exe that would run in the system tray and monitor for specific events and more recently we have used a custom vbscript to monitor the event log using WMI  (see http://blogs.technet.com/netmon/archive/2007/02/22/eventmon-stopping-a-capture-based-on-an-eventlog-event.aspx).  With the new Event Log feature, these tools are no longer necessary.  If you are like me, you prefer installing as few extraneous tools as possible on your machines.  We also have the added benefit of this running as a service so we don’t have to keep a user logged into the system to monitor the event log.

For our example I will show you how to use the Event Log to trigger a memory dump capture of the IIS processes when a specific event occurs.  To get started, you will need to download and install the Debugging Tools for Windows.  I always install to the C:\debuggers folder for convenience.  Make sure you download the correct bitness (x64, x86) for your system.  Open up the Event Viewer and find the specific event you are interested in.  For my example, I have used an ASP.NET Application Event 1309 which is for an Unhandled Exception.  Obviously in this case the Event already has all the info for fixing our problem and a memory dump would be overkill since it is a simple DivideByZeroException but I am just using this to demonstrate the concept.

Now in the Actions pane click on the “Attach Task To This Event…” option.  This will open up the “Create Basic Task Wizard.  The task name is pre-populated for you but you can edit it if you wish.  You can also enter a description for the task here. 

Clicking the next button will bring up the Event Log information.  This will be filled in and you can simply click next to get to the “Action” step.

On the “Action” step, “Start a program” will be selected by default.  Since this is what we want, no change is needed here.  You can probably start to see the possibilities here.  Since we can start a program, this gives us lots of options for things we could do in response to this Event.  In this case we will be launching the debugger to capture a memory dump.  In addition to launching a program we could have chosen to send an email or display a message. 

This will bring up the “Start a program” step.  Here you will enter the command line and arguments.  In this case, I am using the adplus script to capture the dump. You can find more information about adplus here http://support.microsoft.com/kb/286350.

By default this task will only run when the user account is logged on.  To change this, click the checkbox to open the task properties before clicking the Finish button.  Click Next and Finish to activate the task.  On the Properties dialog, select the option “Run whether user is logged on or not”.  You can also select the task to run under a different account if you wish.  There are lots of advanced options you can set from this dialog such as setting up additional Triggers, executing multiple actions, and setting up Conditions.

That’s all there is to it.  A memory dump of the worker process(s) will be triggered each time this event occurs.  Once you have captured the dumps don't forget to go into the Task Scheduler to disable or delete the task to prevent filling up your hard disk with dump files.  As you can see, we have demonstrated one of many possibilities here using this new functionality.  I hope this has been a helpful tutiorial and increased awareness around just one of the many features available in current versions of Windows.

Loopback Security Check Feature IIS 7

I recently had a support case where the user was not able to browse to his web site from the web server itself.  He was getting prompted for credentials 3 times and after entering valid credentials each time he received a 401.1 Logon Failed Error.  We checked the site and it was using Windows Integrated Authentication and we confirmed that we were able to log in successfully from a remote client machine.  Any time you receive a 401.1 error from IIS, a good first step in troubleshooting this is by enabling logon failure audits in the local security policy.   Turn on Failure auditing for "Audit account logon events" and "Audit Logon Events".  Once this is enabled, all failures will show up in the computer's Security Event Log.  Based on the fact that this worked from the remote client, I was fairly sure we were looking at a failure due to the loopback security check. 

We checked the security event logs after turning on auditing and found the following:

You will however see entry similar to the one below
Log Name: Security
Source: Microsoft-Windows-Security-Auditing
Date: 8/28/2008 10:06:27 AM
Event ID: 4625
Task Category: Logon
Level: Information
Keywords: Audit Failure
User: N/A
Computer: server1.somecustomer.com
Description:
An account failed to log on.
Subject:
Security ID: NULL SID
Account Name: -
Account Domain: -
Logon ID: 0x0
Logon Type: 3
Account For Which Logon Failed:
Security ID: NULL SID
Account Name: admin
Account Domain: bigcorp
Failure Information:
Failure Reason: An Error occured during Logon.
Status: 0xc000006d
Sub Status: 0x0
Process Information:
Caller Process ID: 0x0
Caller Process Name: -
Network Information:
Workstation Name: winvista1289
Source Network Address: 10.10.3.20
Source Port: 52105
Detailed Authentication Information:
Logon Process:
Authentication Package: NTLM
Transited Services: -
Package Name (NTLM only): -
Key Length: 0
This event is generated when a logon request fails. It is generated on the computer where access was attempted.
The Subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe. The Logon Type field indicates the kind of logon that was requested. The most common types are 2 (interactive) and 3 (network). The Process Information fields indicate which account and process on the system requested the logon. The Network Information fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases. The authentication information fields provide detailed information about this specific logon request. - Transited services indicate which intermediate services have participated in this logon request. - Package name indicates which sub-protocol was used among the NTLM protocols. - Key length indicates the length of he generated session key. This will be 0 if no session key was requested.

Based on the details it seemed likely that the loopback check was the issue.  Unlike Windows Server 2003 and IIS6, the security event log does not show the normal strange characters identified in http://support.microsoft.com/kb/896861 but the workarounds that are listed still apply to IIS 7 on Windows Server 2008.  Since indentifying this issue is slightly different on IIS 7, I felt it was worthwhile to let our customers know that the loopback security check feature is enabled on Windows Server 2008 IIS installations.  As of this writing the KB 896861 does not indicate that the workarounds apply to IIS 7 but I have confirmed that they work.

Can't browse site by IP address that requires authentication. Page cannot be displayed.

One of the things I am trying to accomplish with my blog, is to shed some light on some IIS/ASP.Net issues that may be very common but that I have seen manifested in some "not so common" ways.  This post is about just one such instance.  Unlike some of my other posts, I am going to lead of with the summary of the problem and the solution and then I will discuss the way it was discovered.  The common issue here is that Windows Integrated Authentication requires that HTTP Keep-Alives be enabled in IIS.  This is a fairly well known and documented fact, however this was discovered in something of a less than traditional way.

I received a call from a customer who had updated Sharepoint on his IIS server and after restarting IIS, he was not able to browse any of his applications using the IP address of the machine.  He went on to say that he could browse the sites fine if he used the Netbios name of the machine from the local box as well as clients inside the domain. 

My first instinct was that something else (besides IIS) was listening on the IP address he was using on port 80.  A quick check of NETSTAT -ANO proved this not to be the case.  I decided to be doubly sure by changing the default web site to run on port 81 instead of 80 but still had the same result.  One thing we did notice was that when we were browsing by IP address, we were being prompted for credentials.  This is to be expected for sites that require Windows Integrated Authentication and are browsed using a FQDN or IP address.  After entering in correct credentials 3 times, we were getting "Page cannot be displayed".  Ok, so we knew we were hitting the right site and we knew we were entering the correct username and password so what was going on?  Well, it turns out that the customer had inadvertently turned off HTTP Keep Alives which was causing the authentication to fail.

All of this made sense now.  Clients inside the domain that were using the netbios name, were getting in because IE (by default) will pass the security token on to IIS in the intranet zone and the regular NTLM handshake isn't required.  When users outside the domain or when browsing using IP address, the Windows Integrated Auth handshake needed to occur.  The handshake was failing in these instances and thus the "Page cannot be displayed" response was given. 

Hopefully this will help someone who may be experiencing this same problem but hasn't been able to put the two seemingly unrelated facts together.

HTTP/1.1 200 OK

HTTP 413 Request Entity too Large - Can't upload large files using IIS6

Well, here's a new one I ran across recently.  My customer was trying to upload some files using Coldfusion by doing a simple post and was getting a HTTP 413 response when the files were larger than 128K.  IIS6 uses a new metabase property called UploadReadAheadSize when passing data to an ISAPI extension.  In this particular case the customer was using CAC (client access certs) and so the SSL preload was using this property to determine the maximum buffer size for the incoming request.  The default size for this buffer is 48k, which was added to prevent anonymous DOS attacks by uploading very large garbage files. 

The fix is simply to up the size of the UploadReadAheadSize metabase property.  One thing I would like to point out is an error in the technet documentation that explains how to do this (link below).  The following link instructs you to set the size as follows:

cscript adsutil.vbs set w3svc/1/uploadreadaheadsize 200

Which should read:

cscript adsutil.vbs set w3svc/1/uploadreadaheadsize 204800

The metabase setting is represented in bytes, not KB as the technet article would suggest.  You can change this setting up to 4GB although you probably wouldn't want to do so unless you had a compelling business need.  Of course a site that required authentication would be an effective way of limiting the chances of DoS, so if you do need to up this limit, you should consider requiring client authentication to connect. 

Client cannot renegotiate request and returns an HTTP 413 error (IIS 6.0)
http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/7e0d74d3-ca01-4d36-8ac7-6b2ca03fd383.mspx?mfr=true 

Other useful related links:

HTTP/1.1 200 OK

Access Denied error when running app on IIS 6 and IIS 5.1. Runs fine on Windows 2000.

First off, there are many different things that can cause an Access Denied error in an application hosted in IIS.  This post will cover a little known and "under documented" cause for this error called Loopback Check.  IIS 5.1 and 6.0 added an additional security measure call Loopback Checking to help prevent reflection attacks on the server.  I am not a hacker so I do not fully understand how one would implement such an attack on a server without Loopback Checking but the general idea is this:  Mr Hacker opens a connection to my IIS server to a site that is secured with Windows Integrated Authentication and is challenged for credentials.  At this point Mr. Hacker opens a second connection to the server and sends the challenge that he received on Connection 1 back to the server in an effort to get a security token based on the servers credentials.  Again, I only have a really high level understanding of this so my explanation may not be technically precise.  On a server without Loopback checking (i.e. IIS 5), you could theoretically gain access to the server using this type of attack. 

Enter Loopback Checking.  In an effort to thwart reflection attacks, IIS 5.1 (XP SP2) and IIS 6 (W2K3 SP1) check any challenges made on the server for credentials to see where they originated from.  I am not privy to the actual implementation on this but it basically checks the source IP address of the challenge and if it is an IP address from the local machine it checks to see if the host name for the challenge matches the machines host name.  If the host name doesn't match the machine name, then the challenge fails which gives us the access denied errors we see in our apps. 

"Ok so how does this affect my application and why am I getting access denied", you say.  Most likely your application is doing some type of activity that requires it to authenticate (doing a CreateObjectEx to instantiate a COM component is where I have see this quite a bit).  You would also have to be using a host-name other than the machine name to access this in order for the problem to occur.  For example, let's say you have an ASP application on a server named WEBSERVER1.  Let's also say that you have created a host-name for the site called KillerApp.CompanyA.com so that users can access the site using this URL.  If your ASP application tries to instantiate a COM component on this machine using CreateObjectEx("MyProg.MyClass", KillerApp.CompanyA.com) and you are using Windows Integrated Authentication, you would get the access denied error.  Why? Good Question.  IIS sees a challenge for credentials coming from one of the local loopback ip's but the host-name is KillerApp.CompanyA.com.  IIS/Windows Security thinks this challenge is being reflected because it's not from WEBSERVER1 so it doesn't grant the token.

The good news is that this is fairly easy to spot and even easier to fix.  To find out if this is what is causing your access denied.  Turn on Audit Logon Events and Audit account logon events for failures in the Local Security Settings - Local Policies - Audit Policy.  Reproduce the access denied error and then check your security event log on the server.  Notice the "strange" characters for the "Logon Process in the screen shot below.  This is a dead giveaway that you are failing due to the loopback check feature. 

There are two methods listed in the KB listed below that you can use to get around this problem.  I recommend method two, since it will keep your attack surface area much smaller.  Method one, completely disables the loopback check.  You could also get around this by using the machine name instead of the FQDN. 

896861 You receive error 401.1 when you browse a Web site that uses Integrated Authentication and is hosted on IIS 5.1 or IIS 6
http://support.microsoft.com/default.aspx?scid=kb;EN-US;896861

HTTP/1.1 200 OK

Server Unavailable, Page Cannot be displayed after reinstalling IIS 6 on W2K3 SP1

Today I'd like to cover another issue that I have seen quite a few times in the past several weeks and one that has a very simple fix.  The problem is that IIS will not serve ASP and ASP.NET pages after reinstalling IIS.  The error that I most frequently see is "Server Unavailable" but it can also show up as "Page Cannot be Displayed".  Browsing regular .htm pages works fine and a check of the Web Service Extensions in IIS Manager shows that ASP and ASP.NET are both installed and enabled.  The problem typically occurs after users decide to reinstall IIS to "fix" some other unrelated issue on their web server.  I have seen this one frequently occur when users in a last ditch effort, have reinstalled IIS after having problems with Outlook Web Access. 

The issue is cause by a the installation of "mismatched" dlls when the reinstall occurs.  Typically during the installation of IIS, the installer will prompt for some additional files from the install media.  If the RTM (release to manufacturing a.k.a your original Windows install disk) is used instead of the SP1 media for this, the wrong versions (RTM instead of SP1) of several important dlls will be installed.  This breaks ASP pages and causes the issue.  A quick check of the properties of w3core.dll in the %SystemRoot%\system32\inetsrv folder will let you know if you are in this broken state or not (see screen shot below).  If your version of w3core.dll is 6.0.3790.0 then you are running the RTM version and you have a mismatched dll.  Windows 2003 SP1 should have a build of 1830. 

As you can see in the screenshot above, this machine has the correct version of w3core.dll. 

The resolution for this issue is simple.  Reapply SP1 and reboot the server. After this has completed ASP and ASP.NET pages should serve correctly.  Another quick check of w3core.dll should show confirm that we know have the right version of the dlls.  For more information on this issue, visit the KB link below. 

907275 IIS fails to serve ASP pages after you install IIS on Windows Server 2003 Service Pack 1
http://support.microsoft.com/default.aspx?scid=kb;EN-US;907275

HTTP/1.1 200 OK

Access Denied 401.1 goes away temporarily after IISRESET

I ran into an interesting issue the other day while helping one of our Support Engineers from Exchange.  The customer was getting a 401.1 response when trying to browse a site with anonymous access only.  Initially the SE from Exchange and I both had the same hypothesis that the IUSR account password was out of sync and a simple resetting of the password should get things up and going again.  Following the steps in my previoius post (see http://blogs.msdn.com/jiruss/archive/2006/05/24/606107.aspx) we proceeded to reset the IUSR account password.  I went to IIS manager and right clicked on the site and selected browse expecting things to gloriously start working (as they usually do) but much to my surprise we were presented with the same error again.  Hmmmm.

At this point, I decided to check the machines local security policy to make sure the correct user rights were in place to allow our IUSR account to log in.  AHA! The IUSR is missing the "Access this computer from the network" right.  I proceeded to explain to the customer that in order for the IUSR account to be able to log on, we would need to add this right.  The right was currently being pushed out via a domain policy so, we would need to move this machine into a different OU or stop pushing out this right as part of domain policy.  It was at this point the customer said "well why does it work without this right if I reset IIS?".  This one had me stumped for a bit.  I spoke with one of my colleagues about this and he mentioned that he had seen IISRESET cause the IUSR account to start working in IIS for a while but he wasn't sure why.  Not satisfied with not understanding this and still not 100% convinced that this could be, I did an IISRESET on the box.  What do you know, I can now browse using my IUSR account.  What?  How can an IISRESET get around this policy? 

Well, as it turns out, IISRESET is not "getting around" the policy, but is modifying the policy.  I inspected the local security policy after doing the reset and sure enough, our IUSR account had been added to the "Access this computer from the network right".  In this case the customer was reseting IIS which was resetting the local policies needed for IIS to function properly but when the domain policy was being pushed out every night our rights were being removed and causing the IUSR logon to fail. 

The moral of the story here is that IISRESET not only resets the IIS admin service and worker processes and recreates the default accounts but also resets the default permissions in the local security policy database. 

HTTP/1.1 200 OK

Password complexity in ASP.NET 2.0

Well, I must admit that my first stab at creating a simple ASP.NET 2.0 site that takes  advantage of the new Membership features was wrought with peril.  Actually it wasn't that bad but I did have a little trouble when I was attempting to add a new user to my site using the Security Wizard.  The problem was with the password complexity being required by the AspNetSqlMembershipProvider.  At first I couldn't figure out where this "policy" was coming from.  After a little searching I found that this is set in the machine config file.  Instead of rewriting another blog post that addresses the issue in great detail, I decided to simply provide a link to Scott Forsyth's blog on the subject.  See below

http://weblogs.asp.net/owscott/archive/2005/05/11/406550.aspx

HTTP/1.1 200 OK

IUSR_ account password out of sync. (Help OWA is not working!)

After seeing this same issue 4 times in the last 2 days, I thought it would be a good idea to post it along with the very simple and quick solution. 

First, I would like to clear up what seems to be a common misconception among IIS administrators out there.  The password field for the anonymous access account under authentication methods does not change the password for the anonymous user account in the machine’s SAM database.  The field is there so you can enter the password for the anonymous user account should you decide to use an account other than the IUSR account.  The password field is stored in the IIS metabase in encrypted form and must match the password for the account in the SAM database.  Second, IIS6 does not enable Subauthentication used in earlier versions of IIS by default because of the inherent security risks.  Basically, the subauthentication component in IIS5, allowed IIS to manage the passwords for the anonymous user accounts and keep them in sync with the SAM database.  Because the Iissuba.dll could be used maliciously to change passwords on other accounts, IIS6 does not enable it.  For more information on
Subauthentication see
http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/bda0c6e5-ae71-463f-be27-f85dafa776b2.mspx?mfr=true. 

Now for the issue:
When the password for the IUSR account stored in the metabase doesn’t match the password for the account on the machine, we have PROBLEMS!  When we have this condition and a user tries to browse the site anonymously, they will be presented with a lovely 401.1 status code.  Users of OWA (Outlook Web Access) will also be affected and will see a screen similar to the one in Figure A.  The reason this happens is because the images and other resources that OWA uses are located in the ExchWeb virtual directory, which by default is set for anonymous access.  Since our anonymous user (IUSR_machinename) cannot be authenticated, none of the images are accessible and downloaded and you end up with a “broken” OWA application.  If you were to check your IIS log files you would see numerous 401.1 status codes for GET requests on items in the ExchWeb virtual directory.

Figure A.

Now for the fix:
The first step we need to take is to make a simple modification to the adsutil.vbs script so that we can see the clear text version of the password.  Open a comand prompt and navigate to <drive>:\inetpub\adminscripts.  Type notepad adsutil.vbs at the command prompt and hit enter.  This will open the adsutil.vbs script in notepad.  Now go to the Edit - Find menu and enter "IsSecureProperty = True" and hit enter.  Change the "True" to "False" and save the file.  You can close notepad now and return to the command window. 

Now we need to make sure that we are only setting the password for the IUSR account in a single location in the Metabase.  Open a command prompt and navigate to the C:\inetpub\adminscripts directory. 

Enter the following command:
cscript adsutil.vbs find anonymoususerpass and press enter. 

This will return a listing of all the nodes where the anonymoususerpass appears in the metabase.  Ideally you should see this in only the W3SVC node (it may also appear in the MSFTPSVC node if you are running FTP).  If you see any locations “under” W3SVC (i.e. W3SVC/1/root) then we should delete these entries.

To delete an entry use the following syntax:
cscript adsutil.vbs delete W3SVC/1/root/anonymoususerpass

Once we have made sure that we only have the password set at a single location, we need to “sync” the password with the one in the SAM database.  The easiest way to do this is to simply open Computer Management if you are on a member server or Active Directory Users and Computers on a domain server and do a change password on the IUSR account.  Then back at our command prompt, we simply set the password to match the one we just set by using the following command:

cscript adsutil.vbs set W3SVC/anonymoususerpass "passwordgoeshere"

Viola! The passwords should now be in sync and the site should now be accessible.  If you would prefer to attack this from the other direction and set the password in the SAM for the IUSR account to match the one assigned in the Metabase, you need to edit the adsutil.vbs file.  Open adsutil.vbs in notepad and find the function called “IsSecureProperty”.  Find the line IsSecureProperty = True and change it to IsSecureProperty = False.  Then run the following command:

cscript adsutil.vbs get W3SVC/anonymoususerpass

This will return the password for the anonymous user that we currently have stored in the database.  Simply copy this value and then do a change password on the IUSR account in ADUC (Active Directory Users and Computers) or Computer Management and paste in this value.  Again the passwords are now in sync and everything should be working properly.  

HTTP/1.1 200 OK

Default Domain setting only applies when using Basic Authentication

I had a case yesterday where my customer was complaining that the Default Domain setting wasn’t working for him.  For those of you who may not be aware, this setting allows the site admin to set the default domain users will be authenticated against when using Basic Authentication.  This provides a nice shortcut for users of the site, so that they are not forced to type in, or for that matter remember, the domain that their user account exists in. 

In the case of my customer, they were accessing a SharePoint site and wanted to use the same URL for both internal and external users.  The problem with this approach was that for internal users, they wanted to use Windows Integrated authentication to prevent them from having to enter credentials to access the site.  Within IIS the customer had both Windows Integrated and Basic Authentication enabled.  When my customer tried to access their site from outside their network, they were prompted for credentials (as expected) but entering only the username and password simply resulted in the dialog box being redisplayed with the DNS domain name pre-pended to the username.  The customer wanted to know why the correct “Default Domain” wasn’t working.

In this case the customer was using IE as the browser.  As you may have already guessed, the issue wasn’t with the Default Domain not working but the authentication method that was being used.  IE will always select Windows Integrated over Basic when both are enabled for a site, so in this case the customer thought he was using Basic but was actually using Integrated.  Since the Default Domain setting only applies to Basic, the customer was not able to log on only using the username and password.

My suggestion to the customer was to set up a second virtual directory or site for his internal users and point it to the same content and set the authentication to Windows Integrated and then remove Windows Integrated from the external site.  The only caveat here is that you would have two different URLs to remember (unless you used some sort of  proxy).

 HTTP/1.1 200 OK

Inaugural Post!!!

Ultimately, the goal here is to provide a good resource that my customers and colleagues can use to make their lives easier.  I hope during our journey you will find the postings useful and informative

HTTP/1.1 200 OK