Troubleshooting HTTP 401.3 errors with Process Monitor

Last week I posted the following blog which showed how to use Process Monitor to troubleshoot service startup issues.

http://blogs.msdn.com/webtopics/archive/2009/06/16/troubleshooting-service-startup-issues-with-process-monitor.aspx

To continue on that topic, I ran across another issue recently where Process Monitor was again very helpful in troubleshooting.

Problem – When browsing ASP pages, we were getting below error in the browser. Browsing any HTML page worked fine.

HTTP Error 401.3 - Unauthorized: Access is denied due to an ACL set on the requested resource.

Note that you may need to uncheck Show friendly HTTP error messages in Internet Explorer on the Advanced tab under Tools/Internet Options as shown below to see this exact error.

It is good practice to leave this option unchecked when you’re troubleshooting an issue that involves an error message in the browser, since you’ll be more likely to get a full detailed error message. You could also check the IIS logs and look at the sc-status and sc-substatus fields to see 401 and 3 respectively.

Now I captured a Process Monitor log while reproducing the issue. As usual, I started by filtering on Result is Access Denied in Process Monitor after reproducing the issue. I found that there were no related Access Denied entries. You would expect that Process Monitor would capture an Access Denied result with an ACL (Access Control List) error like HTTP 401.3.

I found that this can occur when the account the IIS worker process was launched as has access to a particular file, but the user that authenticated through website does not. To see this in Process Monitor, I first filtered on w3wp.exe for the process name, which is the worker process that hosts a web site in IIS 6.0.

I then looked through the Process Monitor output for the access to C:\WINDOWS\Help\iisHelp\common\401-3.htm, which is the file IIS reads in by default to display the 401.3 error in the browser. Once I found that, I looked for what showed up in the log just before then.

The last thing that was accessed before the 401-3.htm file was asp.dll. Checking the permissions on that file and comparing it to a default installation, I found the permissions were the same.

See the entry highlighted above that shows a successful access to asp.dll for a QuerySecurityFile operation. The Detail column shows the information queried was the Owner, Group, and DACL (Discretionary Access Control List). Basically what’s going on here is the w3wp.exe process accesses asp.dll using the process identity (NETWORK SERVICE by default). We don’t see an Access Denied result because the process identity has the appropriate permissions. Now it checks to make sure that whatever user browsed the site also has access to asp.dll. In this case, Anonymous Authentication was enabled so the IUSR_MachineName account was checked, which did not have the appropriate permissions.

The issue here turned out to be that the NT AUTHORITY\Authenticated Users group was removed from the local Users group. Adding it back and then restarting IIS resolved the issue. The point here is that using Process Monitor I was able to narrow the issue down to the user accessing the site not having permissions on a specific DLL. In this case, IUSR_MachineName was an Authenticated Users, which should be a member of the Users group, and Users have permissions on asp.dll.

Summary

1. Capture Process Monitor log while reproducing issue
2. Filter on Access Denied
3. No Access Denied, so filter on w3wp.exe
4. Look for access to 401-3.htm
5. Review entries before the 401-3.htm to determine what file was accessed last
6. Check permissions based on successful QuerySecurityFile operation for last file accessed (in this case was asp.dll)

References

Below are a couple of articles we use frequently for HTTP 401.3 and other permissions related issues with IIS.

Default permissions and user rights for IIS 6.0
http://support.microsoft.com/kb/812614

Troubleshooting HTTP 401 errors in IIS
http://support.microsoft.com/kb/907273

How to Add a Locked Header Row to an ASP.NET GridView Control

The GridView control is often used to display tabular data, much like an Excel spreadsheet. However, unlike Excel, the GridView control doesn't have any automatic way of locking the header row so that it doesn't scroll out of view.

Check out this example of a GridView within a DIV with the overflow-Y property set to scroll. Notice that as you scroll the GridView, the header row scrolls out of view. It would be more convenient to have a locked header row so that the header row is always visible regardless of the scoll position of the GridView. It would also be nice to have a header row that dynamically configured itself as per the data source so that if we add a new field to the GridView, a corresponding column automatically gets added to the locked header row.

Here's another example, but this time, the header row stays locked at the top of the GridView so that you can scroll without losing track of column names. There are a few steps required to implement this functionality.

1. Create a DIV on the page to contain the header row and set the DIV (either statically or dynamically) to the same width as the DIV that contains the GridView.
2. Insert an ASP.NET Table control into the header DIV.
3. In Page_Init, dynamically add the necessary cells to the header table and set the appropriate properties so that it matches the appearance of the rest of the GridView.
4. Check for sortable columns and dynamically add sorting LinkButtons to the header row.
5. Add code to handle the LinkButton's Click event and sort appropriately.

Let's go through each of these steps. (If you'd prefer to work with a working project, you can download the project at the end of this post.)

Creating the Header DIV and Inserting an ASP.NET Table Control (Steps 1 and 2)

To create a locked header row, we need to disable the header for the GridView control and instead dynamically create our own header.

1. Select the GridView control and change the ShowHeader property to false.
2. Create a new DIV above the DIV containing the GridView control using the following code.

<!-- *** Begin Header Table *** -->
<div id="DivHeader" style="vertical-align:top;width:650px;">
    <asp:Table ID="HeaderTable" runat="server"
               CellPadding="4"
               CellSpacing="0"
               Font-Size="11pt"
               BackColor="#274511"
               ForeColor="White">
    </asp:Table>
</div>
<!-- *** End Header Table *** -->

Note that the properties for my Table control were derived from the theme applied to my sample page. You can set these properties as you wish or dynamically set them so that they adjust to the GridView's properties automatically.

Notice that this Table control doesn't contain any cells. We'll add the necessary cells in Page_Init in the next step.

Add Code to Page_Init to Dynamically Create Header Cells and LinkButtons (Steps 3 and 4)

To dynamically add the cells for the header table, add the following code to Page_Init.

protected void Page_Init(object sender, EventArgs e)
{
    TableRow headerRow = new TableRow();

    for (int x = 0; x < GridView1.Columns.Count; x++)
    {
        DataControlField col = GridView1.Columns[x];

        TableCell headerCell = new TableCell();
        headerCell.BorderStyle = BorderStyle.Solid;
        headerCell.BorderWidth = 1;
        headerCell.Font.Bold = true;

        // if the column has a SortExpression, we want to allow
        // sorting by that column. Therefore, we create a linkbutton
        // on those columns.
        if (col.SortExpression != "")
        {
            LinkButton lnkHeader = new LinkButton();
            lnkHeader.PostBackUrl = HttpContext.Current.Request.Url.LocalPath;
            lnkHeader.CommandArgument = col.SortExpression;
            lnkHeader.ForeColor = System.Drawing.Color.White;
            lnkHeader.Text = col.HeaderText;
            lnkHeader.Click += new EventHandler(HeaderLink_Click);
            headerCell.Controls.Add(lnkHeader);
        }
        else
        {
            headerCell.Text = col.HeaderText;
        }

        headerCell.Width = col.ItemStyle.Width;
        headerRow.Cells.Add(headerCell);

    }

    HeaderTable.Rows.Add(headerRow);
}

This code adds a new TableCell to the header table and then sets specific properties on that TableCell based on whether or not the GridView's column has a SortExpression specified. If it doesn't, this code just sets the Text property of the cell to the GridView column's HeaderText property. Otherwise, it creates a LinkButton and sets the following properties on it.

• PostBackUrl - Set to the local URL of the current request.
• CommandArgument - Set to the SortExpression of the GridView's column so that we can apply it when the LinkButton is clicked.
• ForeColor - Set to White in my example, but you can set this dynamically if you wish to make the example more generic.
• Text - Set to the GridView column's HeaderText property.

I also add a new event handler for the Click event of the LinkButton control. Let's look at the code for that event.

Code for the LinkButton's Click Event

The code in the LinkButton's Click event applies the necessary SortExpression based on which LinkButton was clicked. The code for the Click event appears below.

protected void HeaderLink_Click(object sender, System.EventArgs e)
{
    LinkButton lnkHeader = (LinkButton)sender;
    SortDirection direction = SortDirection.Ascending;

    // the CommandArgument of each linkbutton contains the sortexpression
    // for the column that was clicked.
    if (GridView1.SortExpression == lnkHeader.CommandArgument)
    {
        if (GridView1.SortDirection == SortDirection.Ascending)
        {
            direction = SortDirection.Descending;
        }
    }

    GridView1.Sort(lnkHeader.CommandArgument, direction);
}

This code checks the CommandArgument for the LinkButton that was clicked (retrieved from the EventArgs) and applies the ascending or descending SortDirection based upon the result. It then calls the Sort method on the GridView.

AJAX-Enabling the Sample

If you are using ASP.NET AJAX, the code as-is will not do a partial postback. In order to have the LinkButtons sort using an AJAX async postback, you need to replace the line of code that sets the PostBackUrl property of the LinkButton with the following code.

lnkHeader.ID = "Sort" + col.HeaderText;

Doing this will allow ASP.NET Viewstate to track the LinkButton so that ASP.NET AJAX will work with the LinkButton.

Using this example, you can easily apply a locked header row to your GridView controls. If you'd like the completed sample project, you can download it below.

Sample Project

Size: 12KB
Download Now

Note: This download is hosted on my personal site, JimcoBooks.com.

Getting access denied error when querying for user Roles using Authorization Role Manager

Recently I was working on an AuthorizationRoleProvider issue for an ASP.Net application. Customer was using Authorization Role Manager for the Role management of the Active Directory users.

Scenario
=======

You developed an Asp.net application using Authorization Role provider. We logged in as User1 into the application through Forms authentication. We used the following code:

string role="Admin";
if (Roles.IsUserInRole(“User2”,role))
{
//Do something
}

When we try to check if User2 is in role Admin we were getting the following error in the application:

Server Error in '/AzManTest' Application.
Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))
Exception Details: System.UnauthorizedAccessException: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))

Source Error:
Line 38: if (Roles.IsUserInRole(
Line 39: strUserName,
Line 40: role))

Stack Trace:

[UnauthorizedAccessException: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))]
[TargetInvocationException: Exception has been thrown by the target of an invocation.]
System.RuntimeType.InvokeDispMethod(String name, BindingFlags invokeAttr, Object target, Object[] args, Boolean[] byrefModifiers, Int32 culture, String[] namedParameters) +0
System.RuntimeType.InvokeMember(String name, BindingFlags bindingFlags, Binder binder, Object target, Object[] providedArgs, ParameterModifier[] modifiers, CultureInfo culture, String[] namedParams) +337
System.Type.InvokeMember(String name, BindingFlags invokeAttr, Binder binder, Object target, Object[] args, CultureInfo culture) +36
System.Web.Security.AuthorizationStoreRoleProvider.CallMethod(Object objectToCallOn, String methodName, Object[] args) +183
System.Web.Security.AuthorizationStoreRoleProvider.GetClientContextFromName(String userName) +167
System.Web.Security.AuthorizationStoreRoleProvider.GetClientContext(String userName) +70
System.Web.Security.AuthorizationStoreRoleProvider.IsUserInRoleCore(String username, String roleName) +62
System.Web.Security.AuthorizationStoreRoleProvider.IsUserInRole(String username, String roleName) +93
System.Web.Security.Roles.IsUserInRole(String username, String roleName) +398
Login_Page.LoginCtrl_Authenticate(Object sender, AuthenticateEventArgs e) in e:\Inetpub\wwwroot\AzManTest\Login.aspx.cs:38
System.Web.UI.WebControls.Login.OnAuthenticate(AuthenticateEventArgs e) +108
System.Web.UI.WebControls.Login.AttemptLogin() +115
System.Web.UI.WebControls.Login.OnBubbleEvent(Object source, EventArgs e) +101
System.Web.UI.Control.RaiseBubbleEvent(Object source, EventArgs args) +37
System.Web.UI.WebControls.Button.OnCommand(CommandEventArgs e) +118
System.Web.UI.WebControls.Button.RaisePostBackEvent(String eventArgument) +166
System.Web.UI.WebControls.Button.System.Web.UI.IPostBackEventHandler.RaisePostBackEvent(String eventArgument) +10
System.Web.UI.Page.RaisePostBackEvent(IPostBackEventHandler sourceControl, String eventArgument) +13
System.Web.UI.Page.RaisePostBackEvent(NameValueCollection postData) +36
System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +1565

I opened the .net reflector and looked for the failing method AuthorizationStoreRoleProvider.GetClientContextFromName which is internally calling the Azman API method InitializeClientContextFromName.

private object GetClientContextFromName(string userName)
{
    string[] strArray = userName.Split(new char[] { '\\' });
    string str = null;
    if (strArray.Length > 1)
    {
        str = strArray[0];
        userName = strArray[1];
    }
    object[] args = new object[] { userName, str, null };
    return this.CallMethod(this._ObjAzApplication, "InitializeClientContextFromName", args);
}

We were able to reproduce this issue in our lab environment using the Microsoft.Interop.Security.AzRoles.dll assembly and were getting the same error.

Server Error in '/' Application.
Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))
Description: An unhandled exception occurred during the execution of the
current web request. Please review the stack trace for more information
about the error and where it originated in the code.
Exception Details: System.UnauthorizedAccessException: Access is denied.
(Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))
Source Error:
Line 31:
Line 32: clientContext =azApp.InitializeClientContextFromName("user2", "DM", null);

Cause and Resolution
================

We checked the documentation of this failing API method InitializeClientContextFromName in the following msdn article:

http://msdn.microsoft.com/en-us/library/aa377363(VS.85).aspx

Here is an excerpt from the above link:

“There are security requirements to call this API. The caller of the API must have READ permissions to the tokenGroupsGlobalAndUniversal attribute of the targeted user or the caller (USER) of the API should be part of the Pre-Windows 2000 compatible Access Group in the domain for this to succeed. The read access to the tokenGroupsGlobalAndUniversal attribute is granted to the Pre-Windows 2000 Compatible Access group, but new domains contain an empty Pre-Windows 2000 Compatible Access group by default because the default setup selection is Permissions compatible with Windows 2000 and Windows Server 2003”.

For example User1 is checking if User2 is part of any role then User1 should be part of the Pre-Windows 2000 compatible Access Group in the domain or must have READ permissions to the tokenGroupsGlobalAndUniversal attribute of the targeted user.

Useful articles
===========
InitializeClientContextFromName
http://msdn.microsoft.com/en-us/library/aa377363(VS.85).aspx

Some applications and APIs require access to authorization information on account objects
http://support.microsoft.com/kb/331951

Impersonation Shenanigans
http://blogs.msdn.com/azman/archive/2007/01/16/impersonation-shenanigans.aspx

Why cacheRolesinCookie does not work well with the Ajax Extensions 1.0 and RoleManager of .NET Framework 2.0?
http://blogs.msdn.com/webtopics/archive/2009/01/30/why-cacherolesincookie-does-not-work-well-with-the-ajax-extensions-1-0-and-rolemanager-of-net-framework-2-0.aspx

Intermittently System.Web.HttpExceptions(A field or property with the name ’<columnName> ‘ was not found on the selected data source) is thrown when the application is under heavy load

Working on another interesting case where we were occasionally failing with HttpException and after hitting refresh on the browser we were back in business.Next stop was to look at event viewer where bunch of unhandled exceptions like below were reported:

Event Type: Warning
Event Source: ASP.NET 2.0.50727.0
Event Category: Web Event
Event ID: 1309
Date:  5/18/2009
Time:  3:29:10 PM
User:  N/A
Computer: MachineName
Description:
Event code: 3005
Event message: An unhandled exception has occurred. Application information:
    Application domain: /LM/W3SVC/1/Root/Web-1-128871464389591756 Process information:
    Process ID: 3428
    Process name: w3wp.exe
    Account name: NT AUTHORITY\NETWORK SERVICE
Exception information:
    Exception type: HttpException
    Exception message: A field or property with the name 'Contact_Type' was not found on the selected data source.
Thread information:
    Thread ID: 1
    Thread account name: NT AUTHORITY\NETWORK SERVICE
    Is impersonating: False
    Stack trace:    at System.Web.UI.WebControls.BoundField.GetValue(Control controlContainer)
   at System.Web.UI.WebControls.BoundField.OnDataBindField(Object sender, EventArgs e)
   at System.Web.UI.Control.OnDataBinding(EventArgs e)
   at System.Web.UI.Control.DataBind(Boolean raiseOnDataBinding)
   at System.Web.UI.Control.DataBind()
   at System.Web.UI.Control.DataBindChildren()

OR sometime we get error message like:

Exception information:
    Exception type: HttpException
    Exception message: A field or property with the name 'Request_ID' was not found on the selected data source.

Thing to note here was that every time a different column name was specified in the message. This was a good indication to run SQL profiler trace and we found that we were executing a different query (which was of course not having the required columns) and it was not the one which was bound to the control(GridView). At this point question tickled my thought process as to why were the wrong columns returned by Dataset/DataSource or more importantly why was a different query being executed?

Further reviewing the code we figured out that we were having Data Access Layer which are actually Static Classes and all the SQLConnection And SQLCommand objects were declared as static in those classes.

private static void OpenSqlConnection()
private static void CreateCommand(string queryString,string connectionString)

Aha now that’s our culprit !

Static function would show up SQL query whatever was the last one executed. Now that reminds me of an excellent post by one of our Escalation Engineers, Brian Booth, which talks  about real world example which can lead to this unexpected results.

Static(c#) or Shared(vb.net) are global variable across App Domain and would be accessible throughout the process uptime. It’s value would be retain across pages and would be available to all the users!!!That means each user would be able see each other data….

Till then Bye bye…

Fix: UpdatePanel Async Postbacks Slow in Internet Explorer

We've seen a few issues recently where customers were experiencing very slow async postbacks via an UpdatePanel, but only in Internet Explorer. Other browsers worked fine. (When I say "very slow", I mean in the neighborhood of 30 seconds!) The cause for this isn't a mystery to us.

Because of the way our HTML viewer (mshtml.dll) was designed, Internet Explorer is slow to iterate through large DOM collections. As it happens, the PageRequestManager class has a method called _destroyTree that does just that. It uses a for loop to iterate through all of the DOM nodes in the UpdatePanel's DIV. It does this on the client (the browser) before the request to the Web server is ever kicked off, and if you have a large DOM in your UpdatePanel, you might see a very long delay in Internet Explorer when this occurs.

If you're experiencing this kind of problem, I have good news. We've fixed this for the next version of ASP.NET, and we also have some code that you can use in your applications now to mediate the problem. Add the following script block immediately before your closing </body> element on the page experiencing the problem. (If you prefer to add it elsewhere on the page, make sure that the script appears after the ScriptManager control.)

<script language="javascript" type="text/javascript">
function disposeTree(sender, args) {

  var elements = args.get_panelsUpdating();

  for (var i = elements.length - 1; i >= 0; i--) {

    var nodes = elements[i].getElementsByTagName('*');

    for (var j = 0; j < nodes.length; j++) {

      var node = nodes[j];
      if (node.nodeType === 1) {

        if (node.dispose && typeof (node.dispose) === "function") {
          node.dispose();
        }
        else if (node.control && typeof (node.control.dispose) === "function") {
          node.control.dispose();
        }

        var behaviors = Sys.UI.Behavior.getBehaviors(node);

        for (var k = behaviors.length - 1; k >= 0; k--) {
          behaviors[k].dispose();
        }
      }
    }
    elements[i].innerHTML = "";
  }
}

Sys.WebForms.PageRequestManager.getInstance().add_pageLoading(disposeTree);
</script>

After adding this code, you should no longer see the delay in your UpdatePanel's postbacks. If you do still see a delay, it's likely that you aren't encountering a problem due to the issue I've described in this article. In that case, give us a call and we can help you troubleshoot.

Update 6/23/2009:
This issue is now documented in our Knowledge Base at http://support.microsoft.com/?kbid=2000262.

Troubleshooting service startup issues with Process Monitor

Many things can cause a service, like IIS’s World Wide Web Publishing Service, to fail on startup. When troubleshooting such an issue, Process Monitor can be an invaluable tool. What Process Monitor does is monitor all File and Registry access on the system in real-time. The latest version of process monitor can be obtained here.

Most of the time, we use this tool to troubleshoot Access Denied related issues. In those scenarios, Process Monitor shows exactly what user account tried to access what file or registry key when the Access Denied error occurred.

Troubleshooting service startup issue when it’s not a simple Access Denied scenario

When the issue is not Access Denied, it can be difficult to track down the root cause using Process Monitor. Below is an issue I worked on recently that’s a perfect example of this.

Problem - When attempted to start the World Wide Web Publishing Service, a popup with the below error message appears:

Windows could not start the World Wide Web Publishing Service on Local Computer. For more information, review the System Event log. If this is a non-Microsoft service, contact the service vendor, and refer to service-specific error code -2147221164.

One or more of the following events are found in the System Event Log:

Event Type: Error
Event Source: W3SVC
Event Category: None
Event ID: 1036
Description:
A failure occurred while initializing the configuration manager for the World Wide Web Publishing Service. The data field contains the error number.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 80040154

Event Type: Error
Event Source: W3SVC
Event Category: None
Event ID: 1005
Description:
The World Wide Web Publishing Service is exiting due to an error. The data field contains the error number.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 80040154

Event Type: Error
Event Source: Service Control Manager
Event Category: None
Event ID: 7024
Description:
The World Wide Web Publishing Service service terminated with service-specific error 2147746132 (0x80040154).

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Here are the steps I took to capture the necessary information using Process Monitor:

1. Launch Process Monitor.
2. Run net start w3svc from a command prompt to reproduce the issue.

Microsoft Windows [Version 5.2.3790]
(C) Copyright 1985-2003 Microsoft Corp.

C:\>net start w3svc
The World Wide Web Publishing Service service is starting.
The World Wide Web Publishing Service service could not be started.

A service specific error occurred: 2147746132.

More help is available by typing NET HELPMSG 3547.

3. Click the magnifying glass icon () in Process Monitor to stop the capture.

Process Monitor logs a lot of data so it’s best to filter the output accordingly. To get an idea of where to start, I searched on error 80040154 from the event log entries and found this means “Class not registered”. Since “Class not registered” is caused by something missing in the registry, I looked for a Result of “NAME NOT FOUND” in the Process Monitor Output.

As you’ll quickly find out, “NAME NOT FOUND” occurs all the time. This is due to programs searching different locations for something until it finds it and a result of “SUCCESS” appears.

The way I found to make this easier to troubleshoot is to capture another Process Monitor log on a server where the World Wide Web Publishing Service starts successfully and then filter both logs for the specific process that is failing.

Here’s how to configure the filter for IIS 6.0 on Windows Server 2003.

1. In Process Monitor, click Filter/Filter…
2. Select “Command Line” and “is” for the first two drop-down lists.
3. Then enter C:\WINDOWS\System32\svchost.exe -k iissvcs and click Add as follows:

Comparing the output from both, I found the below difference:

On server where W3SVC fails

On server where W3SVC works

Here the HKEY_CLASSES_ROOT\AppID\{A9E69610-B80D-11D0-B9B9-00A0C922E750}\LocalService value was missing from the registry on the server where the service failed to start, but was present on the server where it succeeded.

Resolution

To resolve the issue, use regedit.exe to export the missing key from the working server and import it to the server that fails. Running IISRESET from a command prompt was also required here to resolve the issue.

Troubleshooting a simple error message using FREB

FREB (Failed Requests Tracing – formerly known as Failed Request Event Buffering) is one of the nicest features released with IIS7 which would come in handy during troubleshooting. It does request based tracing, and produces a log file (in .xml format) that shows events and notifications from the various modules that worked on the request during its lifecycle.

This is going to be a series of posts talking about using FREB during troubleshooting various scenarios. In this blog post, I’m going to cover few scenarios like troubleshooting a specific error message, and what to look for in the FREB output.

Troubleshooting a simple error code

First, let’s take an example of troubleshooting a simple straight forward error code – HTTP 400 Bad Request. Whenever you get HTTP 400 Bad Request, the first thing to check is to find out where the HTTP 400 is logged – is it in IIS W3SVC logs, or in the HTTPERR logs. In this example, it would be in the W3SVC log, and that tells us that the request is actually processed by the IIS pipeline, and returned with HTTP 400 to the client. Only if the 400 is in IIS logs, we would be able to use FREB for the troubleshooting. Below is the sample screenshot of the error message (when browsed from the server itself – using localhost):

Let’s follow the step 1 in this article to create a FREB rule (create for HTTP 400 error message) on that website, or application. Below is how my web.config section with a FREB rule for 400 Error code (under <system.webServer> section) look like:

<tracing>
  <traceFailedRequests>
    <remove path="*" />
    <add path="*">
      <traceAreas>
        <add provider="ASP" verbosity="Verbose" />
        <add provider="ASPNET" areas="Infrastructure,Module,Page,AppServices" verbosity="Verbose" />
        <add provider="ISAPI Extension" verbosity="Verbose" />
        <add provider="WWW Server" areas="Authentication,Security,Filter,StaticFile,CGI,Compression,Cache,RequestNotifications,Module,Rewrite,RequestRouting" verbosity="Verbose" />
      </traceAreas>
      <failureDefinitions timeTaken="00:00:00" statusCodes="400" />
    </add>
  </traceFailedRequests>
</tracing>

Now, let us try to browse the page which gives you the HTTP 400 error message. Unless you have turned the IIS HTTP error pages ON, you should see the complete error message when you browse the page from the server itself. By default, IIS HTTP error pages are set to “Detailed errors for local requests and custom error pages for remote requests”. In my example, since it is a local request, and we can see the complete error message.

After browsing the page, you should see the FREB log file created under the %SystemDrive%\inetpub\logs\FailedReqLogFiles\W3SVC<#ID> folder. Below is how it looks after you open this. This is an XML file, and consumes a powerful freb.XSL located in the same folder.

I’ve highlighted the part of the file which you should look at first. Under “Errors & Warnings” you will see the information about the event which in fact set the status of the response to what this trace has been triggered on. The interesting thing to look for is the “Module Name” which has set the status. In this case, we see “MyModule” as the Module Name. If you click on that entry, you will see more information about that event:

Click on the “Compact View” at the top, and you will find all the events which arose during the lifetime of that particular request.

As you see, this module has registered for the RELEASE_REQUEST_STATE notification, and it does set the Response status to 400 in its handler for that notification (In ASP.NET, it is context.ReleaseRequestState Event Handler).

So, now we know that this HTTP 400 is set by a managed module called “MyModule” and its class name is “MyBadModule” (see the trace entry #176). Now, you can go ahead and try debugging that module. In the future blog posts, I will try to cover some more troubleshooting scenarios using FREB.

Happy FREBing!

Why System.Net.WebException comes up post .NET Framework 3.5 SP1?

In the recent past I have come across several issues where our customers have started running into System.Net.WebException while trying to access a web resource (web service, webpage etc) hosted locally on the same machine using some HostName. This issue started coming up after installing SP1 for .NET Framework 3.5 . The exception details are similar to the following:

[WebException: The remote server returned an error: (401) Unauthorized.]
   System.Net.HttpWebRequest.GetResponse() +5313085
   ASP.default_aspx.btnCallService_Click(Object sender, EventArgs e) +60
   System.Web.UI.WebControls.Button.OnClick(EventArgs e) +111
   System.Web.UI.WebControls.Button.RaisePostBackEvent(String eventArgument) +110
   System.Web.UI.WebControls.Button.System.Web.UI.IPostBackEventHandler.RaisePostBackEvent(String eventArgument) +10
   System.Web.UI.Page.RaisePostBackEvent(IPostBackEventHandler sourceControl, String eventArgument) +13
   System.Web.UI.Page.RaisePostBackEvent(NameValueCollection postData) +36
   System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +1565

After investigating this further we found that while accessing web-resource NTLM Authentication was being used. To reproduce the issue, create a simple Test Web-Site running on some different port. Also add a hostname to the site as www.test.com and add an entry for it in your Host file (located at C:\Windows\System32\drivers\etc). Add a simple asp page in this Test Web Site which displays the current time. The code for it will look like below -

<%=Now()%>

In the Default Web Site add a page called default.aspx. The code for that will look like below -

<%@ Page Language="C#" %>
<%@ Import Namespace="System.IO" %>
<%@ Import Namespace="System.Net" %>
<script runat="server">

    protected void btnCallService_Click(object sender, EventArgs e)
    {
        WebRequest request = WebRequest.Create("http://www.test.com:81/Time.asp");
        request.Method = "GET";
        request.Credentials = CredentialCache.DefaultNetworkCredentials;
        WebResponse response = request.GetResponse();
        StreamReader reader = new StreamReader(response.GetResponseStream());
        string responseFromServer = reader.ReadToEnd();
        Response.Write (responseFromServer);
        response.Close();
    }
</script>
<body>
    <form id="form1" runat="server">
        <asp:Button ID="btnCallService" runat="server" onclick="btnCallService_Click" 
            Text="Invoke Web Service" />
    </form>
</body>
</html>

Now browse the Default.aspx and click on button and expected behavior is it should show the current time in the browser. But instead we end up getting exception mentioned above.

The reason for the exception is post .Net Framework 3.5 SP1 Windows authentication defaults to NTLM authentication when the CustomTargetNameDictionary property is not set. NTLM authentication includes a challenge issued by the destination computer and sent back to the client computer. When a computer receives a challenge generated by itself, the authentication will fail resulting in exception mentioned as above unless the connection is a loop back connection. You can find additional details in the article.

There are two possible methods for a server application to work around this change. The recommended approach is to map the host name used in the request URL to the BackConnectionHostNames key in the registry on the server. The BackConnectionHostNames registry key is normally used to map a host name to a loopback address. To specify the host names that are mapped to the loopback address and can connect to Web sites on a local computer, follow these steps -
1. Click Start, click Run, type regedit, and then click OK.
2. In Registry Editor, locate and then click the following registry key:
    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa\MSV1_0
3. Right-click MSV1_0, point to New, and then click Multi-String Value.
4. Type BackConnectionHostNames, and then press ENTER.
5. Right-click BackConnectionHostNames, and then click Modify.
6. In the Value data box, type the host name or the host names for the sites (the host name used in the request URL) that are on the local computer, and then click OK. In our case it would be www.test.com.
7. Quit Registry Editor, and then restart the IISAdmin service and run IISReset.

A less secure work around is to disable the loop back check, as described in this article. This disables the protection against reflection attacks. So it is better to constrain the set of alternate names to only those you expect the machine to actually use.

I hope this helps!

Troubleshooting System.OutOfMemoryExceptions in ASP.NET

When the .NET Framework was first released, many developers believed the introduction of the garbage collector meant never having to worry about memory management ever again. In fact, while the garbage collector is efficient in managing memory in a managed application, it's still possible for an application's design to cause memory problems.

One of the more common issues we see regarding memory involves System.OutOfMemoryExceptions. After years of helping developers troubleshoot OutOfMemoryExceptions, we've accumulated a short list of the more common causes of these exceptions. Before I go over that list, it's important to first understand the cause of an OutOfMemoryException from a 30,000 foot view.

What Is an OutOfMemoryException?

A 32-bit operating system can address 4GB of virtual address space, regardless of the amount of physical memory that is installed in the box. Out of that, 2GB is reserved for the operating system (Kernel-mode memory) and 2GB is allocated to user-mode processes. The 2GB allocated for Kernel-mode memory is shared among all processes, but each process gets its own 2GB of user-mode address space. (This all assumes that you are not running with the /3gb switch enabled.)

When an application needs to use memory, it reserves a chunk of the virtual address space and then commits memory from that chunk. This is exactly what the .NET Framework's garbage collector (GC) does when it needs memory to grow the managed heaps. When the GC needs a new segment for the small object heap (where objects smaller than 85K reside), it makes an allocation of 64MB. When it needs a new segment for the large object heap, it makes an allocation of 32MB. These large allocations must be satisfied from contiguous blocks of the 2GB of address space that the process has to work with. If the operating system is unable to satisfy the GC's request for a contiguous block of memory, a System.OutOfMemoryException (OOM) occurs.

There are two reasons why you might see an OOM condition.

1. Your process is using a lot of memory (typically over 800MB.)
2. The virtual address space is fragmented, reducing the likelihood that a large, contiguous allocation will succeed.

It's also possible to see an OOM condition due to a combination of 1 and 2.

Let's examine some of the common causes for each of these two reasons.

Common Causes of High Memory

When your worker process approaches 800MB in private bytes, your chances of seeing an OOM condition begin to increase simply because the chances of finding a large, contiguous piece of memory within the 2GB address space begin to decrease significantly. Therefore, you want to avoid these high memory conditions.

Let's go over some of the more common causes of high memory that we see in developer support at Microsoft.

Large DataTables

DataTables are common in most ASP.NET applications. DataTables are made up of DataRows, DataColumns, and all of the data contained within each cell. Large DataTables can cause high memory due to the large number of objects that they create.

The most common cause of large DataTables is unfiltered data from a back-end data source. For example, if your site queries a database table containing hundreds of thousands of records and your design makes it possible to return all of those records, you'll end up with a huge amount of memory consumed by the result set. The problem can be greatly exacerbated in a multi-user environment such as an ASP.NET application.

The easiest way to alleviate problems like this is to implement filtering so that the number of records you return is limited. If you are using a DataTable to populate a user-interface element such as a GridView control, use paging so that only a few records are returned at a time.

Storing Large Amounts of Data in Session or Application State

One of the primary considerations during application design is performance. Developers can come up with some ingenious ways to improve application performance, but sometimes at the expense of memory. For example, we've seen customers who stored entire database tables in Application state in order to avoid having to query SQL Server for the data! That might seem like a good idea at first glance, but the end result is an application that uses an extraordinary amount of memory.

If you need to store a lot of state data, consider whether using ASP.NET's cache might be a better choice. Cache has the benefit of being scavenged when memory pressure increases so that you don't end up in trouble as easily.

Running in Debug Mode

When you're developing and debugging an application, you will typically run with the debug attribute in the web.config file set to true and your DLLs compiled in debug mode. However, before you deploy your application to test or to production, you should compile your components in release mode and set the debug attribute to false.

ASP.NET works differently on many levels when running in debug mode. In fact, when you are running in debug mode, the GC will allow your objects to remain alive longer (until the end of the scope) so you will always see higher memory usage when running in debug mode.

Another often unrealized side-effect of running in debug mode is that client scripts served via the webresource.axd and scriptresource.axd handlers will not be cached. That means that each client request will have to download any scripts (such as ASP.NET AJAX scripts) instead of taking advantage of client-side caching. This can lead to a substantial performance hit.

Running in debug mode can also cause problems with fragmentation. I'll go into more detail on that later in this post. I'll also show you how you can tell if an ASP.NET assembly was compiled with debug enabled.

Throwing a Lot of Exceptions

Exceptions are expensive when it comes to memory. When an exception is thrown, not only does the GC allocate memory for the exception itself, the message of the exception (a string), and the stack trace, but also memory needed to store any inner exceptions and the corresponding objects associated with that exception. If your application is throwing a lot of exceptions, you can end up with a high memory situation quite easily.

The easiest way to determine how many exceptions your application is throwing is to monitor the # of Exceps Thrown / sec counter in the .NET CLR Exceptions Performance Monitor object. If you are seeing a lot of exceptions being thrown, you need to find out what those exceptions are and stop them from occurring.

Regular Expression Matching of Very Large Strings

Regular expressions (often referred to as regex) represent a powerful way to parse and manipulate a string by matching a particular pattern within that string.  However, if your string is very large (megabytes in size) and your regex has a large number of matches, you can end up in a high memory situation.

The RegExpInterpreter class uses an Int32 array to keep track of any matches for a regex and the positions of those matches. When the RegExpInterpreter needs to grow the Int32 array, it does so by doubling its size. If your use of regex creates a very large number of matches, you’ll likely see a substantial amount of memory used by these Int32 arrays.

What do I mean by “large number of matches”? Suppose you are running a regex against the HTML from a page that is several megabytes in size. (You might think that this isn’t a feasible scenario, but we have seen a customer do this with HTML code that was over 5MB!) Suppose also that the regex you are using against this HTML is as follows.

<body(.|\n)*</body>

This regex does the following:

• “<body” matches the literal characters “<body”.
• The parenthesis tells the regex engine to match the regex within them and store the match as a back-reference.
• The dot (.) will match any single character that is not a line break.
• The “\n” will match any character that is a line break.
• The “*” repeats the regex in parenthesis between zero and an unlimited number of times. It also indicates a greedy match, meaning that it will match as many times as possible within the string.
• “</body>” matches the literal characters “</body>”.

In other words, if you use this regex against the HTML code from a page, it will match the entire body of the page. It will also store that body as a back-reference. The result is a very large Int32 array.

Incidentally, this problem isn’t specific to our implementation of regex. This same type of problem will be encountered with any regex engine that is NFA-based. The solution to this problem is to rethink the architecture so as to avoid such large strings and large matches.

Common Causes of Fragmentation

Fragmentation is problematic because it can cause allocations of contiguous memory to fail. Assume that you have only 100MB of free address space for a process (you’re almost certain to have much more than that in real life) and one 4KB DLL loaded into the middle of that address space as shown in Figure 1. In this scenario, an allocation that requires 64MB of contiguous free space will fail with an OOM exception.

Figure 1 – Fragmented Address Space

The following are common causes of fragmentation.

Running in Debug Mode

One of the features in ASP.NET that is designed to avoid fragmentation is a feature called batch compilation. When batch compilation is enabled, ASP.NET will dynamically compile each folder of your application into a single DLL when the application is JITted. If batch compilation is not enabled, each page and user control is compiled into a separate DLL that is then loaded into the address space for the process. Each of these DLLs is very small, but because they are loaded into a non-specific address in memory, they tend to get peppered all over the address space. The result is a radical decrease in the amount of contiguous free memory, and that leads to a much greater probability of running into an OOM condition.

When you deploy your application, you need to make sure that you set the debug attribute in the web.config file to false as follows.

<compilation debug="false" />

If you'd like to ensure that debug is disabled on your production server regardless of the setting in the web.config file, you can use the <deployment> element introduced in ASP.NET 2.0. This element should be set in the machine.config file as follows.

<configuration>
    <system.web>
        <deployment retail="true" />
    </system.web>
</configuration>

Adding this setting to your machine.config file will override the debug attribute in any web.config file on the server.

When debugging is enabled, ASP.NET will add a Debuggable attribute to the assembly. You can use .NET Reflector or ildasm.exe to examine an ASP.NET assembly and determine if it was compiled with the Debuggable attribute. If it was, debugging is enabled for the application.

Figure 2 shows two ASP.NET assemblies from the Temporary ASP.NET Files folder opened in .NET Reflector. The top assembly is selected and you can see that the Debuggable attribute is highlighted in red. (In order to see the manifest information in the right pane, right-click the assembly and select Disassemble from the menu.) The application running this assembly is running in debug mode.

Figure 2 - .NET Reflector showing an assembly compiled with debug enabled.

Figure 3 shows .NET Reflector with the second assembly selected. Notice that this assembly doesn't have a Debuggable attribute. Therefore, the application running this assembly is not running in debug mode.

Figure 3 - .NET Reflector showing an assembly compiled without debug enabled.

Generating Dynamic Assemblies

Another common cause of fragmentation is the creation of dynamic assemblies. Dynamic assemblies fragment the address space of the process for the same reason that running in debug mode does.

Instead of going into the details here on how this happens, I'll point you to my colleague Tom Christian's blog post on dynamic assemblies. Tom goes into detail on what can create dynamic assemblies and how to work around those issues.

Resources

The following resources are helpful when tracking down memory problems in your application.

Gathering Information for Troubleshooting

Tom Christian’s blog post on gathering information for troubleshooting an OOM condition will help you if you need to open a support incident with us. Read more from Tom in this post.

Post-mortem Debugging of Memory Issues

Tess Ferrandez is famous for her excellent blog on debugging ASP.NET applications. She’s accumulated quite a collection of excellent posts on memory issues that includes everything from common memory problems to case studies that include debugging walkthroughs with Windbg. You can find Tess’s 21 most popular blog posts in this post.

Using DebugDiag to Troubleshoot Managed Memory

Tess has also recently published a blog post that includes a DebugDiag script that she wrote for the purpose of troubleshooting managed memory problems. The great thing about using DebugDiag with this script is that you can simply point it to a dump file of your worker process and it will automatically tell you a wealth of information that can help you track down memory usage.

You can find out how to use Tess’s script and download a copy of it here.

Understanding GC

If you ever wanted to know how the .NET garbage collector works, Tess can help! She wrote a great blog post that includes links to other great GC resources, and you can read it here.

“I Am a Happy Janitor!”

Maoni, a developer on the Common Language Runtime team, wrote a blog post that explains how the garbage collector works using the colorful analogy of a janitor. Read Maoni’s enlightening post here.

Using GC Efficiently

Maoni also wrote an excellent series on using GC efficiently in order to prevent memory issues. You can read the series here.

Conclusion

I hope that this information will help you to identify memory problems in your ASP.NET application that can lead to OutOfMemoryExceptions. However, if you have exhausted these ideas and are still plagued with memory problems, contact us and open a support ticket. We'll be happy to help you troubleshoot!

3 Simple Steps for Configuring an SPN for your Website

The following article is an excellent reference when trying to figure out how to set SPN’s for your IIS Site.

http://support.microsoft.com/kb/929650

I wanted to focus on the SPN helper script in the article, and how it can be used in a very simple way when setting your own SPN’s, to avoid the common pitfalls we see in support day in and day out.   So, copy the script out of that article into a file named spnhelper.vbs and follow the logic in this post when creating your own SPN’s.

Script Commands

I am posting these here as it is not easy to copy/paste them from the dialog they are presented in.

cscript spnHelper.vbs /f:spn /spn:HTTP/www.test.com /user:mydomain\apppool1
cscript spnHelper.vbs /f:spn /spn:HTTP/www.test.com /computer:iis6server1
cscript spnHelper.vbs /f:user /user:mydomain\apppool1
cscript spnHelper.vbs /f:computer /computer:iis6server1
cscript spnHelper.vbs /f:duplicatespn /spn:HTTP/www.test.com
cscript spnHelper.vbs /f:requiredspn

My Setup

This is the criteria for my test environment.

• Domain:  domain.com
• Application Pool Identity: domain\apppooluser
• Host Header:  spntest.domain.com

NOTE:  In a NLB or Cluster setup the application pool is typically run as a domain user.  This is required so that the SPN can be the same on all of the machines that are part of the NLB or cluster environment.

The 3 Simple Steps

Step 1:  Determine the correct SPN

We can use the /f:requiredspn option in the script file to determine the SPN to set for our web site.  Based on the setup of my web site and environment mentioned above I get the following prompts:

• Is IIS running in a Cluster or NLB   - No
• Is IIS application pool running under domain account – Yes
• Enter the hostname or host header or FQDN that you use to access the application – spntest.domain.com
• Enter the Domain Account that the application pool is running under – domain\apppooluser

When I run the command I get the following output:

C:\Secret\Path>cscript.exe spnhelper.vbs /f:requiredspn

Microsoft (R) Windows Script Host Version 5.6
Copyright (C) Microsoft Corporation 1996-2001. All rights reserved.

You need to set SPN HTTP/spntest.domain.com for domain account domain\apppooluser

Step 2:  Check for Duplicate SPN’s

The first thing you might want to do is immediately set the SPN that is required.  But before doing so we should check for duplicates as follows.  Below is output when duplicates are detected, as well as when no SPN’s are detected.

C:\Secret\Path>cscript.exe spnhelper.vbs /f:duplicatespn /spn:http/spntest.domain.com

Microsoft (R) Windows Script Host Version 5.6
Copyright (C) Microsoft Corporation 1996-2001. All rights reserved.

Class: Computer
CN=NINJA,CN=Computers,DC=domain,DC=com
User Name: NINJA$

Class: Person
CN=Application Pool,CN=Users,DC=domain,DC=com
User Name: apppooluser

Duplicate SPNs found
http/spntest.domain.com for Computer:NINJA$
http/spntest.domain.com for Person:apppooluser

Found 2 accounts

In the above case, I need to remove the Computer SPN, as my application pool is running as a domain user.   Here is the output when a duplicate SPN is not found:

C:\Secret\Path>cscript.exe spnhelper.vbs /f:duplicatespn /spn:http/spntest.domain.com

Microsoft (R) Windows Script Host Version 5.6
Copyright (C) Microsoft Corporation 1996-2001. All rights reserved.

No duplicatespn found with the given criteria.

Step 3:  Setting the SPN

Once duplicates have been checked with no duplicates found, we can set the SPN:

C:\Secret\Path>setspn.exe -a http/spntest.domain.com domain\apppooluser

Registering ServicePrincipalNames for CN=Application Pool,CN=Users,DC=domain,DC=com

        http/spntest.domain.com
Updated object

Conclusion

The main point we want to illustrate here is to check for duplicate SPN’s before registering them.  This will help save a lot of grief when troubleshooting problems with SPN’s and Kerberos authentication.

ASP.NET April 2009 Hotfix Rollup Available

Late last month we released a hotfix rollup that includes several fixes. We released these fixes in a rollup because we felt that many customers would be impacted by these issues and we wanted the fixes available to you easily without having to call and open a support incident.

You can read more about the rollup in KB article 969612. Included in the article is a list of all of the fixes contained in the rollup.

HttpException Due to Invalid Viewstate After Installing .NET Framework 3.5 SP1

[No relevant source lines]

[HttpException (0x80004005): Unable to validate data.]
   System.Web.Configuration.MachineKeySection.GetDecodedData(Byte[] buf, Byte[] modifier, Int32 start, Int32 length, Int32& dataLength) +289
   System.Web.UI.ObjectStateFormatter.Deserialize(String inputString) +140

[ViewStateException: Invalid viewstate. 
	Client IP: ::1
	Port: 55824
	User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1; .NET CLR 2.0.50727; .NET CLR 1.1.4322; InfoPath.2; MS-RTC LM 8; .NET CLR 3.5.21022; .NET CLR 3.5.30729; MS-RTC EA 2; OfficeLiveConnector.1.3; OfficeLivePatch.0.0; .NET CLR 3.0.30729)
	ViewState: /wEPDwUKMTg1NzEyNTU1OWRkfmOpf2LsmvVxtr3aHeaANTZVGQM=
	Referer: http://localhost/ActionTest/FormWithActionTag.aspx
	Path: /ActionTest/Test.aspx]

[HttpException (0x80004005): Validation of viewstate MAC failed. If this application is hosted by a Web Farm or cluster, ensure that <machineKey> configuration specifies the same validationKey and validation algorithm. AutoGenerate cannot be used in a cluster.]
   System.Web.UI.ViewStateException.ThrowError(Exception inner, String persistedState, String errorPageMessage, Boolean macValidationError) +106
   System.Web.UI.ViewStateException.ThrowMacValidationError(Exception inner, String persistedState) +14
   System.Web.UI.ObjectStateFormatter.Deserialize(String inputString) +242
   System.Web.UI.ObjectStateFormatter.System.Web.UI.IStateFormatter.Deserialize(String serializedState) +4
   System.Web.UI.Util.DeserializeWithAssert(IStateFormatter formatter, String serializedState) +37
   System.Web.UI.HiddenFieldPageStatePersister.Load() +207
   System.Web.UI.Page.LoadPageStateFromPersistenceMedium() +105
   System.Web.UI.Page.LoadAllState() +43
   System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +6785
   System.Web.UI.Page.ProcessRequest(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +242
   System.Web.UI.Page.ProcessRequest() +80
   System.Web.UI.Page.ProcessRequestWithNoAssert(HttpContext context) +21
   System.Web.UI.Page.ProcessRequest(HttpContext context) +49
   ASP.test_aspx.ProcessRequest(HttpContext context) in c:\Windows\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\actiontest\fae72529\40d789a2\App_Web_2ek84umu.1.cs:0
   System.Web.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +181
   System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +75

After investigating this further we found an action attribute defined in form tags of the aspx page. Removing the action attribute prevented the exception. To reproduce the issue, add the following code to an aspx page called default.aspx  –

<%@ Page Language="C#" %>

<script runat="server">
    protected void Button1_Click(object sender, EventArgs e)
    {
        Response.Write("Test");
    }
</script>

<head runat="server">
    <title>Form With Action Tag</title>
</head>
<body>
    <form id="form1" method="post" action="Test.aspx" runat="server">
    <div>
        <asp:Button ID="Button1" runat="server" onclick="Button1_Click" Text="Button" />
    </div>
    </form>
</body>
</html>

Add a Test.aspx in the same application. When you browse the default.aspx page and click on the button, you will get the above exception.

SP1 for the 3.5 Framework (and SP2 for the 2.0 Framework) added an “action” property to the HtmlForm class that allows developers to programmatically set the action attribute of the form element. Prior to this change, action attributes in existing form elements were ignored, but after the service pack is installed, ASP.NET will honor the action attribute and will begin posting the ASP.NET page to a different page. This causes the validation of the Viewstate to fail.

To avoid the exception, remove the action attribute from the form element on your ASP.NET page. Alternatively, you can use the following code in your global.asax to remove the form attribute for all of your ASP.NET pages at runtime.

void Application_PreRequestHandlerExecute()
{
    var page = Context.Handler as Page;
    if (page != null) page.Init += delegate
    {
        if (page.Form != null && !string.IsNullOrEmpty(page.Form.Action))
        {
            page.Form.Action = string.Empty;
        }
    };
}

I hope this helps!

How To View what ASP.NET Requests are doing at runtime on IIS 6.0

This is just a quick blog to mention a forgotten tool.   The managed stack explorer can be run on an IIS 6.0 Server running ASP.NET 2.0 to investigate the managed call stacks.  Looking at the call stacks when an ASP.NET application is not responding may help identify what the requests are doing.

Setting up the tool

Download the Managed Stack Explorer from here:

http://www.microsoft.com/downloads/details.aspx?FamilyID=80cf81f7-d710-47e3-8b95-5a6555a230c2&displaylang=en

Run the MSI and install to a directory of your choice.  The default location is C:\Program Files\Power Toys for Visual Studio\Managed Stack Explorer

How the Tool Works

The Managed Stack Explorer contains a User Interface that allows you to choose a process and thread and see the managed call stack at that point in time.  See below:

The UI also gives a quick glimpse at the % time a process has spent in the GC.  As a general rule, if that number is > 10-15% it might indicate a performance problem.   For more information on monitoring ASP.NET Performance see this whitepaper:  http://msdn.microsoft.com/en-us/library/ms972959.aspx

The Managed stack explorer also has a stack logging feature that can log the stack trace over time.  When using the UI it takes a snapshot whenever a thread is selected in the thread window.  It takes a snapshot of only this thread at that point in time.  It does this by attaching to the process, collecting information about the thread and then detaching.  When using the stack logging, the tool can be configured to scan the managed threads of a single process at a specified interval and log that to a file.

Using the stack Logging feature

For most cases where someone will want to use this tool, the stack logging feature is a good idea.   IIS Server administrators might get notified that their ASP.NET Application is not responding, or the site is slow.   When this happens the Managed Stack Explorer can be run with stack logging enabled for a period of time, then the log analyzed to determine what the threads and ASP.NET Requests were doing during the period of poor performance. 

 The following steps show how to configure Managed Stack Explorer with stack logging.

1)      Choose Options->Preferences and configure the stack logging rate.  The default is to log the call stack every 5 seconds.  The other settings only apply to the auto refresh of the User interface.

2)      Determine the Process ID Of the application to monitor.  The Managed Stack Explorer shows the processes that have the .NET runtime loaded in the left pane.  When multiple W3WP.exe’s are running use the following command to determine the Application Pools process ID. 

a.       Cscript.exe %systemroot%\system32\iisapp.vbs

3)      In the processes window double-click the w3wp.exe in which the ASP.NET Application is running.  This will cause the thread window to populate with a list of managed threads in that process.

4)      In the Thread list window select all threads, by selecting the first thread, holding shift and then selecting the last thread in the list. The stack logging will only monitor the selected threads, so make sure to select all of them.

5)      Choose Action->Start Stack Logging from the menu and specify a file to log the stack traces too.  This will start the stack logging

6)      Stop the trace after enough data is collected and then choose Action->Stop Stack Logging from the menu.

7)      Open the generated log file and look at what the threads were doing.

Analyzing the stack log file

Open the generated log file in notepad or another text editor.   You will get output similar to the following for each thread:

4/28/2009 11:31:14 AM

Stack trace depth is set to show all frames

Thread ID: 5196

                0. _Default.Page_Load (Source Unavailable)

                1. System.Web.Util.CalliHelper.EventArgFunctionCaller (Source Unavailable)

                2. System.Web.Util.CalliEventHandlerDelegateProxy.Callback (Source Unavailable)

                3. System.Web.UI.Control.OnLoad (Source Unavailable)

                4. System.Web.UI.Control.LoadRecursive (Source Unavailable)

                5. System.Web.UI.Page.ProcessRequestMain (Source Unavailable)

                6. System.Web.UI.Page.ProcessRequest (Source Unavailable)

                7. System.Web.UI.Page.ProcessRequest (Source Unavailable)

                8. System.Web.UI.Page.ProcessRequestWithNoAssert (Source Unavailable)

                9. System.Web.UI.Page.ProcessRequest (Source Unavailable)

                10. ASP.default_aspx.ProcessRequest (Source Unavailable)

                11. System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute (Source Unavailable)

                12. System.Web.HttpApplication.ExecuteStep (Source Unavailable)

                13. System.Web.HttpApplication.ApplicationStepManager.ResumeSteps (Source Unavailable)

                14. System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest (Source Unavailable)

                15. System.Web.HttpRuntime.ProcessRequestInternal (Source Unavailable)

                16. System.Web.HttpRuntime.ProcessRequestNoDemand (Source Unavailable)

                17. System.Web.Hosting.ISAPIRuntime.ProcessRequest (Source Unavailable)

The key pieces are the Thread ID and the Date/Time.   Look for instances where the same thread ID is in the same call for more than a couple of intervals.  In the above stack we can see that it is currently executing the Default.aspx Page_Load routine.  If this stack is identical at our configured 5 second intervals, then this would indicate that something in the Default.aspx page’s Page_load routine is taking greater than  5-10 seconds.  Therefore,  that routine should be investigated.

Depending on the application, the call stacks may be different.  The stacks may indicate a connectivity problem to a backend database, call to a web service, or something else.  Either way the Managed Stack explorer can be used to quickly collect data that may be helpful before using more invasive tools.   And the information generated can help developers of the application understand what the application is doing when it is performing slowly.

If the output does not show any activity on any of the threads, then another form of troubleshooting might be needed.  Some causes of the threads being empty could be:

1)       The request finished before stack logging started

2)      The request is being handled by an Asynchronous handler, and it hasn’t notified ASP.NET that the request has finished.

3)      The request has not been delivered to ASP.NET yet and is still running on a native thread.

Conclusions

The Managed Stack Explorer is a tool that can be used to view what .NET call stacks are doing at runtime.  It does this by attaching to the process and taking a snapshot of the thread and displays it in the UI or logs it to a file.   The managed stack can be used to determine what an ASP.NET request is doing at the time of the snapshot, and by capturing snapshots over a period of time, where the ASP.NET Request might be running slowly, it can be used to help identify what the bottleneck may be.

Lastly, this tool also has source code available for anyone that might want to extend its features.  Please see the following site for information on how to do that.

http://www.codeplex.com/MSE

Caution while xcopying IIS 7.0 config files

Metabase.xml is the central store where IIS 6.0 stores most of its configuration information. Its a plain text file and stores all the information in a simple XML format. The XML format naturally raised a notion of being able to XCOPY the config file to another server and transferring the settings with it. But if you copy over a metabase.xml file from another server your IIS admin service will no longer start. This happens because the metabase.xml file contains ACLs that control access to any metabase key. Stored under the AdminACL tag these keys are encoded based on the machinekeys of the server. When you move the metabase.xml to another server the keys can no longer be decoded and hence your IIS Admin service will not be able to start.

With IIS 7.0 we moved to a new XML based configuration store that is modeled after ASP.NET. It is no longer centralized into a single file. The hierarchical store starts with the applicationHost.config file and can be distributed among web.config files under your application.

This move also enables the long lasting idea of xcopy-deployment. You can now have all the settings in a web.config along with your application content and move it around. 

Another change that was made is that the local accounts/groups that IIS 6.0 used (IUSR_MACHINENAME / IIS_WPG) were replaced by built-in accounts (IUSR / IIS_IUSRS). The built-in accounts have the same SID across Windows 2008 servers and are not machine specific.

So technically you now have an IIS configuration store that is virtually machine independent and you can just copy your applicationHost.config from one server to another and IIS will pick up the settings and just work.

But there is a catch. Try this.

On an IIS 7.0 server change the application pool identity (for the DefaultAppPool)  to a custom domain identity. (Advanced Settings > Application Pool Identity > Custom Account)

Then move the applicationHost.config to a different IIS 7.0 server.

When you try to run a website using the DefaultAppPool you will find that the Application will get disabled with the following error in the event log.

So lets try to change the application pool identity to another domain account or reset the password for that account.

You type in the username and password and hit OK and you will get the following error message

Any username / password will not work. (You can however set the identity of the application pool to one of the built in accounts.)

Wondering what’s going on ? Initially when you set the application pool identity to a domain account IIS has to keep a local copy of the username and password. So it stores a copy in its applicationHost.config and since it is not advisable to keep the password in clear text format it goes ahead an encrypts it. You will see something like this in the config file.

For the encryption it uses machine specific keys in the iisConfiguration and iisWasKey containers. When the applicationHost.config is moved to a different server IIS can no longer decrypt the settings.

To get this working you can export and import the keys from the original server.

Export using the following commands

And for the import use

So whenever you are trying to xcopy-deploy your application on multiple servers you need to check if there are any encrypted sections and if you do ensure you port the iisConfigurationKey and the iisWasKey as well.

Also I would recommend using the Web Deployment Tool ( MSDeploy ) which makes deployment a lot easier. You can create a package (settings and content)  of the whole server / specific application and use it to deploy. But the tool is in BETA still.

Cross posted from Vijay’s Blog ( http://blogs.msdn.com/vijaysk/archive/2009/03/14/caution-while-xcopying-iis-7-0-config-files.aspx )

High Memory due to System.WeakReference

We recently saw an issue that was manifesting as very high memory utilization by a w3wp.exe process that was hosting an ASP.net application.  To start troubleshooting, we gathered a memory dump of the process when its memory usage was very high.

A standard way of starting out the debugging of a high memory problem in a managed application is to run the !dumpheap –stat command.  When we did so, we found millions of 16 byte System.WeakReference objects (I’ve truncated the output for brevity’s sake):

0:000> !dumpheap -stat

Loading the heap objects into our cache.

total 28,603,114 objects

Statistics:

        MT    Count    TotalSize Class Name

0x648efc44    4,199       83,980 System.Configuration.ConfigurationValue

0x6639b220    4,368       87,360 System.Web.VirtualPath

0x6639e3fc    4,534       90,680 System.Web.Caching.CacheKey

0x663b4074    2,063       90,772 System.Web.HttpCookie

0x648ea704    2,003       96,144 System.Configuration.ConfigurationValues

0x79131840       99      104,612 System.DateTime[]

0x663b27f4    2,066      107,432 System.Web.HttpValueCollection

0x663aa5e8    2,072      107,744 System.Web.HttpModuleCollection

0x66435948    2,073      107,796 System.Collections.Generic.Dictionary`2

0x65409d70    1,772      113,408 System.Data.SimpleType

0x663ada24    4,126      115,528 System.Web.Hosting.RecyclableCharBuffer

0x663aa6f8    4,126      132,032 System.Web.HttpAsyncResult

0x7910d7e8    4,148      132,736 System.AsyncCallback

0x6641f33c    3,028      133,232 System.Web.UI.Control+OccasionalFields

0x65405fdc    1,102      141,056 System.Data.SqlClient._SqlMetaData

0x7a766474    8,912      142,592 System.ComponentModel.CollectionChangeEventArgs

0x79111038    2,748      153,888 System.Reflection.RuntimePropertyInfo

0x7910efbc   13,608      163,296 System.Runtime.Remoting.Messaging.CallContextSecurityData

0x663af054    2,063      165,040 System.Web.HttpWriter

0x653fe4d4    2,080      208,000 System.Data.SqlClient.SqlParameter

0x663b08c8    4,126      231,056 System.Web.HttpCookieCollection

0x02525e84    2,073      240,468 ASP._global_asax

0x79109778    4,395      246,120 System.Reflection.RuntimeMethodInfo

0x663af964    2,072      248,640 System.Web.SessionState.SessionStateModule

0x79102290   25,426      305,112 System.Int32

0x663aa0a0    2,063      354,836 System.Web.HttpRequest

0x663a9dac    2,063      387,844 System.Web.HttpContext

0x663a4adc    2,063      404,348 System.Web.HttpResponse

0x79108ce0   11,536      415,296 System.Collections.Hashtable+HashtableEnumerator

0x7911a2d0   13,610      489,960 System.Runtime.Remoting.Messaging.LogicalCallContext

0x79104de8   13,616      490,176 System.Threading.ExecutionContext

0x663b205c   31,080      497,280 System.Web.HttpApplication+SyncEventExecutionStep

0x663ad7c0    2,063      503,372 System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6

0x7912d7c0    7,403      522,620 System.Int32[]

0x7a7566e8   26,959      539,180 System.ComponentModel.EventHandlerList+ListEntry

0x65412bb4      514      539,672 System.Data.RBTree`1+Node[[System.Data.DataRow, System.Data]][]

0x7a75a878   35,598      569,568 System.Collections.Specialized.NameObjectCollectionBase+NameObjectEntry

0x79104368   25,337      608,088 System.Collections.ArrayList

0x0252788c    2,027      770,260 ASP.default_aspx

0x654088b4    5,281      781,588 System.Data.DataColumn

0x79108afc   11,541      877,116 System.Threading.ExecutionContext+ExecutionContextRunData

0x654359c8    1,038    1,125,544 System.Data.RBTree`1+Node[[System.Int32, mscorlib]][]

0x66437650    2,073    1,252,092 System.Collections.Generic.Dictionary`2+Entry

0x791044dc   53,803    1,721,696 System.EventHandler

0x79101fe4   31,106    1,741,936 System.Collections.Hashtable

0x000e2dc0    3,644    3,621,108      Free

0x7912d9bc   31,396    4,984,728 System.Collections.Hashtable+bucket[]

0x790fd8c4   95,552    9,502,548 System.String

0x7912dd40    8,376   12,904,904 System.Char[]

0x7912d8f8   53,376  336,430,944 System.Object[]

0x79104c38 27,851,257  445,620,112 System.WeakReference

Total 28,603,114 objects, Total size: 836,223,424

NOTE: For more information on the !dumpheap debugger command, have a look at the following blog post:

http://blogs.msdn.com/tess/archive/2005/11/25/496973.aspx

The next step was to dump out some of the System.WeakReference instances to try to dig in to why they’re there:

0:000> !dumpheap -mt 0x79104c38

Using our cache to search the heap.

   Address         MT     Size  Gen

0x025d7ab8 0x79104c38       16    2 System.WeakReference

0x025dc2d0 0x79104c38       16    2 System.WeakReference

0x025dc540 0x79104c38       16    2 System.WeakReference

0x025dd330 0x79104c38       16    2 System.WeakReference

0x025dd8d8 0x79104c38       16    2 System.WeakReference

0x025def80 0x79104c38       16    2 System.WeakReference

0x025def90 0x79104c38       16    2 System.WeakReference

0x025defa0 0x79104c38       16    2 System.WeakReference

0x025defb0 0x79104c38       16    2 System.WeakReference

0x025defc0 0x79104c38       16    2 System.WeakReference

0x025defd0 0x79104c38       16    2 System.WeakReference

0x025defe0 0x79104c38       16    2 System.WeakReference

0:000> !do 0x025defa0

Name: System.WeakReference

MethodTable: 79104c38

EEClass: 79104bd4

Size: 16(0x10) bytes

GC Generation: 2

 (C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)

Fields:

      MT    Field   Offset                 Type VT     Attr    Value Name

791016bc  40005a9        4        System.IntPtr  1 instance 33389776 m_handle

7910be50  40005aa        8       System.Boolean  1 instance        0 m_IsLongReference

All of the objects we dumped out looked the same, and none of them appeared to be rooted.  Looking around for other signs of problems, we found an components built in debug mode:

0:000> !sos.FindDebugModules

Loading all modules.

Searching for modules built in debug mode...

custom_component.DLL not built release

custom_component2.DLL not built release

custom_component3.DLL not built release

Having modules built in debug mode running on a Production server is never a good idea.  And as it turned out in this case, the debug mode modules combined with the fact that these modules implement the __ENCLIST helper class for Visual Studio’s Edit and Continue feature.

When a component is built in debug mode, the  Edit and Continue debugging feature of Visual Studio is enabled.  The Edit and Continue debugging feature in Visual Studio 2005 and 2008 maintains a list of weak references to objects that have been created. If the object that has been created is a class that contains an event, these weak references are maintained for the duration of the program. This behavior increases memory usage.  To run into the memory usage problem, the component must be built in DEBUG mode, and in must use the __ENCLIST help class that allows edit and continue.

In this scenario, the problem got resolved by rebuilding those components in Release mode instead of Debug mode.