WebTopics

How To: Collect a Crash dump of an IIS worker process on IIS 7.0 (and above)

puneetg — Wed, 25 Nov 2009 14:17:00 GMT

Web server Administrators have been using the Debug Diagnostic 1.1 to troubleshoot crash\hang issues on IIS 5 and 6. The Debug Diagnostic tool needs no introduction. It is a great tool and it provides you a lot of flexibility in collecting crash dumps for any user mode process running on a server. But as of now, the Debug Diagnostic tool is *not* compatible with Windows Server 2008 and it is recommended that you not use it on a Windows 2008 server because you may get unexpected results. The good news is that the Debug Diagnostic team is currently working on releasing a version that is compatible with Windows 2008, but what to do until that time?

The answer is to leverage the powerful Windows Error Reporting (WER) feature that comes with Windows 2008. You can use WER on a Windows 2008 server to collect a crash dump of any process without any other tools.

The good news is that Windows Error Reporting is enabled by default on every Windows 2008 server and takes dumps of all application crashes on the server. If you have experienced crashes of an IIS worker process, all you have to do is check if there is a dump generated for you already. To do that, please follow these steps.

Click on Start-> Run and type WERCON
This will open up the “Problems Reports and Solutions” dialog box and it will look like this.
From here, just click on “View Problem History” and see if you can find any crashes for “IIS Worker Process”
You can select the particular IIS Worker Process crash entry and right click on it and choose “View Problem details”. This will open up a window like this which will give you an overall idea of the crash.
To get the dump files for this crash, just right click on “View a temporary copy of these files” and this will copy the relevant .hdmp , .mdmp and version.txt to your local temporary folder. These are the dump files which you can either analyze using WinDBG or send to Microsoft for further analysis.

Windows Error Reporting introduced a new feature (in Windows Vista and above) which lets you create a LocalDumps registry key under HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting and then under that you can create a subkey with the name of the executable that you want to take dumps for. This approach had a few limitations as of now. It turned out that :

This feature works only if HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug\Auto is set to 0. If Auto is set to 1, the control is given to the debugger specified in HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug\Debugger setting and the debugger decides what to do with the unhandled exception and WER is not given a chance to handle the exception.
The .net framework version 3.5 and below do not recognize this registry key. This means that if your process has some managed code and if some unhandled exception happens while managed code execution (the exception may be native or a CLR exception), the LocalDumps method will not get you dumps for the process. The functionality inside the .net CLR to read the LocalDumps registry key is added in .net framework 4.0. Because of this limitation, the LocalDump feature becomes less useful for an IIS worker process because most of the time an IIS worker process is running ASP.NET applications or some managed code.

What if WER is not taking the dumps for the process of interest

If you notice that dumps are not getting generated, you can check the general WER settings on your machine. These settings can be configured at HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\Windows Error Reporting. The complete list is available at http://msdn.microsoft.com/en-us/library/bb513638(VS.85).aspx. I will just include the most relevant ones in regard to taking dumps :

Disabled	Should be 0. If this is set to 1, WER will be disabled completely
ForceQueue	Default value is 0 but you can set it to 1 to ensure that dumps are generated on every crash
DisableQueue	Should be 0
DisableArchive	Should be 0
CorporateWERServer	If you want to keep the data locally on the server and don't want the dumps to be moved to the WER server, you can set this setting to a non-existing server in your domain. This will ensure that the data is stored locally on the server.
MaxQueueCount	Default is 50, You can increase this to a higher number to increase the total number of dumps that will be saved to the ReportQueue(For more information on ReportQueue, refer to this article)

Another important thing to be aware of while using Windows Error Reporting is that a lot of Windows Error Reporting settings can be configured via group policy. To look at the WER settings on your server, open up Local Group Policy Editor (GPEDIT.MSC) and you will be able to see these settings under Computer Configuration –> Administrative Templates –> Windows Components –> Windows Error Reporting. This is how it looks.

The relevant WER settings defined in group policy will override the local settings. Most of these group policies don't apply to Windows Vista and above. The ones that do apply are

Disable Windows Error Reporting

(Under Advanced Error Reporting Settings )

Configure Report Archive

Configure Corporate Windows Error Reporting

List of applications to be excluded

Configure Report Queue

There may be specific reasons why the WER settings are disabled on domain and it is recommended to get in touch with an Administrator before making any changes to the above policies. Explaining these group policy entries would result in a discussion outside of the scope of this post, so I recommend looking at the MSDN documentation for the above policy settings for more information on them.

Windows Error Reporting and ADPLUS

Administrators who are aware of Debug Diagnostic tool may also know about another tool called ADPLUS.VBS that ships with the Debugging Tools For Windows. Though ADPLUS.VBS tool has a -crash switch that lets you attach a debugger to a process in CRASH mode, it is not the best option to use while troubleshooting worker process crashes because ADPLUS doesn’t have the ability to auto-attach to a new process. You can only attach ADPLUS to a process which is already running on the machine. This becomes a big limitation for an IIS worker process because a worker process may recycle or shut-down based on what is configured in the application pool settings. Another problem with ADPLUS is that it starts a new window for cdb.exe and someone may accidently close this window which may end up killing the worker process itself.

You may have noticed that WER just lets you generate a crash dump for a process when it crashes and doesn’t provide you the flexibility that Debug Diagnostic provides you though (e.g. taking dump on specific exceptions, specific breakpoints, events etc.). If you want to dump your process on a particular exception on Windows 2008 and above, you still have to use the ADPLUS tool. The discussion how to use ADPLUS to troubleshoot crashes for a process is out of the scope of this article.

ProcDump

This is one of the latest additions in the Sysinternals suite of tools. ProcDump is a command-line utility whose primary purpose is monitoring an application for CPU spikes and generating crash dumps during a spike that an administrator or developer can use to determine the cause of the spike. ProcDump also includes hung window monitoring (using the same definition of a window hang that Windows and Task Manager use) and unhandled exception monitoring. It also can serve as a general process dump utility that you can embed in other scripts. I won’t spent too much time talking about PROCDUMP in this blog post because we already have another post which talks about this tool in detail - http://blogs.msdn.com/webtopics/archive/2009/08/08/using-procdump-exe-to-monitor-w3wp-exe-for-cpu-spikes.aspx

References

Windows Error Reporting (WER) for developers http://blogs.msdn.com/oanapl/archive/2009/01/28/windows-error-reporting-wer-for-developers.aspx
Using WER: Collecting User-Mode Dumps http://msdn.microsoft.com/en-us/library/bb787181(VS.85).aspx
How to use ADPlus to troubleshoot "hangs" and "crashes" http://support.microsoft.com/kb/286350

Configuring FTP 7.5 with Host Header and SSL

Saurabh Singh — Fri, 06 Nov 2009 00:04:00 GMT

FTP 7.5 comes with new features like supporting Host headers (Virtual host) and SSL. For compatibility purposes, FTP clients can check whether the FTP server supports host headers by sending a FEAT command to check for supported features. An FTP server would respond with HOST being one of the extended features supported by it and from here on client can use this feature.

There are two ways of using this feature currently by the FTP clients:

1. They can send the virtual host name along with the Username while getting authenticated by the FTP server as below:

C:\>ftp ftp.me.com
Connected to ftp.me.com.
220 Microsoft FTP Service
User (ftp.me.com:(none)): ftp.me.com|<username>
331 Password required for ftp.me.com|<username>
Password:
230 User logged in.
ftp>

or,

2. They can send the following command to connect to a specific Virtual host name as below:

ftp> quote host ftp.me.com
220 Host accepted.
ftp> user
Username <username>
331 Password required for <username>.
Password:
230 User logged in.
ftp>

In this example the test is done through the default ftp.exe that comes bundled with Windows OS. Smart FTP clients can send the HOST seamlessly without the end user knowing about it.

If you are setting up your FTP site on IIS 7 over SSL using the host header there are some caveats you need to remember as discussed below.

I have an FTP site as shown below which is using a Host header and is configured to accept SSL connections.

Under FTP SSL Settings I have the following configuration as shown below:

If we try to access the FTP site using an SSL enabled FTP client it will fail as shown below:

Command:    AUTH TLS
Response:    534-Local policy on server does not allow TLS secure connections.
Response:     Win32 error:   Access is denied.
Response:     Error details: SSL certificate was not configured.
Response:    534 End
Command:    AUTH SSL
Response:    534-Local policy on server does not allow TLS secure connections.
Response:     Win32 error:   Access is denied.
Response:     Error details: SSL certificate was not configured.
Response:    534 End

*Output from Filezilla FTP client

NOTE: AUTH TLS/SSL Negotiation for Primary connection is done based on the certificate at the global level and uses the certificate installed at the site level for Data connection.

Some FTP clients like FileZilla require the same SSL certificate to be used for both the control and the data channel. If the certificates don’t match the primary connection will be established but the data transfer connection will be aborted as shown below:

Command:    LIST
Response:    150 Opening BINARY mode data connection.
Error:    Primary connection and data connection certificates don't match.
Error:    Transfer connection interrupted: ECONNABORTED - Connection aborted
Response:    226 Transfer complete.
Error:    Failed to retrieve directory listing

*Output from Filezilla FTP client

If we have the SSL Certificate only at the global level and not at the FTP site level we will see an error again as below:

Command:    PROT P
Response:    431-Failed to setup secure session.
Response:     Win32 error:
Response:     Error details: SSL certificate hash has invalid length.
Response:    431 End
....
....
Command:    LIST
Response:    534-Protection level negotiation failed.
Response:     Win32 error:   Access is denied.
Response:     Error details: Protection negotiation failed. PROT command with recognized parameter must precede this command.
Response:    534 End
Error:    Failed to retrieve directory listing

*Output from FileZilla FTP client

So overall, we need to ensure that a valid SSL certificate is set both at the global and the individual FTP site levels. It finally depends upon the FTP client whether to expect the same certificate or different ones for primary and data connections.

Till next time, Saurabh

Configuring DebugDiag to Automatically Capture a Full User Dump on a Managed Function

bretb — Thu, 01 Oct 2009 21:30:00 GMT

I recently worked an issue with a customer where we needed to capture a full user dump of the IIS worker process when a specific function was being called. The problem was that the function was in managed code. I thought it would be good to share how to do this with the Web Topics Community. I will first show how to set a managed breakpoint in a normal debugger. Then I will show how to automate it with DebugDiag, and have it take an action of creating a full user mode dump file when the breakpoint is hit.

The managed function I will use to illustrate this is:

System.Web.HttpResponse.ReportRuntimeError

Normal Debugger

Load sos.dll:
.loadby sos mscorwks

Method 1: Set the Breakpoint with !bpmd

0:012> !bpmd System_Web_ni System.Web.HttpResponse.ReportRuntimeError
Found 1 methods...
MethodDesc = 65f688bc
Setting breakpoint: bp 660A27C0 [System.Web.HttpResponse.ReportRuntimeError(System.Exception, Boolean, Boolean)]

Method 2: Find the Jitted address and set with bp command

0:012> !name2ee System_Web_ni System.Web.HttpResponse.ReportRuntimeError
Module: 65f21000 (System.Web.dll)
Token: 0x060005fb
MethodDesc: 65f688bc
Name: System.Web.HttpResponse.ReportRuntimeError(System.Exception, Boolean, Boolean)
JITTED Code Address: 660a27c0
0:012> bp 660a27c0

When you run the debugger, and it breaks, you can use !clrstack to see the call stack:

0:003> !clrstack
OS Thread Id: 0x68c (3)
ESP EIP
0063f2f0 660a27c0 System.Web.HttpResponse.ReportRuntimeError(System.Exception, Boolean, Boolean)
0063f2fc 66090280 System.Web.HttpRuntime.FinishRequest(System.Web.HttpWorkerRequest, System.Web.HttpContext, System.Exception)
0063f344 660900d6 System.Web.HttpRuntime.OnHandlerCompletion(System.IAsyncResult)
0063f374 6608fafe System.Web.HttpAsyncResult.Complete(Boolean, System.Object, System.Exception, System.Web.RequestNotificationStatus)
0063f394 6608c6e1 System.Web.HttpApplication+ApplicationStepManager.ResumeSteps(System.Exception)
0063f3e4 660808ac System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest(System.Web.HttpContext, System.AsyncCallback, System.Object)
0063f400 66083e1c System.Web.HttpRuntime.ProcessRequestInternal(System.Web.HttpWorkerRequest)
0063f434 66083ac3 System.Web.HttpRuntime.ProcessRequestNoDemand(System.Web.HttpWorkerRequest)
0063f444 66082c5c System.Web.Hosting.ISAPIRuntime.ProcessRequest(IntPtr, Int32)
0063f658 79f68cde [ContextTransitionFrame: 0063f658]
0063f68c 79f68cde [GCFrame: 0063f68c]
0063f7e8 79f68cde [ComMethodFrame: 0063f7e8]

This can be useful when a particular runtime error is occurring in ASP.NET and you are not sure why. However, attaching a live debugger and running these commands will stop the processing of web requests for the amount of time you spend in the live debugger. Therefore, the following steps will show how to automate this with DebugDiag.

Automating a Managed Breakpoint Dump with DebugDiag

DebugDiag can be downloaded here: http://www.microsoft.com/downloads/details.aspx?FamilyID=28bd5941-c458-46f1-b24d-f60151d875a3&displaylang=en

After Installing DebugDiag, the steps we will take are as follows:

Create a Crash Rule, but do not activate it.
Modify the Crash Rule Script
Activate the Crash Rule

Create a Crash Rule that is not activated

On the rules tab click the Add Rule... button to bring up the Rule creation wizard.
Choose the Crash Rule Option:
Choose the option "A specific IIS web application pool"
Choose the application pool that will be monitored
Click next twice to move to the end of the Wizard and choose "Do not activate the rule at this time"
Click Finish to create the rule

Modify the Crash Rule Script

The crash rule is a VBS File that will be located in the DebugDiag installation Scripts sub-directory. The VBS File is used when DebugDiag attaches to the IIS worker process for debugging. We will be modifying two functions in this script file.

Open the script file in notepad or your favorite text editor. It will be named CrashRule_WebAppPool_AppPoolName.vbs. Where the "AppPoolName" part of the file will be the name of the application pool the rule was configured for. Search for Debugger_OnInitialBreakpoint in the script file. This is a function that is called when DebugDiag first attaches to the process. This is where we will add code to set the breakpoint. The function should look like this when completed.

Sub Debugger_OnInitialBreakpoint()
Dim strRet, strBP, bpCommand
WriteToLog "Initializing control script"
On Error Resume Next
Set ServiceController = CreateObject("DbgSVC.Controller")
Set ServiceState = ServiceController.ServiceState
On Error Goto 0 WriteToLog "Clearing any existing breakpoints"
WriteToLog Debugger.Execute("bc *")
WriteToLog Debugger.Execute(".loadby sos mscorwks")
strRet = Debugger.Execute("!bpmd System_Web_ni System.Web.HttpResponse.ReportRuntimeError")
WriteToLog strRet
WriteToLog "Report Runtime Error BP Configured"
DbgState("BP_RUNTIME_ERROR_COUNT") = 0
WriteToLog "Current Breakpoint List(BL)"
Debugger.Write Debugger.Execute("bl")
End Sub

We added the following code to load the debugger extension and set the breakpoint. The last line will output the result of the !bpmd command to our log file for our rule.

WriteToLog Debugger.Execute(".loadby sos mscorwks")
strRet = Debugger.Execute("!bpmd System_Web_ni System.Web.HttpResponse.ReportRuntimeError")
WriteToLog strRet
WriteToLog "Report Runtime Error BP Configured"

Then we added the following we can use to keep track of a dump count for our breakpoint:

DbgState("BP_RUNTIME_ERROR_COUNT") = 0

This will set our breakpoint. When the breakpoint is actually hit, the Debugger_OnBreakpoint event is fired in our script. We will modify this function to handle our breakpoint and generate a full user dump file. We will also hard code a limit of 5 dump files.

To change the second function, look for Debugger_OnBreakpoint in the script file and re-configure it as follows:

Sub Debugger_OnBreakPoint(ByVal BreakPoint, ByVal CausingThread)
WriteToLog "Breakpoint at " & BreakPoint.OffsetExpression & " caused by " & CausingThread.SystemID
If BreakPoint.OffsetExpression = "System.Web.HttpResponse.ReportRuntimeError(System.Exception, Boolean, Boolean)" Then
  If DbgState("BP_RUNTIME_ERROR_COUNT") < 5 Then
   CreateDump Breakpoint.OffsetExpression, false
   DbgState("BP_RUNTIME_ERROR_COUNT") = DbgState("BP_RUNTIME_ERROR_COUNT") + 1
  End If
End If
End Sub

Save the script file.

Activate the Crash Rule and verify the breakpoint is set

Right click the rule in DebugDiag's rule view, and click the Activate Rule menu item. This will activate our rule. But how can we be sure our breakpoint was set? To do this go to the logs folder and open the w3wp__xxx_xxx.txt file that was created for the worker process when the rule was activated. The easiest way to get there is View->Logs Folder menu in DebugDiag. Locate the w3wp__PID__XXXX__Log.txt file for the PID of the currently running w3wp.exe and open it in notepad. Search for "Report Runtime Error BP Configured" in the log file, and make sure the breakpoint is correct. Here is what it should look like in the log file if it is correct:

Found 1 methods...
MethodDesc = 65f688bc
Setting breakpoint: bp 660A27C0 [System.Web.HttpResponse.ReportRuntimeError(System.Exception, Boolean, Boolean)]
[10/1/2009 1:22:10 PM] Report Runtime Error BP Configured

Conclusions

This post explains how to automate a managed breakpoint with DebugDiag to generate a Full User dump when the managed breakpoint is hit. The particular example used can be used to get a full user dump when the System.Web.HttpResponse.ReportRuntimeError function is called. Getting a full user dump when this function is called may help identify why a certain ASP.NET Runtime error occurs. For more information on how to analyze the dumps created please see the following .NET Debugging Demo's on Tess's blog:

http://blogs.msdn.com/tess/pages/net-debugging-demos-information-and-setup-instructions.aspx

A Tale of Two Developers and Visual Studio Sites and Apps

jamesche — Wed, 30 Sep 2009 16:40:00 GMT

We've encountered several scenarios in which customers have multiple developers working on a single web application in Visual Studio. In such cases, it's not uncommon for each developer to be working on a portion of the application. There isn't any problem in working with this way, but you may encounter some very real problems when you deploy your application unless you plan in advance.

The Scenario

Suppose you have two developers working on your web application. Developer A (we'll call him John) is working on a forum for the application and developer B (who we'll call Jill) is working on a photo gallery for the site. In order to keep files and folders organized, the forum and the photo gallery will both be contained in their own folders on the final site. In other words, by browsing to www.site.com/forum, users will be able to hit the forum, and by browsing to www.site.com/photogallery, users will hit the photo gallery.

Jill has created a new web site in Visual Studio called PhotoGallery, and this is where she's going to create her pages for her portion of the application. John has created a new web site called Forum for his portion of the site. Both John and Jill frequently test their code against the ASP.NET Development Server and, after a long period of development, are both ready to deploy to the test web server.

Jill's PhotoGallery in Solution Explorer

Jill copies the PhotoGallery folder she created for her portion of the site to the c:\inetpub\wwwroot folder on the server and John copies the Forum folder he created to the wwwroot folder. The resulting directory structure in IIS is shown below.

Folder Structure in IIS

Everything looks fine with this scenario so far, but as soon as either John or Jill attempts to access his or her portion of the site, something bad happens as shown below.

An error occurs when John or Jill browse the site.

The Problem

The problem here is that Visual Studio expects that the web sites that John and Jill created are application roots in IIS. An application root in IIS is a folder that is marked as an application. When a folder is marked as an application, the folder's icon appears as a globe with a page on it as shown below.

An Application in IIS.

Important
I converted the "forum" folder into an application for the figure above simply so that I could show you what the icon looks like. The act of copying a folder to IIS will not make a folder an application in IIS.

Since John and Jill simply copied folders into IIS, the applications they each created in Visual Studio are simply folders in IIS (like the photogallery folder in the figure above) and are not recognized as applications. Because ASP.NET does not allow for certain settings to be configured in a subdirectory of an application, both John and Jill see an error when browsing their portion of the site.

Keep in mind that this is only one example of the kind of problem you can encounter in this scenario. For example, if the root of your site was created in Visual Studio as a web application instead of a web site and developers copy web sites from Visual Studio into that directory structure, you can end up with more problems that are tough to solve.

More Info
See http://msdn.microsoft.com/en-us/library/aa983474.aspx on MSDN for an explanation of web sites vs. web applications in Visual Studio.

The Solution

How can John and Jill solve the error message they are seeing? They could simply convert the folders that they copied to IIS into application roots, but while doing so would get rid of the error, it would likely introduce other problems. For example, there Application and Session variables are specific to an application, so if both the photogallery and forum folders were marked as application roots, Application and Session variables couldn't be shared between them. Also, as we saw in the error that Jill saw in the browser, authentication modes are specific to an application. Therefore, if the site as a whole relies on ASP.NET Forms authentication, mucking around with application roots can break authentication.

A good approach to the problem that John and Jill are encountering is to change the way that they both develop the site in Visual Studio. Each should develop against a web site in Visual Studio called site.com and create a folder within that site for the part of the application they are developing. The resulting project structure in Visual Studio's Solution Explorer is shown below.

The right way to create the site in Visual Studio.

If your team consists of multiple developers and you want to ensure that you control who is working on particular parts of the project, it's wise to have your developers work off of a common site.com folder and use a source control system like TFS to check out the content that he or she needs to develop. If you don't have a source control system, you'll want to make sure that the web.config file in the site.com folder is the same for each developer and it's a good idea for each developer to change only files in his or her particular subfolder for the app.

The Terminology Can Cause Confusion

It's important that you understand the terminology involved in this kind of scenario. For example, Visual Studio can create an ASP.NET web site or an ASP.NET web application. An ASP.NET web site is not the same thing as a web site in IIS, and an ASP.NET web application is not the same as an application in IIS. I realize that's a bit confusing, so here's some information that might help to clear things up.

Web Site in Visual Studio

Web sites in Visual Studio were introduced with Visual Studio 2005. There is no project files in a web site (no .csproj or .vbproj) and the location for the web site can be anywhere on a disk. There is not a References folder in Solution Explorer when working with web sites. Instead, you set refereces in the Property Pages for the site. You are also not required to build (compile) a web site prior to running it. All of that takes place dynamically.

You create an ASP.NET web site in Visual Studio by selecting File, New Web Site.

Web Site in IIS

A web site in IIS is a node that is identified by its bindings. By default, there is one web site called Default Web Site with an HTTP binding all IPs to port 80. If you want to create a new web site, you would have to create a new binding to either a different port, a different IP, etc.

As you can see web site in Visual Studio is completely unrelated to web site in IIS.

Web Application in Visual Studio

A web application in Visual Studio uses the same model that was used for ASP.NET projects in Visual Studio .NET 2003. It was then reintroduced for Visual Studio 2005 as an add-on called Web Application Projects. That add-on became a full-fledged component of Visual Studio 2005 in SP1 and remains a component of Visual Studio 2008.

When the application is created, a project file is created along with the other files for the project. A gray-colored References folder appears in Solution Explorer so that you can review project references and/or add your own. There is also a special folder called Properties that contains the AssemblyInfo.cs or AssemblyInfo.vb file for the project. When you're ready to run the application, you must first compile it from the Build menu.

You create an ASP.NET web site in Visual Studio by selecting File, New Project.

Web Application in IIS

A web application in IIS is a node that has been marked as an application. An application in IIS is the level at which you would place your global.asax file. ASP.NET authentication methods (i.e. Windows, Forms, etc.) can be specificed at this level, but not lower in the hierarchy. Session variables and Forms authentication tickets are isolated to a web application in IIS.

As you can see a web application in Visual Studio is completely unrelated to a web application in IIS.

Final Thoughts

We worked with a customer at one point who had created a web application in Visual Studio that would eventually be deployed to the Default Web Site in IIS. During development, some of the developers who were creating portions of this site determined that they'd better create web sites in Visual Studio for their portion of the site because they new that this was eventually going to be deployed to a web site.

On the day of deployment, someone realized that the main project for the site was a web application, so they began the process of trying to convert the Visual Studio web application to a Visual Studio web site. In fact, we have thorough documentation about how to convert a web site to a web application, but going the other way is another matter altogether. The customers projects was huge, made up of thousands of pages and folders. After they had spent a considerable amount of time trying this on their own, they called us.

Getting everything back to working order took days of work, much of which was spent with the customer frustrated over a seemingly endless number of exceptions. Fix one exception and 200 others spring up to take its place. It's grueling work, and it was all the result of misunderstanding the terms and drawing correlations where none exists.

Hopefully the information I've provided here can help you to avoid a similar pitfall. Plan your projects carefully with deployment in mind, and make sure that you understand the fact that even though IIS and Visual Studio use some of the same terms, those terms mean something entirely different in each context.

ASP.NET MVC: What is it and should I use it?

jamesche — Tue, 01 Sep 2009 22:49:00 GMT

In March of this year, we released ASP.NET MVC. Since then, many ASP.NET developers have been perplexed about MVC. Many developers aren't quite sure what MVC is and what it means for ASP.NET developers. Many more developers have some level of understanding when it comes to MVC but are not sure when it should be used and why. This post will hopefully shed some light on some of the confusion surrounding MVC.

More Info
You can download ASP.NET MVC from the ASP.NET MVC site. A more convenient means of downloading MVC and other Microsoft Web technologies is the Web Platform Installer available here.

A Brief Overview of ASP.NET MVC

ASP.NET MVC is a framework that adds support for the MVC design pattern to ASP.NET. Therefore, in order to really understand what ASP.NET MVC is, you need to first understand what MVC is.

MVC is an acronym that stands for Model / View / Controller. MVC is a programming architecture that aids developers with separating different components of an application. Let's review each component of MVC individually as they relate to ASP.NET MVC.

The Model

The model in an MVC application stores the application data or state of the application. The model is often a database, an XML file, etc. However, because the model is designed to encapsulate the data layer in the application, you will typically not see the data source correlated with the model with regards to MVC.

In an ASP.NET MVC application, the model typically uses LINQ to SQL or LINQ to Entities.

The View

The view is the user interface that your site visitors to see data from your model. In an ASP.NET MVC application, web forms (ASPX pages) are typically used to display the view, but there's a significant difference between an MVC view's page and a typical ASP.NET web form. Most specifically, an MVC view doesn't use the typical postback model and the page lifecycle that you are used to when using web forms doesn't exist with MVC views.

More Info
You don't have to use ASP.NET web forms as the view engine for MVC. Check out MVC Contrib on CodePlex for more information.

It's easy to make the mistake of thinking of the view as the component in MVC that handles user input and controls the interaction with the user. In fact, it's the controller that takes on this role. The view is strictly limited to displaying data from the model.

The Controller

The controller is responsible for handling the interaction with the user, for communicating with the model, and for determining which view to display to the user. The controller is derived from System.Web.Mvc.Controller.

The controller defines one or more actions that can be invoked using a URL entered into a web browser. For example, consider the following URL:

http://www.mysite.com/products/display/43

A request such as this one would be handed off to the ProductsController where the display action would be invoked. In this particular case, that action might then display a view of product number 43. The routing of the URL to a particular controller is configured using routes that are defined in the global.asax of the MVC application.

More Info
For more detailed information on controllers and actions, see the MSDN documentation.

What exactly is an action? An action is simply a method of the controller class that meets a few specific requirements.

An action method must be public
An action method cannot be overloaded
An action method cannot be static

Warning
It's important to keep security in mind when developing your action methods. For example, based on the above URL, one could conclude that browsing to http://www.mysite.com/products/delete/43 would delete product number 43! The Authorize attribute can be used to restrict access to a controller's actions. See http://msdn.microsoft.com/en-us/library/dd381413.aspx for more information.

Benefits of MVC

In order to maintain state in an ASP.NET web forms application, ASP.NET uses encoded data in a hidden form field via a feature called viewstate. Viewstate does pretty well at providing the illusion of a stateful application, but there are times when problems are encountered. For example, in large applications, viewstate can become very large. Large viewstate not only increases your payload across the network, but it can also impact your search engine ranking by pushing readable page content far down in the rendered code.

More Info
The problem of large viewstate was greatly improved in ASP.NET 2.0, and you can expect even more improvements in ASP.NET 4.0.

ASP.NET web forms developers also have to take the page lifecycle into account, and dealing with when to do certain things can be as much or more frustrating than figuring out how to do something. MVC does away with this kind of frustration because it is truly stateless. Postbacks and the web forms page lifecycle no longer exist.

Another advantage when using MVC is that it allows for full control over the rendered HTML. In a web forms application, HTML code is rendered in large part by server controls, and developers have relatively little control over the code that is generated. The HTML code in an MVC view, on the other hand, is entirely controlled by the developer of the view.

MVC also lends itself well to a separated approach to development. One developer can work on the controller class while another developer works on the view. This design methodology also aids in testing. It's extremely difficult to test one particular piece of a web forms application because of the reliance of all of the other controls on the page and the managed runtime. An MVC application can be tested much more easily and efficiently because the model, the view, and the controller are all separate components.

More Info
You can find a great overview of the benefits and drawbacks of both MVC and web forms in Dino Esposito's Cutting Edge article. More great information is available on Shiju Varghese's blog post.

MVC or Web Forms?

Now that you've got a basic understanding of what ASP.NET MVC is and what it offers, you may be wondering if MVC is the way to go with your future ASP.NET applications. Let's look at a few scenarios that might help you decide if MVC is right for your next ASP.NET project.

Choose MVC if . . .

You are well-versed in the architecture of MVC. If you aren't comfortable with how to design a controller, MVC probably isn't a good choice.
You want full control over the HTML that is rendered in the browser and you can afford the development time and overhead to do all of your own markup.
You need to create efficient unit tests for your user interface without the overhead of the entire managed runtime, etc.
You want full control over how your URLs are formed.

Choose web forms if . . .

You are not familiar with designing MVC applications.
You need to minimize development time.
You want a feature-rich user-interface (such as GridViews, etc.) to display data with rich interaction without substantial development investment.
You are already invested in server controls, either from your own development or from 3rd parties.

These are just a few of the reasons why you might choose one framework over another. Obviously, no one can tell you what decision to make for your application. Ultimately, the decision is up to you, but hopefully the information I've provided here (along with some good links) will help you to decide as you approach your next project.

High memory in IIS due to scriptproxy.dll

Mike Laing — Wed, 19 Aug 2009 00:15:00 GMT

One of my customers had a problem on his IIS server, where the IIS process hosting his web application was suffering from very high memory utilization over time. Eventually the process got so high in memory that the web application became completely unresponsive and incoming requests were no longer getting served. Based on these details, we began troubleshooting the issue as a memory leak/high memory issue.

The first step was to capture a memory dump of the problematic process, when its memory usage was at its worst point. The customer had actually already captured the memory dump before calling, so we moved to the next step immediately: data analysis.

NOTE: The customer had not used the Debug Diagnostics tool to capture the dump file. The Debug Diagnostics tool is able to capture specific memory leak data which would have made the analysis much easier. However, in this instance we were still able to get to the root cause even without the leak tracking functionality.

The dump files showed the CRuntime heap growing with various allocation sizes and various number of allocations. If we looked specifically at the allocation groups, we could see that some contained data while others contained pointers to objects. Generally, large allocations contained the data (such as a 100k ASP response) and smaller allocations contained pointers (such as a pointer to a script object).

Here is a snippet from one of the larger allocations and the data it contained:

0:000> dc
098b0080 002a002a 002a002a 002a002a 002a002a *.*.*.*.*.*.*.*.
098b0090 002a002a 002a002a 002a002a 002a002a *.*.*.*.*.*.*.*.
098b00a0 002a002a 002a002a 002a002a 002a002a *.*.*.*.*.*.*.*.
098b00b0 002a002a 002a002a 002a002a 002a002a *.*.*.*.*.*.*.*.
098b00c0 002a002a 002a002a 002a002a 002a002a *.*.*.*.*.*.*.*.
098b00d0 002a002a 002a002a 002a002a 000a000d *.*.*.*.*.*.....
098b00e0 00200027 00750050 00700072 0073006f '. .P.u.r.p.o.s.
098b00f0 003a0065 00530009 00740065 00200073 e.:...S.e.t.s. .
0:000>
098b0100 006c0061 0020006c 00740073 006e0061 a.l.l. .s.t.a.n.
098b0110 00610064 00640072 00700020 00670061 d.a.r.d. .p.a.g.
098b0120 00200065 006e0069 006f0066 006d0072 e. .i.n.f.o.r.m.
098b0130 00740061 006f0069 000d006e 0027000a a.t.i.o.n.....'.
098b0140 000a000d 00200027 006e0049 00750070 ....'. .I.n.p.u.
098b0150 00730074 0009003a 006f006e 0065006e t.s.:...n.o.n.e.
098b0160 000a000d 000d0027 0027000a 00520020 ....'.....'. .R.
098b0170 00740065 00720075 0073006e 0009003a e.t.u.r.n.s.:...
0:000>
098b0180 006f006e 0065006e 000a000d 002a0027 n.o.n.e.....'.*.
098b0190 002a002a 002a002a 002a002a 002a002a *.*.*.*.*.*.*.*.
098b01a0 002a002a 002a002a 002a002a 002a002a *.*.*.*.*.*.*.*.
098b01b0 002a002a 002a002a 002a002a 002a002a *.*.*.*.*.*.*.*.
098b01c0 002a002a 002a002a 002a002a 002a002a *.*.*.*.*.*.*.*.
098b01d0 002a002a 002a002a 002a002a 002a002a *.*.*.*.*.*.*.*.
098b01e0 002a002a 002a002a 002a002a 002a002a *.*.*.*.*.*.*.*.
098b01f0 002a002a 002a002a 002a002a 002a002a *.*.*.*.*.*.*.*.
0:000>
098b0200 002a002a 002a002a 002a002a 002a002a *.*.*.*.*.*.*.*.
098b0210 002a002a 002a002a 002a002a 002a002a *.*.*.*.*.*.*.*.
098b0220 000a000d 00090027 0070006f 00690074 ....'...o.p.t.i.
098b0230 006e006f 00650020 00700078 0069006c o.n. .e.x.p.l.i.
098b0240 00690063 000d0074 0072000a 00730065 c.i.t.....r.e.s.
098b0250 006f0070 0073006e 002e0065 00750062 p.o.n.s.e...b.u.
098b0260 00660066 00720065 003d0020 00740020 f.f.e.r. .=. .t.
098b0270 00750072 000d0065 0027000a 00750020 r.u.e.....'. .u.

And here is a sample of one of the smaller allocations (16 bytes each):

030c2f00 18036524 00000000 0315cfd8 00000000 $e..............
030c2f10 00030007 00080101 09c242e0 140bc5b8 .........B......
030c2f20 00000000 00000000 0b3efb7c 00000042 ........|.>.B...
030c2f30 00000005 1c284fc8 a15ac8d4 00000000 .....O(...Z.....
030c2f40 030d7178 52495748 00070039 00080101 xq..HWIR9.......
030c2f50 6b658ba8 6b658b90 6b658b7c 6b658b64 ..ek..ek|.ekd.ek
030c2f60 6b658b4c 6b658b38 6b658b28 6b658af4 L.ek8.ek(.ek..ek
030c2f70 6b658ae4 6b658ad4 6b658ac4 6b658aac ..ek..ek..ek..ek
dc 0x030c33b0+0x8
030c33b8 18036524 00000000 09c31ab8 06a8d500 $e..............
030c33c8 00000000 00000000 00040003 00080101 ................
030c33d8 18036524 00000000 0a09ba90 00000000 $e..............
030c33e8 00030007 00080101 41d8ff20 00000000 ........ ..A....
030c33f8 00000000 00000000 0b62065c 0000004a ........\.b.J...
030c3408 00000000 6bba0068 1a858c47 00000000 ....h..kG.......
030c3418 00000800 00000000 00070005 00080101 ................
030c3428 00000190 00000000 0311a750 00000000 ........P.......
dc 0x030c33d0+0x8
030c33d8 18036524 00000000 0a09ba90 00000000 $e..............
030c33e8 00030007 00080101 41d8ff20 00000000 ........ ..A....
030c33f8 00000000 00000000 0b62065c 0000004a ........\.b.J...
030c3408 00000000 6bba0068 1a858c47 00000000 ....h..kG.......
030c3418 00000800 00000000 00070005 00080101 ................
030c3428 00000190 00000000 0311a750 00000000 ........P.......
030c3438 00000000 030c4f20 00000000 00000000 .... O..........
030c3448 00050003 00080101 18036524 00000000 ........$e......

In the large allocations we could actually see the data in a readable format. It was an ASP script sample, from one of the ASP template objects. But looking at the smaller allocations we saw that the data was in HEX, and there was a repeating 1st parameter of 18036524. This first parameter gave us a good clue as to what allocated the memory; looking at the location that the memory address pointed to, we see it was a function of the component Scriptproxy.dll:

0:000> dds 18036524
18036524 18031000 scriptproxy+0x1000
18036528 180310b0 scriptproxy+0x10b0
1803652c 180310e0 scriptproxy+0x10e0
18036530 180310f0 scriptproxy+0x10f0
18036534 18031160 scriptproxy+0x1160

dc 0x04bccbb0+0x8
04bccbb8 18036cd0 18036cb8 18036ca8 18036c94 .l...l...l...l..
04bccbc8 18036c80 18036c68 05872ad8 05872ad8 .l..hl...*...*..
04bccbd8 05872adc 00000000 00000000 05872ae0 .*...........*..
04bccbe8 00000000 05866218 04bccc00 07e42008 .....b....... ..
04bccbf8 00000000 00000000 00000000 00000000 ................
04bccc08 00000000 00000000 00000000 00000000 ................
04bccc18 00000000 00000000 00000000 00000000 ................
04bccc28 00000000 00000000 00000000 00000000 ................
dc 0x04bccfc8+0x8
04bccfd0 18036cd0 18036cb8 18036ca8 18036c94 .l...l...l...l..
04bccfe0 18036c80 18036c68 05872910 05872910 .l..hl...)...)..
04bccff0 05872914 00000000 00000000 00000000 .)..............
04bcd000 00000000 07dd5398 626f2000 04bd0cf0 .....S... ob....
04bcd010 00000000 00000000 00000000 00000000 ................
04bcd020 00000000 00000000 00000000 00000000 ................
04bcd030 00000000 00000000 00000000 00000000 ................
04bcd040 00000000 00000000 00000000 00000000 ................

:000> dds 18036cd0
18036cd0 18031f30 scriptproxy!DllGetClassObject+0x850
18036cd4 180320f0 scriptproxy!DllGetClassObject+0xa10
18036cd8 18032590 scriptproxy!DllGetClassObject+0xeb0
18036cdc 18032110 scriptproxy!DllGetClassObject+0xa30
18036ce0 18032130 scriptproxy!DllGetClassObject+0xa50
18036ce4 18032150 scriptproxy!DllGetClassObject+0xa70
18036ce8 18032170 scriptproxy!DllGetClassObject+0xa90
18036cec 18032190 scriptproxy!DllGetClassObject+0xab0
18036cf0 180321b0 scriptproxy!DllGetClassObject+0xad0
18036cf4 180321d0 scriptproxy!DllGetClassObject+0xaf0
18036cf8 18032200 scriptproxy!DllGetClassObject+0xb20
18036cfc 18032220 scriptproxy!DllGetClassObject+0xb40
18036d00 18032240 scriptproxy!DllGetClassObject+0xb60
18036d04 18032260 scriptproxy!DllGetClassObject+0xb80
18036d08 18032280 scriptproxy!DllGetClassObject+0xba0
18036d0c 180322a0 scriptproxy!DllGetClassObject+0xbc0

From this data we could see that a component called scriptproxy.dll appeared to be allocating memory to store instances of its class object, and appeared to be populating its class objects with the ASP template data. Unfortunately, the memory it was allocating was not ever being released, which was resulting in the high memory.

Because Scriptproxy.dll is not a Microsoft component, we were not able to discern from the memory dump why it was acting this way, nor whether this was expected behavior. The easiest was to resolve the customer’s high memory problem quickly was to remove the Scriptproxy.dll component and disable it from loading in the IIS processes. Once the customer did that, the memory problem no longer occurred. I then suggested to the customer that they follow up with the vendor of Scriptproxy.dll for further assistance, since really they would be the experts and would know why Scriptproxy.dll was behaving this way. The lm command inside the windbg debugger can be used to show details about a component, so I ran it against Scriptproxy to get the vendor info:

0:000> lmvm scriptproxy
start end module name
18030000 1803b000 scriptproxy (deferred)
Image path: C:\Program Files\Network Associates\VirusScan\scriptproxy.dll
Image name: scriptproxy.dll
Timestamp: Thu Nov 11 12:31:26 2004 (4193CC1E)
CheckSum: 00000000
ImageSize: 0000B000
File version: 8.0.0.955
Product version: 8.0.0.0
File flags: 0 (Mask 3F)
File OS: 40004 NT Win32
File type: 2.0 Dll
File date: 00000000.00000000
Translations: 0000.04b0
CompanyName: Network Associates, Inc.
ProductName: VirusScan Enterprise
ProductVersion: 8.0.0
FileVersion: 8.0.0.955
PrivateBuild: VSE.8.0.0.955 F1
FileDescription: VirusScan Script Proxy
LegalCopyright: Copyright© 1995-2004 Networks Associates Technology, Inc. All
Rights Reserved.

My customer did get back to me after speaking with the vendor, and it turns out that they had seen these types of issues before with Scriptproxy when it was run on server application platforms that relied on script processing. They also pointed to some publicly available documentation, https://kc.mcafee.com/corporate/index?page=content&id=KB55963, to assist our customer further.

Deadlock when storing Asp.net sessions in SQL server during peak load

jaskis — Wed, 12 Aug 2009 21:35:06 GMT

When there is more than 1 server in web farm scenario it is an obvious choice to go for session out-of-proc either with SQL server or State server. While storing session with SQLServer if you see issues with SQL deadlock during peak loads involving DeleteExpiredSessions . It goes like this…

Looking at the current implementation of the DeleteExpiredSessions stored procedure:

CREATE PROCEDURE DeleteExpiredSessions
AS
DECLARE @now DATETIME
SET @now = GETUTCDATE()
DELETE ASPState.dbo.ASPStateTempSessions
WHERE Expires < @now
RETURN 0

This would take the locks on all the expired records for deletion and these locks may be promoted to page locks. This can give rise to deadlocks with other ‘session state write statements’ when the number of records marked for deletion increases. By default this stored procedure is supposed to run every minute.

For Example: In one of the case. Looking at no. of records which are marked for deletion

Select count (*) from ASPState.dbo.ASPStateTempSessions WHERE Expires < getdate()

The number of record marked for expiry was 138,000 out of 146,000 records and the size of “ASPStateTempSessions” table was crossing ~20 GB!!

Next step was to check the session timeout value which can play a vital role in mitigating the problem.

If we have timeout value set to a high value (Which in my case was set at: 360 min) then no matter how many times “DeleteExpiredSessions” runs, it won't be able to delete records resulting in a pile-up and hence deadlock.

In a different case I found the session timeout value was set to 60 mins and we could not reduce to default 20 mins due to business requirement .Then we suggested to change the “DeleteExpiredSessions” stored procedure implementation as mentioned in the blog. (Tweaking is not recommended as you would run out of support boundaries. )

Another option is to reduce amount of data stored in Session (sessions are meant to store small things like ID’s) and lastly increase the number of servers in SQL environment to handle high load that can help in reducing chances of deadlock.

In ASP.NET 4.0 the “DeleteExpiredSessions” stored procedure has already been modified using the same approach so that it does not hold any locks on table level and we delete the records individually.

Also it introduces a new compression option for both the “out-of-process session-state” providers. When the “CompressionEnabled” configuration option is set to true, ASP.NET will compress (and decompress) serialized session state by using the .NET Framework “System.IO.Compression.GZipStream” class and that would reduce the table size.

<UPDATE>

KB 973849 fix for ASP.NET 2.0 is out :)

</UPDATE>

Till then bye bye…

Using ProcDump.exe to monitor w3wp.exe for CPU spikes

jaskis — Sat, 08 Aug 2009 02:12:26 GMT

Procdump is a light weight Sysinternal's command-line utility whose primary purpose is monitoring an application for CPU spikes and generating crash dumps during such spikes which an administrator or developer can use to determine the cause or to share the dump with PSS for further analysis. It also can serve as a general process dump utility that you can embed in other scripts.

It completely eliminates the need to maintain different utilities like Debug Daig 1.1 for 32 bit processes and adplus for 64 bit processes on the same server.

The Problem

Looking at task manager we can see w3wp.exe process is spiking CPU between 49%-60% intermittently and then immediately it goes down.

In order to troubleshoot we need to capture 2 sets of dump for w3wp.exe when process reaches 50% CPU and stays there for 3 consecutive seconds or so.This, using Debug Diag 1.1 or Adplus could be a tricky task as it would require great deal of accuracy to manually generate dumps when the actual CPU spike happens or we may end up taking dumps without capturing real activity responsible for the spike.

The Savior

ProcDump.exe comes in handy in such situation to speed up the course of action and help in getting the right set of data. It would monitor the target process for CPU spike within specified time limit and take snapshot at that point.

For example:

procdump -ma -c 50 -s 3 -n 2 5844(Process Name or PID)

-ma Write a dump file with all process memory. The default dump format includes thread and handle information.

-c CPU threshold at which to create a dump of the process.

-s Consecutive seconds CPU threshold must be hit before dump written (default is 10).

-n Number of dumps to write before exiting.

The above command would monitor the w3wp.exe till CPU spikes 50% for 3 seconds and it would take full dump at least for two iterations.

C:\Users\jaskis\Downloads\procdump> procdump -ma -c 50 -s 3 -n 2 5844

ProcDump v1.1 - Writes process dump files
Copyright (C) 2009 Mark Russinovich
Sysinternals - www.sysinternals.com

Process:            w3wp.exe (5844)
CPU threshold:      50% of system
Duration threshold: 3s
Number of dumps:    2
Hung window check: Disabled
Exception monitor: Disabled
Dump file:          C:\Users\jaskis\Downloads\procdump\w3wp.dmp

Time        CPU Duration
[23:48.35] 59% 1s
[23:48.36] CPU usage below threshold.
[23:48.37] 54% 1s
[23:48.38] 55% 2s
[23:48.39] 61% 3s
Process has hit spike threshold.
Writing dump file C:\Users\jaskis\Downloads\procdump\w3wp_080309_114839PM.dmp... Dump written.

[23:48.44] 61% 1s
[23:48.45] 59% 2s
[23:48.46] 57% 3s
Process has hit spike threshold.
Writing dump file C:\Users\jaskis\Downloads\procdump\w3wp_080309_114846PM.dmp...
Dump written.

Few of other interesting switches

-64 By default Procdump will capture a 32-bit dump of a 32-bit when running on 64-bit Windows. This option overrides to 64-bit dump.

-o Overwrite an existing dump file.

Another area worth investing time would be to write a batch file and run it as window service to take dump on CPU spike even when the user logoff from the server.

Stayed tuned for more on this….

IIS7 (and above) – Using FREB to capture dumps for a long running request

rakkim — Tue, 04 Aug 2009 20:38:13 GMT

In my previous post, I wrote about using FREB to troubleshoot a simple error message to identify the module which sets the error. For this post, let’s assume we are troubleshooting a slow running page (any page, be it aspx, asmx, html, etc). Not always I would recommend you to collect a dump to understand what’s going on inside. There are a few other ways to troubleshoot a slow running page, say enabling FREB to find out which module is taking long time to process, using RSCA (runtime status control API) data to understand the state of the server, using ETW to understand what’s going on, etc.

Let’s assume that we are at a point where we need to collect dumps and figure out what’s happening inside the process causing the slowness of the page.

FREB in IIS7 (and above) comes with an ability to execute an executable when failure definition is satisfied. For example, you can execute an executable when the page takes more than 15 seconds to come up. We can use the AdPlus.vbs shipped with the Windows Debugging Toolkit to collect manual hang dumps of the processes. Below are the steps to configure the same:

Enable Failure Request Tracing on the Website
Go into the “Failed Request Tracing Rules” feature, and click on “Add..” from the Actions pane
Select the content you want to trace
- If you want to trace only a specific page, select custom, and give the page name, and click on Next
Uncheck “Status code(s)”, and select “Time taken (in seconds)”, and enter the number of seconds as 15 seconds
- Note: This number of seconds is a value decided by you who is troubleshooting, and some pages may take more than 20 seconds to come up based on the logic, or DB activity
Click on Next, and choose Finish to activate the rule

The UI does not have an option to specify the EXE to be executed when the page takes 15 seconds to come up. You need to configure the settings using appcmd, or Configuration Editor. Before we see the appcmd commands to configure, let’s see what are those settings that need to be configured.

Take a look at the configuration reference for the system.webServer/tracing/traceFailedRequests/add. You have the customActionExe, customActionsParams, customActionsTriggerLimit, and the path. So, if you want to execute the AdPlus to collect dumps of all the workerprocesses, below are the values you need to set to:

customActionExe                 c:\windows\system32\cscript.exe

customActionParams              c:\debuggers\adplus.vbs -hang -pn w3wp.exe -o c:\dumps-quiet

customActionTriggerLimit        10

Most importantly, you need to set the customActionsEnabled at the website level (under system.applicationHost) to true.

customActionsEnabled true

Below are the appcmd to configure them:

Set the customActionExe

appcmd.exe set config "Default Web Site" -section:system.webServer/tracing/traceFailedRequests /[path='test.aspx'].customActionExe:"c:\windows\system32\cscript.exe"

Set the customActionParams

appcmd.exe set config "Default Web Site" -section:system.webServer/tracing/traceFailedRequests /[path='test.aspx'].customActionParams:"’c:\debuggers\adplus.vbs’ -hang -pn w3wp.exe -o c:\dumps -quiet"

Set the customActionTriggerLimit

appcmd.exe set config "Default Web Site" -section:system.webServer/tracing/traceFailedRequests /[path='test.aspx'].customActionTriggerLimit:"5"

or everything at one go

appcmd.exe set config "Default Web Site" -section:system.webServer/tracing/traceFailedRequests /[path='test.aspx'].customActionExe:"c:\windows\system32\cscript.exe" /[path='test.aspx'].customActionParams:"c:\debuggers\adplus.vbs’ -hang -pn w3wp.exe -o c:\dumps -quiet" /[path='test.aspx'].customActionTriggerLimit:"5"

and, most importantly setting customActionsEnabled to true on the website level

appcmd.exe set config -section:system.applicationHost/sites /[name='Default Web Site'].traceFailedRequestsLogging.customActionsEnabled:"true" /commit:apphost

Below is how your web.config would look like:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <system.webServer>
        <tracing>
            <traceFailedRequests>
                <add path="test.aspx" customActionExe="c:\windows\system32\cscript.exe" customActionParams="c:\debuggers\adplus.vbs -hang -pn w3wp.exe -o c:\dumps -quiet" customActionTriggerLimit="5">
                    <traceAreas>
                        <add provider="ASP" verbosity="Verbose" />
                        <add provider="ASPNET" areas="Infrastructure,Module,Page,AppServices" verbosity="Verbose" />
                        <add provider="ISAPI Extension" verbosity="Verbose" />
                        <add provider="WWW Server" areas="Authentication,Security,Filter,StaticFile,CGI,Compression,Cache,RequestNotifications,Module,FastCGI" verbosity="Verbose" />
                    </traceAreas>
                    <failureDefinitions timeTaken="00:00:15" />
                </add>
            </traceFailedRequests>
        </tracing>
    </system.webServer>
</configuration>

Above configuration would collect a dump of all w3wp.exe processes running, and place them in the c:\dumps folder when a request for test.aspx takes more than 15 seconds.

Hope this helps!

Uploading a file using FileUpload control fails in IE8

jaskis — Mon, 27 Jul 2009 21:36:06 GMT

I had another interesting customer issue this week, where in uploading a file using the FileUpload control was failing only with IE8. As you would do with any client-side issue, I started by collecting Fiddler traces for both the working (IE7) and non-working (IE8) scenarios. Following were the observations:

· IE 7
Content-Disposition: form-data; name="targetFile"; filename="C:\Users\<username>\Desktop\test.txt"
· IE 8
Content-Disposition: form-data; name="targetFile"; filename="test.txt"

The question that struck me was - Why would a client-side behavior affect my application? So I started reviewing the code on the server. The customer had server-side code to parse the filename and look for “\” to determine the filename extension (although this can easily be determined using the FileName property of the FileUpload control). This server-side code was failing in the case of IE8 because the entire path was not being passed and hence no “\”.

While researching further on this to determine why IE8 does not send the complete file path to the server, I came across an IE blog here. It explains that this is one of the security features implemented in IE8. An excerpt from that blog (below) explains this:

Additionally, the “Include local directory path when uploading files” URLAction has been set to "Disable" for the Internet Zone. This change prevents leakage of potentially sensitive local file-system information to the Internet. For instance, rather than submitting the full path C:\users\<username>\documents\secret\image.png, Internet Explorer 8 will now submit only the filename image.png.

Further to this, I gave the following options to the customer:

1. Change the logic in the server-side code - It should not be dependent on the client-side file path location and should simply use the FileName property of the FileUpload control.

2. Enable the IE 8 option to include the local directory path for the Internet Zone on the client-side.

However considering the number of end users a web application could have, it’s always better to control things on the server-side rather than asking client users to change their configuration.

Till then bye bye…

In-Proc Session State Management

Saurabh Singh — Wed, 22 Jul 2009 05:51:08 GMT

What is ASP.NET Session?

A session is defined as the period of time a unique user interacts with a particular web application.

HTTP is a stateless protocol, in the sense that a Web server is concerned only with the current HTTP request for any given Web page. The server retains no knowledge of previous requests.

The stateless nature of HTTP requests presents unique challenges when writing Web applications.

ASP and ASP.NET applications maintain a user’s session using a feature called session state.

In this series we are going to talk in detail about how to troubleshoot ASP.NET session state related issues.

ASP.NET SESSION MANAGEMENT METHODOLOGIES

There are three modes available for ASP.NET session state, within the worker process, SQL server or in a dedicated process which can run either on the same machine or on a different machine. We can select where to store the session data and this can be configured using the mode attribute of the <sessionState> element in the web.config file. Below is an example of the <sessionState> element:

<sessionState mode="InProc"

stateConnectionString="tcpip=127.0.0.1:42424"

sqlConnectionString="data source=127.0.0.1;Trusted_Connection=yes"

cookieless="false" timeout="20" />

The following table describes the various options and a brief description of each mode

Attribute	Option	Description
mode		Specifies where to store the session state.
	Off	Disables session state management.
	InProc	Session state is stored locally in memory of ASP.NET worker process.
	StateServer	Session state is stored outside of the ASP.NET worker process and is managed by a Windows service. The location of this service is specified by stateConnectionString attribute.
	SQLServer	Session state is stored outside of the ASP.NET worker process in a SQL Server database. The location of this database is specified using the sqlConnectionString attribute.

The table lists the optional attributes that can be used along with the mode attribute within the <sessionState> element

Attribute	Option	Description
cookieless		Specifies whether sessions without cookies should be used to identify client sessions.
	true	Indicates that sessions without cookies should be used.
	false	Indicates that sessions without cookies should not be used. This is the default setting.
timeout		Specifies the number of minutes a session can be idle before it is abandoned. The default is 20.
stateConnectionString		Specifies the server name and port where session state is stored remotely. For example, "tcpip=127.0.0.1:42424". This attribute is required when mode is StateServer.
sqlConnectionString		Specifies the connection string for a SQL Server. For example, "data source=localhost;Integrated Security=SSPI;Initial Catalog=ASPState". This attribute is required when mode is SQLServer.
stateNetworkTimeout		When using StateServer mode to store session state, specifies the number of seconds the TCP/IP network connection between the Web server and the state server can be idle before the session is abandoned. The default is 10.

InProc Session State

InProc state is considerably faster than either the state server or SQL Server session storage techniques. SQL Server and state server attribute their slowness to serialization/deserialization that happens while reading in/storing out the Session data from the SQL/State server.

. However storing the session state InProc has its own share of limitations. Some of these are

- With each app domain restart Session state is lost

- Process restart will result in loss of Session state data

- If considerable amount of data are stored in Session state, memory consumption for the process may increase to the point of experiencing issues due to high memory. Therefore, the amount of data being stored in Session state and the time frame during which these data need to be stored should be limited as much as possible.

- In case one wants to implement InProc Session state in a web farm scenario ensure we have sticky sessions enabled for the web farm as it involves storing of the session data among processes hosted on multiple servers.

Troubleshooting InProc Session Loss

If you have established that you are running into a session loss issue you should determine whether all users face the issue at the same time or the issue happens for a specific user.

In a scenario where all users face the issue at the same time, an application domain or a worker process recycle might be the cause of the issue. Process recycle can happen due to multiple reasons including an IISRESET or in case recycling options are enabled in the Application pool properties. If the process was recycled due to an IISRESET you will see the following in the System event logs

“IIS stop command received from user <domain\username>. The logged data is the status code”

However if the process recycle is due to the application pool recycling options, then we may see different events in the system log. One such example is:

You can configure the Application Pool recycling options by going to the properties of the application pool

You can also configure IIS to log application pool recycle events that might cause session loss. This can be done by using the LogEventOnRecycle metabase key. By default the key is configured to capture the following events

1. AppPoolRecycleTime

2. AppPoolRecycleMemory

3. AppPoolRecyclePrivateMemory

But we can configure it to log events for other events as well. Below is the list of the recycling events that can be logged using this key

Value (Decimal)	Description
1	AppPoolRecycleTime
2	AppPoolRecycleRequests
4	AppPoolRecycleSchedule
8	AppPoolRecycleMemory
16	AppPoolRecycleIsapiUnhealthy
32	AppPoolRecycleOnDemand
64	AppPoolRecycleConfigChange
128	AppPoolRecyclePrivateMemory

In order to enable logging for all recycle events we need to run the following command

C:\Inetpub\AdminScripts>cscript adsutil.vbs SET W3SVC/AppPools/LogEventOnRecycle 255

Here is a screenshot of the event that will be logged in the System event log when one of the above events occur.

For e.g. if we have the value for AppPoolRecycleSchedule set in the IIS metabase you will see something similar as shown below.

Reference: http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/6f43da84-38b0-422b-aa2b-195643d05d22.mspx?mfr=true

In case of a process crash, you may see the following events generated in the system event logs

In this case you need to configure a crash rule with either Debug Diagnostic 1.1 (32 bit server) or ADPLUS (works for both 32 bit and a 64 bit server however does not have a GUI).

To Download Debug Diagnostic 1.1 - http://www.iis.net/downloads/default.aspx?tabid=34&g=6&i=1286

To Download ADPLUS - ADPlus is included with the latest Microsoft Debugging Tools for Windows. To obtain the latest Microsoft Debugging Tools for Windows - http://www.microsoft.com/whdc/devtools/debugging/default.mspx

How to use ADPlus to troubleshoot "hangs" and "crashes" - http://support.microsoft.com/kb/286350/

It is possible that you will not see any entries in the event logs. In that case the problem may be due to an app domain recycle.You can figure out what is causing app domain recycles by implementing one of the methods below :-

Enable health monitoring by adding the following code in the web.config file under <System.web> element

<healthMonitoring>

<rules>

<add name="Application Events"

eventName="Application Lifetime Events"

provider="EventLogProvider"

profile="Default"

minInterval="00:01:00" />

</rules>

</healthMonitoring>

Save your web.config file and run your application. View a couple of pages and change the web.config (just add a space and delete the space anywhere in your web.config and save it). Now have a look at your Event Viewer. It should have the following logs -

Event Type: Information
Event Source: ASP.NET 2.0.50727.0
Event Category: Web Event
Event ID: 1305
Date: 4/13/2006
Time: 6:19:30 PM
User: N/A
Computer: <computer name>
Description:
Event code: 1001 Event message: Application is starting. Event time: 4/13/2006 6:19:29 PM Event time (UTC): 4/13/2006 12:49:29 PM Event ID: a01d7c9ff12a43a9baed46f7823f11cd Event sequence: 1 Event occurrence: 1 Event detail code: 0 Application information: Application domain

Event Type: Information
Event Source: ASP.NET 2.0.50727.0
Event Category: Web Event
Event ID: 1305
Date: 4/13/2006
Time: 6:19:30 PM
User: N/A
Computer: <computer name>
Description:
Event code: 1002 Event message: Application is shutting down. Event time: 4/13/2006 6:19:29 PM Event time (UTC): 4/13/2006 12:49:29 PM Event ID: a01d7c9ff12a43a9baed46f7823f11cd Event sequence: 1 Event occurrence: 1 Event detail code: 0 Application information: Application domain

Reference – http://msdn.microsoft.com/en-us/library/2fwh2ss9.aspx

You can also add the following code snippet to the Application_End event in the global.asax of your application.

public void Application_End() 
{
     HttpRuntime runtime = (HttpRuntime) typeof(System.Web.HttpRuntime).InvokeMember("_theRuntime",  BindingFlags.NonPublic | 
                            BindingFlags.Static |BindingFlags.GetField,  null,  null,  null); 
     if (runtime == null) return;
     string shutDownMessage = (string) runtime.GetType().InvokeMember("_shutDownMessage", BindingFlags.NonPublic  | BindingFlags.Instance | 
                              BindingFlags.GetField,  null,  runtime,  null); 
     string shutDownStack = (string) runtime.GetType().InvokeMember("_shutDownStack", BindingFlags.NonPublic  | BindingFlags.Instance | BindingFlags.GetField,  null,  runtime,  null); 

     if (!EventLog.SourceExists(".NET Runtime")) 
                {
                      EventLog.CreateEventSource(".NET Runtime", "Application");
                }

     EventLog log = new EventLog();
     log.Source = ".NET Runtime";
     log.WriteEntry(String.Format("\r\n\r\n_shutDownMessage={0}\r\n\r\n_shutDownStack={1}", shutDownMessage, shutDownStack), 
     EventLogEntryType.Error);
}

Reference - http://weblogs.asp.net/scottgu/archive/2005/12/14/433194.aspx

If it’s happening for only some users there is a possibility that the Session state is set to "InProc” and Web garden is enabled in IIS. If this is indeed true then you should disable Web Gardening.

If you are running Web farm and using InProc Session, then this could be a possible cause of session loss as all the Session information is stored within the Worker process. Ensure that you have NLB with sticky sessions enabled. In order to verify this we need to take a network trace and ensure that all the requests are reaching the same server they originated from.

Another common issue seen with Web farm scenarios is when the IIS Metabase is not same for all the Web servers in the farm. The main issue comes with Web site identifier mismatch. ( http://support.microsoft.com/?id=325056 )

Another thing to check is whether the <domain name> \ <server name> has special characters such as “_” in it? Check this http://support.microsoft.com/?id=316112

There can be scenarios where users get Session loss from a specific machine but not from other machine. This could be due to the browser dropping cookie. The best way to approach this is to try browsing the site with an alternate browser or use tools such as Fiddler or Network Monitor to capture the traffic and see whether we are getting the cookie.

Some of the known issues with IE are described here

Session variables are lost if you use FRAMESET in Internet Explorer 6 à http://support.microsoft.com/kb/323752

Before we troubleshoot Session state, we need an application to test it with. Below is the code for a simple ASP.NET page that writes to and reads from session state.

<%@ Page Language="VB" AutoEventWireup="false" CodeFile="Default.aspx.vb" Inherits="_Default" %>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" >
<head runat="server">
<title>Untitled Page</title>
</head>
<body>
<script runat="server">
Sub Session_Add(ByVal sender As Object, ByVal e As EventArgs)
Session("MySession") = text1.Value
span1.InnerHtml = "Session data updated! <P>" + "Your session contains: <font color=red>" + Session("MySession").ToString()
End Sub
Sub CheckSession(ByVal sender As Object, ByVal e As EventArgs)
If (IsNothing(Session("MySession"))) Then
span1.InnerHtml = "NOTHING, SESSION DATA LOST!"
Else
End If
span1.InnerHtml = "Your session contains: <font color=red>" + Session("MySession").ToString() 
End Sub
</script>
<form id="form1" runat="server">
<input id="text1" type="text" runat="server"/>
<input type="submit" runat="server" onserverclick="Session_Add" value="Add to Session State"/>
<input type="submit" runat="server" onserverclick="CheckSession" value="View Session State" /> 
<span id="span1" runat="server" />
</form>
</body>
</html>

The above page has two buttons one for adding a Session state value, and the other for retrieving the value. When you browse the application you initially get the following page

The corresponding Fiddler trace gives you the following result

At this moment no cookie has been created and only the Authorization Header is being passed to the server in the request. Now enter a random value in the textbox and click Add to Session state. In my example, I am adding “test” as the session state value. Observe the following

The corresponding Fiddler trace shows the following result

Notice in the response headers that the server is sending a Session cookie.

When the Session value is retrieved, I get the following results:

The corresponding Fiddler trace gives the following result

- The client has sent the session cookie. If the cookie is lost in transit or is not sent by the client, the server will be unable to locate the Session which will lead to a Session loss scenario.

Tools to use

Download Network Monitor 3.3:- http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=983b941d-06cb-4658-b7f6-3088333d062f

Download Fiddler: - http://www.fiddlertool.com/fiddler/

Note: - Fiddler doesn’t capture data over localhost. So use machine name or FQDN while trying to access the Web site.

Some of the other common issues that can contribute to a session loss are

1. Deleting ASP.NET 2.0 Application Sub-Directories Shuts Down the AppDomain

Reference - http://blogs.msdn.com/toddca/archive/2005/12/01/499144.aspx

2. Unhandled exceptions cause ASP.NET-based applications to unexpectedly quit in the .NET Framework 2.0

Reference - http://support.microsoft.com/?id=911816

3. Overwhelming Change Notifications

Reference - http://support.microsoft.com/?id=920970

4. Application domain recycle due to file change notifications

Reference - http://support.microsoft.com/kb/911272/en-us

5. In case of other queries regarding Session loss issues refer to - http://forums.asp.net/p/7504/7504.aspx#7504

*CREDIT goes to Mohit Hotchandani from PSS IIS/ASP.Net support for this post.

Getting the Sys.WebForms.PageRequestManagerParserErrorException while making an AJAX callback

amitsh — Tue, 21 Jul 2009 22:51:38 GMT

Recently I encountered an issue wherein our customer was running into a client side AJAX exception, Sys.WebForms.PageRequestManagerParserErrorException, while trying to make an Async Postback from an ASPX page. This exception was happening for all the clients in a specific network and looked like this:

---------------------------
Microsoft Internet Explorer
---------------------------
Sys.WebForms.PageRequestManagerParserErrorException: The message received from the server could not be parsed. Common causes for this error are when the response is modified by calls to Response.Write(), response filters, HttpModules, or server trace is enabled.

Details: Error parsing near <SOME PLACE>
---------------------------
OK
---------------------------

This issue generally comes up when the client is making an Async Postback and the response which comes back is not as expected. If anything gets appended to the response other than the expected HTML, or if the client gets a completely different response than what it expected, the client is not able to parse the response correctly and ends up throwing the above exception. The location of the response/request modification could be anywhere between the client and the server, including intermediary network devices. This article explains it really well.

Though the exception is pretty self explanatory it can be hard to track down who is modifying the response and/or even the request itself. In such cases one may have to rely on network traffic capturing tools like Netmon and Fiddler to troubleshoot it. So for investigating this issue we captured netmon and fiddler traces from the client and the server for both working and non-working scenarios. Though this clearly looked like a network issue as it was coming only for clients in specific network, we still wanted to know what was getting changed. Here is how the fiddler request and the response looks like for the test page in working scenario -

Request -

POST /AJAXWebSite/Test.aspx HTTP/1.1
Accept: */*
Accept-Language: en-us
Referer: http://mymachine/AJAXWebSite/Test.aspx
x-microsoftajax: Delta=true
Content-Type: application/x-www-form-urlencoded; charset=utf-8
Cache-Control: no-cache
UA-CPU: x86
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.2; .NET CLR 1.1.4322; MS-RTC LM 8; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; InfoPath.2; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; MS-RTC EA 2)
Host: mymachine
Content-Length: 246
Connection: Keep-Alive
Pragma: no-cache

ScriptManager1=ScriptManager1%7CTimer1&__EVENTTARGET=Timer1&__EVENTARGUMENT=&__VIEWSTATE=%2FwEPDwUJNjIzMzE1NzA5ZGT3MKmu9xSktVp3SfAtEYDZUmMU3Q%3D%3D&__EVENTVALIDATION=%2FwEWAgLSiq3NAwLs0bLrBj7PuCpvPWwAmr3CkDtU%2B87Qq6o8&TextBox1=&

Expected Response -

HTTP/1.1 200 OK
Cache-Control: private
Content-Length: 566
Content-Type: text/plain; charset=utf-8
Server: Microsoft-IIS/6.0
MicrosoftOfficeWebServer: 5.0_Pub
X-Powered-By: ASP.NET
X-AspNet-Version: 2.0.50727
Date: Tue, 21 Jul 2009 14:09:27 GMT

113|updatePanel|UpdatePanel1|
<input name="TextBox1" type="text" value="7/21/2009 10:09:27 AM" id="TextBox1" />
|0|hiddenField|__EVENTTARGET||0|hiddenField|__EVENTARGUMENT||52|hiddenField|__VIEWSTATE|/wEPDwUJNjIzMzE1NzA5ZGT3MKmu9xSktVp3SfAtEYDZUmMU3Q==|48|hiddenField|__EVENTVALIDATION|/wEWAgLSiq3NAwLs0bLrBj7PuCpvPWwAmr3CkDtU+87Qq6o8|6|asyncPostBackControlIDs||Timer1|0|postBackControlIDs|||13|updatePanelIDs||tUpdatePanel1|0|childUpdatePanelIDs|||12|panelsToRefreshIDs||UpdatePanel1|2|asyncPostBackTimeout||90|9|formAction||Test.aspx|

But when we took similar traces for the failing scenario here is what we saw for the same request -

HTTP/1.1 200 OK
Cache-Control: private
Content-Length: 2718
Content-Type: text/html; charset=utf-8
Server: Microsoft-IIS/6.0
MicrosoftOfficeWebServer: 5.0_Pub
X-Powered-By: ASP.NET
X-AspNet-Version: 2.0.50727
Date: Tue, 21 Jul 2009 14:16:53 GMT

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml">
<head><title>

</title></head>
<body>
    <form name="form1" method="post" action="Test.aspx" id="form1">
<div>
<input type="hidden" name="__EVENTTARGET" id="__EVENTTARGET" value="" />
<input type="hidden" name="__EVENTARGUMENT" id="__EVENTARGUMENT" value="" />
<input type="hidden" name="__VIEWSTATE" id="__VIEWSTATE" value="/wEPDwUJNjIzMzE1NzA5ZGT3MKmu9xSktVp3SfAtEYDZUmMU3Q==" />
</div>

<script type="text/javascript">
//<![CDATA[
var theForm = document.forms['form1'];
if (!theForm) {
    theForm = document.form1;
}
function __doPostBack(eventTarget, eventArgument) {
    if (!theForm.onsubmit || (theForm.onsubmit() != false)) {
        theForm.__EVENTTARGET.value = eventTarget;
        theForm.__EVENTARGUMENT.value = eventArgument;
        theForm.submit();
    }
}
//]]>
</script>

<script src="/aspnet_client/system_web/2_0_50727/WebForms.js" type="text/javascript"></script>

<script src="/AJAXWebSite/ScriptResource.axd?d=ktG6RH-5YekKOFKmPczdQtpmuIwrQ0_IN1WVRbcxrxJarwiTwmePaK1rX5K7SKYF6RcFAExALSoSedjYxsT-46bYYF7ifZVAG_-JOynQe0A1&t=ffffffffa46422ae" type="text/javascript"></script>
<script src="/AJAXWebSite/ScriptResource.axd?d=ktG6RH-5YekKOFKmPczdQtpmuIwrQ0_IN1WVRbcxrxJarwiTwmePaK1rX5K7SKYFksyhgYfs5XBNWIel6UBRrkKMZWUoedc8dmuiqHrMtPg1&t=ffffffffa46422ae" type="text/javascript"></script>
<script src="/AJAXWebSite/ScriptResource.axd?d=ktG6RH-5YekKOFKmPczdQtpmuIwrQ0_IN1WVRbcxrxJarwiTwmePaK1rX5K7SKYFQnR-aK0Xy6ZPnKAS2x4xTc249FkSXpb_JOTlKStYU3c1&t=ffffffffa46422ae" type="text/javascript"></script>
<div>

    <input type="hidden" name="__EVENTVALIDATION" id="__EVENTVALIDATION" value="/wEWAgLSiq3NAwLs0bLrBj7PuCpvPWwAmr3CkDtU+87Qq6o8" />
</div>
    <div>
        <script type="text/javascript">
//<![CDATA[
Sys.WebForms.PageRequestManager._initialize('ScriptManager1', document.getElementById('form1'));
Sys.WebForms.PageRequestManager.getInstance()._updateControls(['tUpdatePanel1'], ['Timer1'], [], 90);
//]]>
</script>

        <span id="Timer1" style="visibility:hidden;display:none;"></span>
        <div id="UpdatePanel1">
                <input name="TextBox1" type="text" value="7/21/2009 10:16:53 AM" id="TextBox1" />
</div>
    </div>

<script type="text/javascript">
//<![CDATA[
Sys.Application.initialize();
Sys.Application.add_init(function() {
    $create(Sys.UI._Timer, {"enabled":true,"interval":10000,"uniqueID":"Timer1"}, null, null, $get("Timer1"));
});
//]]>
</script>
</form>
</body>
</html>

Instead of getting a response for the Async Postback we were getting a response for the entire page. So it was not possible for the client to parse through it and attach the content to the controls in the UpdatePanel1 and hence it was throwing the Sys.WebForms.PageRequestManagerParserErrorException exception for the HTML tags etc. Now this led us to understand that the server was not treating the request as an Asynch Postback; rather it was treating it as a full postback and sending the entire HTML response back to the client.

In ASP.NET 3.5 and earlier we recognize the AJAX callback by the HTTP header “x-microsoftajax: Delta=true”. When we took a netmon trace between the client and the server we found the client was sending the above header as part of the HTTP request headers but it was never reaching the server. We then found that the customer had a non-Microsoft firewall which was removing all unrecognized HTTP headers including x-microsoftajax. This resulted in the server treating the request as a normal postback and the server responded back with the complete HTML instead of the partial response, causing the above exception on the client side.

There are few firewalls which are capable of removing the HTTP Headers. There are some which do not recognize the x-microsoftajax header as well. We can use one of the options below to fix the issue:

1. Turn off the option in the Firewall settings which disallows unrecognized HTTP Headers – the HTTP protocol gives the capability to write custom header(s) to developers. By disallowing unrecognized HTTP Headers the firewall is limiting this capability of the HTTP protocol.

2. Upgrade to Service Pack 1 of .NET Framework 3.5 – As there are few firewalls which disallow the x-microsoftajax header, we did a fix for UpdatePanel to send a form value in addition to the header to work around the broken proxies/firewalls issue in 3.5 SP1. Now AJAX callback requests send a form field “__ASYNCPOST=true” along with the request and x-microsoftajax header. On the server side we check both the Form Field and the header to verify the AJAX request.

I hope this helps!

“Parser Error Message: This implementation is not part of the Windows Platform FIPS validated cryptographic algorithms.” when .net page has debug=”true”

vivekkum — Mon, 20 Jul 2009 19:37:00 GMT

Recently, I came across an issue where the customer faced an FIPS (Federal Information Processing Standards) related error on the .aspx pages which had debug=”true”. His ASP.net application was hosted on IIS7 running on Windows Server 2008 SP2. And, he was able to reproduce the issue using a very simple page. The error message was:

Looking at the error, we know that there are articles like KB 911722 and a good blog - Enforcing FIPS Certified Cryptography which discuss the same issue.

In Windows Server 2008, the FIPS related registry key is:

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa\FipsAlgorithmPolicy] "Enabled"=dword:00000000

With the setting "Enabled"=dword:00000000 we know the FIPS enforcement is disabled.

While troubleshooting, we checked the Local Security Policy setting on the server, at Administrative Tools -> Local Security Policy -> Local Policies -> Security Options -> “System cryptography: Use FIPS compliant algorithms for encryption, hashing and signing” and it showed that it was disabled.

The customer said that sometime back his domain administrators enabled FIPS on their Windows Server 2003 DC and that they enforced it domain-wide using Group Policy. They had then disabled the setting due to the application starting to fail with the Parser Error.

The customer did have a workaround, they can follow the article Disabling the FIPS Algorithm Check i.e. adding <enforceFIPSPolicy enabled="false"/> in their web.config file, however the customer was keen to know the cause of the issue..

So, we know on Windows Server 2003, the registry key for FIPS is

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa] "fipsalgorithmpolicy"=dword:00000001

With this in mind, we checked the registry hive on the customer’s WIN2K8 server and there was this WIN2K3 dword value set ("fipsalgorithmpolicy"=dword:00000001). This was in fact causing the issue. It looked like the Win2k3 domain Group Policy added the registry key on the Windows Server 2008 but did not change the value of the key back when the Group Policy was changed.

I tried to reproduce the issue on my own server by forcing FIPS using Windows Server 2003 DC on my Windows Server 2008 IIS server, and the registry key does gets modified when I enable FIPS from GPO. After disabling FIPS from GPO I did “gpupdate /force” from the command prompt on my Windows Server 2008 server and the registry value is changed from 1 to 0.

Resolution:

To resolve the customer’s problem, we removed "fipsalgorithmpolicy"=dword:00000001 from [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa].

Doing a bit more research into the mechanics of the issue, here is what I believe happens: first .NET checks to see if the applicable config file (web.config) has the FIPS enforcement disabled. If the config file doesn't have it disabled then it calls BCryptGetFipsAlgorithmMode (public API) to see what is set in the registry. Inside BCryptGetFipsAlgorithmMode it first reads "HKLM\System\CurrentControlSet\Control\Lsa\FIPSAlgorithmPolicy\Enabled". If that is not set it then reads "HKLM\System\CurrentControlSet\Control\Lsa\FIPSAlgorithmPolicy"

HTH.

Troubleshooting HTTP 401.3 errors with Process Monitor

KyleT — Fri, 26 Jun 2009 04:16:00 GMT

Last week I posted the following blog which showed how to use Process Monitor to troubleshoot service startup issues.

http://blogs.msdn.com/webtopics/archive/2009/06/16/troubleshooting-service-startup-issues-with-process-monitor.aspx

To continue on that topic, I ran across another issue recently where Process Monitor was again very helpful in troubleshooting.

Problem – When browsing ASP pages, we were getting below error in the browser. Browsing any HTML page worked fine.

HTTP Error 401.3 - Unauthorized: Access is denied due to an ACL set on the requested resource.

Note that you may need to uncheck Show friendly HTTP error messages in Internet Explorer on the Advanced tab under Tools/Internet Options as shown below to see this exact error.

It is good practice to leave this option unchecked when you’re troubleshooting an issue that involves an error message in the browser, since you’ll be more likely to get a full detailed error message. You could also check the IIS logs and look at the sc-status and sc-substatus fields to see 401 and 3 respectively.

Now I captured a Process Monitor log while reproducing the issue. As usual, I started by filtering on Result is Access Denied in Process Monitor after reproducing the issue. I found that there were no related Access Denied entries. You would expect that Process Monitor would capture an Access Denied result with an ACL (Access Control List) error like HTTP 401.3.

I found that this can occur when the account the IIS worker process was launched as has access to a particular file, but the user that authenticated through website does not. To see this in Process Monitor, I first filtered on w3wp.exe for the process name, which is the worker process that hosts a web site in IIS 6.0.

I then looked through the Process Monitor output for the access to C:\WINDOWS\Help\iisHelp\common\401-3.htm, which is the file IIS reads in by default to display the 401.3 error in the browser. Once I found that, I looked for what showed up in the log just before then.

The last thing that was accessed before the 401-3.htm file was asp.dll. Checking the permissions on that file and comparing it to a default installation, I found the permissions were the same.

See the entry highlighted above that shows a successful access to asp.dll for a QuerySecurityFile operation. The Detail column shows the information queried was the Owner, Group, and DACL (Discretionary Access Control List). Basically what’s going on here is the w3wp.exe process accesses asp.dll using the process identity (NETWORK SERVICE by default). We don’t see an Access Denied result because the process identity has the appropriate permissions. Now it checks to make sure that whatever user browsed the site also has access to asp.dll. In this case, Anonymous Authentication was enabled so the IUSR_MachineName account was checked, which did not have the appropriate permissions.

The issue here turned out to be that the NT AUTHORITY\Authenticated Users group was removed from the local Users group. Adding it back and then restarting IIS resolved the issue. The point here is that using Process Monitor I was able to narrow the issue down to the user accessing the site not having permissions on a specific DLL. In this case, IUSR_MachineName was an Authenticated Users, which should be a member of the Users group, and Users have permissions on asp.dll.

Summary

Capture Process Monitor log while reproducing issue
Filter on Access Denied
No Access Denied, so filter on w3wp.exe
Look for access to 401-3.htm
Review entries before the 401-3.htm to determine what file was accessed last
Check permissions based on successful QuerySecurityFile operation for last file accessed (in this case was asp.dll)

References

Below are a couple of articles we use frequently for HTTP 401.3 and other permissions related issues with IIS.

Default permissions and user rights for IIS 6.0
http://support.microsoft.com/kb/812614

Troubleshooting HTTP 401 errors in IIS
http://support.microsoft.com/kb/907273

How to Add a Locked Header Row to an ASP.NET GridView Control

jamesche — Wed, 24 Jun 2009 03:22:00 GMT

The GridView control is often used to display tabular data, much like an Excel spreadsheet. However, unlike Excel, the GridView control doesn't have any automatic way of locking the header row so that it doesn't scroll out of view.

Check out this example of a GridView within a DIV with the overflow-Y property set to scroll. Notice that as you scroll the GridView, the header row scrolls out of view. It would be more convenient to have a locked header row so that the header row is always visible regardless of the scoll position of the GridView. It would also be nice to have a header row that dynamically configured itself as per the data source so that if we add a new field to the GridView, a corresponding column automatically gets added to the locked header row.

Here's another example, but this time, the header row stays locked at the top of the GridView so that you can scroll without losing track of column names. There are a few steps required to implement this functionality.

Create a DIV on the page to contain the header row and set the DIV (either statically or dynamically) to the same width as the DIV that contains the GridView.
Insert an ASP.NET Table control into the header DIV.
In Page_Init, dynamically add the necessary cells to the header table and set the appropriate properties so that it matches the appearance of the rest of the GridView.
Check for sortable columns and dynamically add sorting LinkButtons to the header row.
Add code to handle the LinkButton's Click event and sort appropriately.

Let's go through each of these steps. (If you'd prefer to work with a working project, you can download the project at the end of this post.)

Creating the Header DIV and Inserting an ASP.NET Table Control (Steps 1 and 2)

To create a locked header row, we need to disable the header for the GridView control and instead dynamically create our own header.

Select the GridView control and change the ShowHeader property to false.
Create a new DIV above the DIV containing the GridView control using the following code.

<div id="DivHeader" style="vertical-align:top;width:650px;">
    <asp:Table ID="HeaderTable" runat="server"
               CellPadding="4"
               CellSpacing="0"
               Font-Size="11pt"
               BackColor="#274511"
               ForeColor="White">
    </asp:Table>
</div>

Note that the properties for my Table control were derived from the theme applied to my sample page. You can set these properties as you wish or dynamically set them so that they adjust to the GridView's properties automatically.

Notice that this Table control doesn't contain any cells. We'll add the necessary cells in Page_Init in the next step.

Add Code to Page_Init to Dynamically Create Header Cells and LinkButtons (Steps 3 and 4)

To dynamically add the cells for the header table, add the following code to Page_Init.

protected void Page_Init(object sender, EventArgs e)
{
    TableRow headerRow = new TableRow();

    for (int x = 0; x < GridView1.Columns.Count; x++)
    {
        DataControlField col = GridView1.Columns[x];

        TableCell headerCell = new TableCell();
        headerCell.BorderStyle = BorderStyle.Solid;
        headerCell.BorderWidth = 1;
        headerCell.Font.Bold = true;

        // if the column has a SortExpression, we want to allow
        // sorting by that column. Therefore, we create a linkbutton
        // on those columns.
        if (col.SortExpression != "")
        {
            LinkButton lnkHeader = new LinkButton();
            lnkHeader.PostBackUrl = HttpContext.Current.Request.Url.LocalPath;
            lnkHeader.CommandArgument = col.SortExpression;
            lnkHeader.ForeColor = System.Drawing.Color.White;
            lnkHeader.Text = col.HeaderText;
            lnkHeader.Click += new EventHandler(HeaderLink_Click);
            headerCell.Controls.Add(lnkHeader);
        }
        else
        {
            headerCell.Text = col.HeaderText;
        }

        headerCell.Width = col.ItemStyle.Width;
        headerRow.Cells.Add(headerCell);

    }

    HeaderTable.Rows.Add(headerRow);
}

This code adds a new TableCell to the header table and then sets specific properties on that TableCell based on whether or not the GridView's column has a SortExpression specified. If it doesn't, this code just sets the Text property of the cell to the GridView column's HeaderText property. Otherwise, it creates a LinkButton and sets the following properties on it.

PostBackUrl - Set to the local URL of the current request.
CommandArgument - Set to the SortExpression of the GridView's column so that we can apply it when the LinkButton is clicked.
ForeColor - Set to White in my example, but you can set this dynamically if you wish to make the example more generic.
Text - Set to the GridView column's HeaderText property.

I also add a new event handler for the Click event of the LinkButton control. Let's look at the code for that event.

Code for the LinkButton's Click Event

The code in the LinkButton's Click event applies the necessary SortExpression based on which LinkButton was clicked. The code for the Click event appears below.

protected void HeaderLink_Click(object sender, System.EventArgs e)
{
    LinkButton lnkHeader = (LinkButton)sender;
    SortDirection direction = SortDirection.Ascending;

    // the CommandArgument of each linkbutton contains the sortexpression
    // for the column that was clicked.
    if (GridView1.SortExpression == lnkHeader.CommandArgument)
    {
        if (GridView1.SortDirection == SortDirection.Ascending)
        {
            direction = SortDirection.Descending;
        }
    }

    GridView1.Sort(lnkHeader.CommandArgument, direction);
}

This code checks the CommandArgument for the LinkButton that was clicked (retrieved from the EventArgs) and applies the ascending or descending SortDirection based upon the result. It then calls the Sort method on the GridView.

AJAX-Enabling the Sample

If you are using ASP.NET AJAX, the code as-is will not do a partial postback. In order to have the LinkButtons sort using an AJAX async postback, you need to replace the line of code that sets the PostBackUrl property of the LinkButton with the following code.

lnkHeader.ID = "Sort" + col.HeaderText;

Doing this will allow ASP.NET Viewstate to track the LinkButton so that ASP.NET AJAX will work with the LinkButton.

Using this example, you can easily apply a locked header row to your GridView controls. If you'd like the completed sample project, you can download it below.

Sample Project

Size: 12KB
Download Now

Note: This download is hosted on my personal site, JimcoBooks.com.