Never doubt thy debugger

There are a variety of web applications out there which are relying on http headers for different purposes: automatic redirection, streaming a binary file to the client, controlling how content is cached on the client, adapting the site’s functionalities and interface to the capabilities of the browse and a lot more I’m sure you can think to.

If you’re upgrading to ASP.NET 2.0 (or higher) an existing application which relies on http headers, you might encounter some problems, especially in the case you’re producing binary context (say a PDF file) to your clients: corrupted file, or type not supported, or inability to print the downloaded document are some of the symptoms you may get.

First, check your code if you have something like the following:

Response.ContentType = "application/pdf"; 
Response.AppendHeader("Content-Disposition", "attachment; filename=document.pdf"); 
Response.AddHeader("Content-Length", m.GetBuffer().Length.ToString()); 
ObjPdf writer = ObjPdf.getInstance(document, m); 
document.Open(); 

Try changing it to this:

Response.ContentType = "application/pdf"; 
Response.AppendHeader("Content-Disposition", "attachment; filename=document.pdf"); 
ObjPdf writer = ObjPdf.getInstance(document, m); 
document.Open(); 
Response.AddHeader("Content-Length", m.GetBuffer().Length.ToString()); 

If m.GetByffer().Length is zero then you have a problem, so it’s important to open the writer object before adding the header.

useUnsafeHeaderParsing

If that’s enough or you don’t want to change your code (maybe because too many pages are affected) then you can change your web.config:

<configuration> 
    <system.net> 
        <settings> 
            <httpWebRequest useUnsafeHeaderParsing=”true” /> 
        </settings> 
    </system.net> 
</configuration>

The useUnsafeHeaderParsing config option will relax the header parsing so that headers do not have to strictly follow the standard described in the HTTP RFC. This option has been added for backwards compatibility, because the header parsing has been hanged to be very strict. Unfortunately a fair number of servers do not correctly follow the RFC, so clients using these servers will probably break due to this change. However, not using strict header parsing is a security risk, because malicious servers could send the client malformed headers which the client will then handle incorrectly. If you don't use the config option to turn off the strict parsing you probably won't get the server protocol error, but you open up the client to attack. The best solution is to try and get the server fixed.

When this property is set to false, the following validations are performed during HTTP parsing:

• In end-of-line code, use CRLF; using CR or LF alone is not allowed
• Headers names should not have spaces in them
• If multiple status lines exist, all additional status lines are treated as malformed header name/value pairs
• The status line must have a status description, in addition to a status code
• Header names cannot have non-ASCII chars in them. This validation is performed whether this property is set to true or false

When a protocol violation occurs, a WebException exception is thrown with the status set to ServerProtocolViolation. If the UseUnsafeHeaderParsing property is set to true, validation errors are ignored. Setting this property to true has security implications, so it should only be done if backward compatibility with a server is required.

Carlo

I saw this error twice recently, but as often happens for two completely different cases so here they are, hope it helps someone to same their time…

Invalid postback or callback argument. Event validation is enabled using <pages enableEventValidation="true"/> in configuration or <%@ Page EnableEventValidation="true" %> in a page. For security purposes, this feature verifies that arguments to postback or callback events originate from the server control that originally rendered them. If the data is valid and expected, use the ClientScriptManager.RegisterForEventValidation method in order to register the postback or callback data for validation

Nested forms

The first problematic application was dynamically building page layout and manipulating the HTML stream sent to the client; in my specific case there was some manipulation carried on the client through Javascript, but I think the same may happen if for example the HTML stream is changed through an HttpHandler after the main ASP.NET processing has been completed. The page was rendered correctly within the browser, but a postback thrown the exception above and this was due to a malformed page structure, where we had nested <form> tags like in the following example:

<body>
    <form id="form1" runat="server">
        <div>
            <asp:Button ID="Button1" runat="server" Text="Button" />
            <form></form>
        </div>
    </form>
</body>

There was a bug in the page, so the solution is to fix it to avoid the nested <fom> tags.

Hidden postback control

Here we had a DataGrid and on the ItemCreated event the customer was calling Item.Attributes.Add() to add a call to __DoPostBack() and have an automatic postback when a record was selected; needless to say, the postback attempt returned the error at the beginning of the post.

The customer had defined a “Select” column with a button to actually select the row within the DataGrid and set the Visible property of this button to false; ASP.NET renders the button to select the record as a LinkButton. This also emits the __DoPostBack() javascript method and registers it for event validation calling ClientScript.RegisterForEventValidation(); as you can imagine, if the control is not visible (i.e. Visible=”false”) the control is not rendered and therefore is not registered for event validation. But the DataGrid uses that event anyway, which ultimately throws the error.

The solution in this case is to manually register each row for event validation with code similar to the following:

protected override void Render(HtmlTextWriter writer) 
{ 
    foreach (DataGridItem row in DataGrid1.Items) 
    ClientScript.RegisterForEventValidation(row.UniqueID.ToString() + ":_ctl0"); 
    base.Render(writer); 
} 

The event validation must be registered within the Render() method of the page.

Also note that there could be different degrees of added complexity to this scenario: for example if the name of the control causing the postback is built dynamically and maybe the application has been migrated from ASP.NET 1.1 to 2.0, you have to be aware that the naming convention for dynamic controls has changed. ASP.NET 1.1 uses DataGrid1$_ctl2$_ctl0 while ASP.NET 2.0 uses DataGrid1$ctl02$ctl00. It is possible to control this behavior setting xhtmlConformance=”Legacy” (see xhtmlConformance); however note that reverting back to Legacy mode causes ASP.NET 2.0 to use a column (“:”) sign in control names for event validation, instead of the dollar sign (“$”) which uses for rendering.

Carlo

For my work I have two desktops and a laptop I always bring with me, and  despite all the online synchronization tools out there (SkyDrive, FolderShare, Groove, Mesh etc…) I’m used to SyncToy to keep my important files and folders updated across the three machines; The same is true for my backup .pst files: the laptop is my main machine, I usually make my changes and archives there and then copy the pst on the other two machines.

But since Vista (and now also with Windows 2008) when I copy the new file and then try to load the data file in Outlook, I always get an access denied error:

Clearly a permission issue and running Outlook with elevated privileges resolves the problem; but explicitly granting Full Control to my account (by the way, I’m member of the Administrators group), taking ownership of the file etc… is not enough, I was still unable to open the file (and I don’t want to run Outlook as Administrator). After many attempts as a last resource I tried to create through Outlook a new empty pst file with the same name of my archive one, and then I overridden it the file I was desperately trying to open and… magic… it worked!

Apparently from the UI everything was fine, so I gave the command line a try; here’s what icacls shows for the the working (pst) and non working (pst.bad) archive folder and files:

See the difference?

New Integrity Levels and Mandatory Labels in Vista/Win2008

To make a long story short, the behavior is due to the new Integrity Mechanism first introduced in Vista (see Windows Vista Integrity Level Technical Reference):

The Windows integrity mechanism is a core component of the Windows security architecture that restricts the access permissions of applications that are running under the same user account and that are less trustworthy.

The Windows Vista® integrity mechanism extends the security architecture of the operating system by assigning an integrity level to application processes and securable objects.

Appendix B: icacls and file integrity levels has some more details, and also the article Internet Explorer 7 no longer works after you move the contents of the Temporary Internet Files folder to another folder on a Windows Vista-based computer suggests an interesting solution:

icacls folder_path /setintegritylevel L

Actually on my machine I set integrity level to “M” (medium instead of low), and now I have my archive back in Outlook!

I wasted a lot of time and bumped my head on the desk because of this problem, hope at least saves some time to others… 

Carlo

Remember this problem? Well, Tom (and Jeremy and Vandana) has a solution

Carlo

I stumbled on this while trying to repro a remote debug problem for a customer: as you might know from this post from Johan, debugging ASP.NET on IIS7 needs some special care about the application pool recycling policy otherwise if you’ll not be fast enough IIS will kill your process.

Well, I was doing some tests and wanted to disable the Regular Time Interval value through the IIS Manager (default for this property is 1749 minutes, i.e. 29 hours):

But if you try to set the value to zero, I’ll get an error message (at least in Windows 2008 RTM).

Luckily we can still use AppCmd to change our config store, here it is how:

appcmd.exe set AppPool <AppPoolNameHere> /recycling.periodicRestart.time:00:00:00

And if you now go back to the UI you’ll see the change correctly reflected

Of course you can manually change your applicationHost.config file, but handle it with care and be sure you always have a backup first!

appcmd add backup <BackupNameHere>

Use the Administration Pack

It’s still a preview as of now, but it contains some really interesting new features for the IIS Manager. For example you can select the server node within the Connections left panel and under the Management section you’ll now have a new Configuration Editor feature if you don’t want to use AppCmd or edit the xml file directly:

If you select applicationPools and then enter the (Collection) box, you can change the periodicRestart.time value through the new dialog and from here you can set it to zero without error messages!

By the way, also note the small hint I highlighted: that means you can set intervals to “00:00:00”, “00:01:00”, “00:02:00” etc…, but not “00:01:30” for example; you’ll get the same from AppCmd too:

Carlo

As my fond readers might remember, I already rambled about the possibility of changing the blog title last year; nothing has really changed since that post, except the fact that every now and then I’ve been playing with the idea but at last I always put it back in a drawer with the promise to go back to it, sooner or later.

A couple of days ago I by chance found a site with a few quotes from Dr.House. This is a TV series which I didn’t want to see for quite some time because I don’t really like those “first aid” movies (E.R. is another example) and also I guess because I’m always a bit reluctant to “follow the crowd” and didn’t want to find myself watching something just because it is so popular. But then on a sleepy evening, too tired (or should I say lazy?) to go outside and with nothing interesting to watch on TV, I decided to give this “Dr. House thing” a try… and I liked it . Maybe it is for his care and acumen concealed as cinism, or the fact that they proceed with some analysis, make assumptions while trying to understand what’s going wrong and test their therapy till the resolution of the case (I see a parallel with what we in CSS do every day, don’t you?), maybe for the subtle psychological implications… but without being an addict I try to not lose my weekly episode now.

Well, back to the Dr. House quote site, I found a few of them funny and interesting:

• I don't ask why patients lie. I just assume they all do
• I've been a doctor for years. Why do I have to keep assuring people I know what I'm doing?
• Dr. Chase: How would you feel if I interfered in your personal life? Dr. House: I'd hate it. That's why I cleverly have no personal life
• Patients always want proof. We're not making cars here, we don't give guarantees
• You could think I'm wrong, but that's no reason to stop thinking?
• If you can fake sincerity, you can fake pretty much anything
• There's nothing in this universe that can't be explained. Eventually
• How come God gets credit whenever something good happens?
• Make a note: I should never doubt myself

The list is much longer, but that’s not the point.

The last quote is the easier one to adapt to a debugging blog like mine, and since I also like Shakespeare and ancient English literature (not as easy at the beginning being a non native English speaker, but very nice once you get into it) here is my choice:

“Never doubt thy debugger”

What do you think?

Debugger never lies, at most it tells things we cannot understand… but that’s another story (and making it clear is the purpose of this and other blogs, by the way!)

Carlo

If you have a dual boot machine like me (I installed Vista and Windows 2008 on one of my desktops in office) and you’re tired of synchronizing your Favorite folders like I was, here’s a quick trick: open the registry editor (as usual be careful!) and set HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Explorer\Shell Folders\Favorites and HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Explorer\User Shell Folders\Favorites to the same folder location (for example “c:\favorites”). The former is the one read by Internet Explorer, while the latter is used to show your Favorites under the Start menu.

Happy bookmarking

Carlo

I already touched the subject in a couple of previous posts and replying to direct comments and question I got, and to confirm that we’re doing something (hopefully in the right way ) on this matter I want to highlight this news from Jeff: Migrating hotfixes to MSDN Code Gallery.

The essence is that Visual Studio, .NET and other technologies hotfixes can be downloaded directly from http://code.msdn.microsoft.com/Project/ProjectDirectory.aspx?ProjectSearchText=hotfix and start a discussion with other people on the matter, hope you’ll find this useful (and keep the feedback coming of course).

Carlo

Let’s say you developed an ASP.NET application the assemblies are hosted on a network share: if you try to get the authenticated user using System.Environment.UserName you’ll get the account configured to access the network share in IIS (Connect As User). Side note: you’re also using Integrated Authentication and impersonate=”true” in web.config.

Why?

Well… ASP.NET impersonates the “Connect As” user you specify in IIS console, therefore Context.Identity.Name returns that account. If you need the authenticated user you must rely on the “old” Request.ServerVariables[“LOGON_USER”] as you were used to do in classic ASP.

Carlo

This is what we got couple of weeks ago: an Ajax enabled ASP.NET web application was using a TreeView control within an UpdatePanel to show a complex tree of hierarchical data (the sample we got had more than 2.000 nodes with varying degrees of nesting). The result was that browsing the tree within IE was considerably slower if compared with Firefox, about 4 times (where Firefox took 2 seconds to complete the operation, IE took 8 seconds a and sometimes a bit more).

With a repro in our hands, a network trace demonstrated that the communication between client and server was working fine (the problem reproduced also on localhost); while having the problem IE was burning a lot of CPU and the relevant thread had a callstack link this:

kernel32!TlsGetValue+0xb 
jscript!GcContext::GetGc+0xc
jscript!NameTbl::FBaseThread+0xb 
jscript!NameTbl::GetVal+0xe 
jscript!VAR::InvokeByName+0x10d 
jscript!VAR::InvokeDispName+0x43 
jscript!VAR::InvokeByDispID+0xb9 
jscript!CScriptRuntime::Run+0x167f 
jscript!ScrFncObj::Call+0x8d 
jscript!NameTbl::InvokeInternal+0x40 
jscript!VAR::InvokeByDispID+0xfd 
jscript!VAR::InvokeByName+0x165 
jscript!VAR::InvokeDispName+0x43 
jscript!VAR::InvokeByDispID+0xb9 
jscript!CScriptRuntime::Run+0x167f 
jscript!ScrFncObj::Call+0x8d 
jscript!NameTbl::InvokeInternal+0x40 
jscript!VAR::InvokeByDispID+0xfd 
jscript!VAR::InvokeByName+0x165 
jscript!VAR::InvokeDispName+0x43 
jscript!VAR::InvokeByDispID+0xb9 
jscript!CScriptRuntime::Run+0x167f 
jscript!ScrFncObj::Call+0x8d 

[...]

mshtml!CBase::InvokeEvent+0x1ad 
mshtml!CBase::FireEvent+0x105 
mshtml!CXMLHttpRequest::Fire_onreadystatechange+0x5b 
mshtml!CXMLHttpRequest::CReadyStateSink::Invoke+0x1d 
[...]

The code behind that was quite simple:

protected void onSelectNodeChange(object sender, EventArgs e)
{
    Session["selectedNode"] = TreeView1.SelectedNode.ValuePath;
    Session["idFirstTree"] = TreeView1.SelectedNode.Value;

    UpdatePanel2.Update();
}

The problem is that the hole tree is destroyed and recreated at every iteration (Sys$WebForms$PageRequestManager$_destroyTree in MicrosoftAjaxWebForms.js); the method traverses the whole DOM tree, iterates through and releases them. The tree is actually made of a table with many TableRows and TableCells and many of those cells have nested DIVs as for example

<TD>
    <DIV style="WIDTH: 20px; HEIGHT: 1px"></DIV>
</TD>
<TD>
    <DIV style="WIDTH: 20px; HEIGHT: 1px"></DIV>
</TD>
<TD>
    <DIV style="WIDTH: 20px; HEIGHT: 1px"></DIV>
</TD>
<TD>
    <DIV style="WIDTH: 20px; HEIGHT: 1px"></DIV>
</TD>
<TD>
    <A id=ctl00_TreeView1n121 href="javascript:TreeView_ToggleNode(ctl00_TreeView1_Data,121,ctl00_TreeView1n121,' ',ctl00_TreeView1n121Nodes)">
    <IMG style="BORDER-TOP-WIDTH: 0px; BORDER-LEFT-WIDTH: 0px; BORDER-BOTTOM-WIDTH: 0px; BORDER-RIGHT-WIDTH: 0px" alt="Expand Note" src="/MyApp/WebResource.axd?d=poTVB8BI5WzFAOGATmVHQ92ltsYfnmkuDjQn4Ufor001&amp;t=633363364222850001">
    </A>
</TD>
<TD class="ctl00_TreeView1_2 ctl00_TreeView1_4" onmouseover="TreeView_HoverNode(ctl00_TreeView1_Data, this)" style="WHITE-SPACE: nowrap" onmouseout=TreeView_UnhoverNode(this)>
    <A class="ctl00_TreeView1_0 ctl00_TreeView1_1 ctl00_TreeView1_3" id=ctl00_TreeView1t121 title="Node one \ Subnode one \ Subnode two \ Subnode three \ Leaf" onclick="TreeView_SelectNode(ctl00_TreeView1_Data, this,'ctl00_TreeView1t121');" href="javascript:__doPostBack('ctl00$TreeView1','5\\6\\7\\8\\9')">Select
    </A>
</TD>

This is a very CPU intensive operation in the mshtml model, and unfortunately the IE script and HTML engine have some troubles and performance problems in some scenarios… anyway this is not one of them. The point is that destroy and recreate the content of an UpdatePanel is normal (this is how Ajax works), what’s wrong is that the entire tree is destroyed and recreated thousands of times…

The TreeView control uses some scripts injected by ASP.NET but those are in conflict with the uses used by Ajax for the UpdatePanel and this is what origins the sort of recursion we saw in the dump. From a supportability point of view this is a known issue and one of the reasons because the TreeView control is incompatible and not supported when hosted within an UpdatePanel:

The following ASP.NET controls are not compatible with partial-page updates, and are therefore not designed to work inside an UpdatePanel control:

· TreeView control under several conditions. One is when callbacks are enabled that are not part of an asynchronous postback. Another is when you set styles directly as control properties instead of implicitly styling the control by using a reference to CSS styles. Another is when the EnableClientScript property is false (the default is true). Another is if you change the value of the EnableClientScript property between asynchronous postbacks. For more information, see TreeView Web Server Control Overview.

(from http://msdn2.microsoft.com/en-us/library/bb386454.aspx)

A TreeView is always completely rendered in HTML and sent to the client. Depending on the ExpandedState, it sets only the parts visible that should to be seen initially and leaves the rest invisible; this is done by setting the display style of the DIVs and other tags which are part of the tree structure.

• Without the UpdatePanel (i.e. without using Ajax) here’s what happens:
  • When EnableClientScript=”true” then the TreeView client site script expands/collapses the nodes; this is done calling TreeView_ToggleNode(), setting the display stile to block and updating some status values like ExpandedState. This makes the relevant parts of the tree visible without requesting data from the server. A request is sent to the server only when a node is selected; in such a case the whole tree is sent back to the server with the selected node’s style modified accordingly.
  • With EnableClientScript=”false” on every expand/collapse operation the currently visible part of the tree is sent back to the server
• When using a TreeView inside an UpdatePanel:
  • When EnableClientScript=”true” then the TreeView client side script expands/collapses the nodes like happens in the case without Ajax. When a node is selected then the whole content of the UpdatePanel (which is the whole TreeView server control) is posted back to the server with the selected node’s style modified
  • With EnableClientScript=”false” the server renders only the visible part of the tree and sends it to the client, i.e. the server only send to the client the markup needed to display the visible part of the tree. Every time a node is expanded, and Ajax request is sent to the server to get the portion of the tree to be displayed; this partial tree is what is sent back to the server with an Ajax postback (the already visible tree part plus the newly expanded tree).

Conclusion

In this specific scenario we set EnableClientScript=”false” for the TreeView control and let the Ajax scripts deal with the dynamic behaviors, and that is enough as long as you don’t need specific behaviors you get only through the TreeView client-side scripts. Bottom line: test very carefully this scenario since it might work but it’s still not supported (and not tested by Microsoft) therefore something might suddenly go wrong and we would have very limited possibilities to help you effectively…

P.s.
Thanks to Stefano Pronti and Karl Tietze for sharing this one.

Carlo

An interesting change which you might have missed in .NET 2.0 is about the autoConfig setting for the <processModel> element. The customer was running out of available connections while accessing a remote web service and was trying to apply what’s suggested in Contention, poor performance, and deadlocks when you make Web service requests from ASP.NET applications but apparently the change had not effect: a memory dump shown still 12 available connections even if he set MaxConnection=96. Interesting to note that reverting the application to .NET 1.1 the problem disappeared.

The different behavior is determined by the autoConfig value, which affects only ASP.NET applications and not Console, WinForm etc… (while the <processModel> threading configuration in .NET 1.x affected all .NET apps running on the machine):

Specifies whether to automatically configure the following settings to achieve optimal performance based on the machine configuration:

• The maxWorkerThreads attribute
• The maxIoThreads attribute
• The minFreeThreads attribute of the httpRuntime element.
• The minLocalRequestFreeThreads attribute of the httpRuntime element
• The maxConnection attribute of the <connectionManagement> Element (Network Settings) element.

The values are set according to the KB article at http://support.microsoft.com/?id=821268.
This attribute does not affect the .NET Framework client applications; only ASP.NET applications.
The autoConfig attribute can be one of the following values.

• True: Indicates that ASP.NET automatically configures the attributes in the preceding list to achieve optimal performance based on the machine configuration.
• False: Indicates that ASP.NET should use the explicitly defined values for the attributes in the preceding list.

The default in the Machine.config file is True, unless there is a previously existing configuration.

If you don’t like this auto threading feature you can simply set autoConfig=”false” or set System.Net.ServicePointManager.DefaultConnectionLimit=<your value here>

Carlo

Things have gone a bit crazy lately, we’ve been under a huge workload and the time left for blogging was virtually non existent… but the good thing is that I’ve been working on a few interesting cases I hope I'll have time to blog about, and this is the first one.

The problem

The application was running on a Windows 2003 cluster and the well known solution to avoid session problems in a multi server environment is to store user sessions out-of-process, either using a State Server or Sql Server, and in this case we were using Sql Server; this is also the right approach if you are using Web Garden for your application pool and this is the situation we were in (web garden + state on Sql Server). But some session-wide arrays the customer was using were suddenly empty as soon as the customer increased the number of worker processes above 1 for the application pool, which is exactly what is not supposed to happen with <sessionState mode=”SqlServer” … />. At the beginning I thought to a configuration problem on the affected servers but the usual quick check on the metabase shown that was not the case, so the next logical step was to get a repro from the customer and continue from there.

Well, despite mi attempts I’ve not been able to reproduce the problem but the customer confirmed he could, so were be back again to some machine-specific issue? I was still not convinced and decided to have a closer look to the source code of that repro; the essential part was made by a couple of static arrays the customer was filling on the Page Load event and then binding to a couple of DropDownList controls: changing the selection on those controls (handled by the SelectedIndexChange event) was changing some other UI parts on the page and here is where the customer was getting an IndexOutOfRangeException, clear sign the array was unexpectedly empty.

Ok, so this time I decided to have a closed look at the internal behavior on that code within Visual Studio, and while I was inspecting some variables in the debugger I suddenly got the exception the customer reported, a Javascript alert which was created within a catch clause.

This made me think; not that IIS7 (I’m using Windows 2008 on my main workstation) was not respecting the fact that there was a debugger attached to one of its pools (Johan quickly blogged about this issue here), but rather about the fact that due to its health monitoring mechanism IIS7 recycled the application pool I was debugging, and as a consequence the application thrown the exception. But neither the customer reported a process crash not the event log had any relevant entry… Moreover the repro was not even using neither Session() nor Cache() objects, so this could not be a “classical” session lost problem. From my past experience I turned my attention to the static arrays (there is only one “instance” of a static object for the whole application, and if someone changes the array that affects everyone using it), but that was not our case since the customer and I were able to reproduce the problem on a single machine, single user. But static objects have another peculiarity: they are scoped to the AppDomain.

If you have the detective instinct you might have guessed where I am going from there… The static arrays are “loaded automatically by the .NET Framework common language runtime (CLR) when the program or namespace that contains the class is loaded” (http://msdn2.microsoft.com/en-us/library/79b3xss3.aspx), and the first time a user requested the page the array was filled with some data coming from a database. To avoid useless travels to the database, the arrays were filled only if we were not in a postback (IsPostBack=false, so we are in a GET request); when we post the page back (so we’re in a POST) the initialization code does not run because IsPostBack is true and the code assumes the array is already filled and ready to use, and if we’re lucky everything works fine. This is true until IIS decides (for its own internal philosophical reasons) that our POST will be served by the same w3wp.exe instance which already served our first GET. But… what happens if we’re sent to a new w3wp.exe instance? Remember everything was working fine with 1 process per application pool and the problem appeared only increasing the number of processes per pool (Web Garden >= 2)…

Well… we will be still issuing a POST, which will be served by a new w3wp.exe instance which will load (for the first time) the CLR and initialize the application (the AppDomain and its static fields), the requested page will run… and IsPostBack will still equals to true. So we’ll not query the database to fill the array, we’ll simply assume the array is valid (and the array object is actually valid, we’re not getting a NullReferenceException, it’s just empty) and we’ll get the IndexOutOfRangeException when trying to read it.

A bit of theory

The main subject here is isolation; this post from Chris Brumme (see also this one for more details on AppDomains) explains the very basic of this matter:

By default, static fields are scoped to AppDomains.  In other words, each AppDomain gets its own copy of all the static fields for the types that are loaded into that AppDomain.  This is independent of whether the code was loaded as domain-neutral or not.  Loading code as domain neutral affects whether we can share the code and certain other runtime structures.  It is not supposed to have any effect other than performance

Processes are isolated by definition, it’s just the OS architecture which works that way (and not only on Windows, but also on other systems); if you think at how messages are exchanged in the Windows OS, there is a complex infrastructure which works at kernel level to make the communication between processes possible (Windows is a system based on messages, which are sent to windows and processes through low level system APIs), you can understand that isolation is essential for security and stability of the entire OS and for the application running on top of it. If an application could easily read/write data belonging to other processes, it would be a perfect environment for hackers and viruses… we need the opposite (of course with some flexibility left to run our applications). Since an AppDomain is loaded within a given process, and the main purpose of an AppDomain is isolation, e.g. prevent that one application can affect other applications running within the same process, change its data etc…, it’s also clear that when you load an AppDomain within a specific process, it cannot be accessed from AppDomains loaded in other processes (of course unless you want it explicitly, then you can use .NET Remoting to create this kind of communication).

How should be memory, objects, memory addresses, resources, threads etc… shared between different processes? And if a process throws an exception which affect that shared memory, it will kill all other processes which are using that same shared memory region… this is the opposite of isolation and safety, what we must have an our operating systems. And think to a cluster/NLB environment: how could we share those static objects (again, memory, threads, resources and so on) across different machines? How could a static variable travel across processes and machines?

So, an AppDomain is specific to a process, static fields are specific to an AppDomain… Sql Server (or any kind of state server) is not the solution for the problem the customer reported; this is rather an design issue. Also the customer enabled Web Garden to have performance benefits and this is actually the idea behind Web Garden, but in specific circumstances; you lose more performance than you gain in 90% of cases and it cannot overcome the system architecture, where again processes are isolated from each other. Web Garden only increases performance in cases where you don't rely on cache and where your application is not CPU intensive; in essence it really has a measurable positive impact on performance is if you have a site like www.microsoft.com that is mostly static content and completely stateless. In most cases it causes performance to be worse because we have to maintain one cache per process and because of the overhead of having multiple processes.

Conclusion

So, the best solution for this problem is to not use Web Garden; alternatively, if the customer likes the idea to have multiple processes serving the application pool, then some re-coding is needed to change those static arrays into “simple” private objects and store them in Session() or Cache(), which is perfectly maintained using Sql Server. Bottom line is: does not use statics with Web Garden, or be sure you always check if your data objects are correctly initialized and do not simply assume they are in a good shape because you’re in a postback.

Carlo

There are circumstances where taking a dump is not possible or simply not convenient; imagine a situation where you're hunting a nasty exception you don't know where it comes from and you need to follow some complex steps to get there, or when you have dozens of that exception (let's say an Access Violation) but you're interested in a specific one you get only when on click on a specific button, or after you paginate long enough through a GridView (and of course you don't want dozens of full dumps, 1 Gb each), or if you're hunting for some security related problem and you need to inspect the security token your thread is using in a particular moment (something I did here) etc... For sure if you ask Doug he'll have dozens of good answers to this question  

Getting started

Let's assume you already have correctly setup your Windbg workspace, symbols etc..., then you can go from File > Attach to a process... (or hit F6) to get the Attach to process dialog below:

Note that if you expand the tree for your target process you'll see some information about the process' session, this is useful if for example you have multiple instances of ASP.NET worker process which loads different versions of the runtime, or if you're interested in a specific application pool (check this post) etc...

The Noninvasive flag is important. With noninvasive debugging, you do not have as many debugging actions, however you can minimize the debugger's interference with the target application. In noninvasive debugging, the debugger does not actually attach to the target application, but it rather suspends all of the target's threads and has access to the target's memory, registers, and other such information. However, the debugger cannot control the target, so commands like "g" (Go, more on this later) do not work. If you try to execute commands that are not permitted during noninvasive debugging, you receive an error message that states "The debugger is not attached, so process execution cannot be monitored".

When you end the debugging session, the debugger releases the target application and the application continues running. You should close the session by using q (Quit), .detach (Detach from Process), or WinDbg's Debug > Detach Debuggee or Debug > Stop Debugging command. If you close the debugging session by closing the debugger window or by using the Exit command on the File menu in WinDbg, your target application typically stops responding. Noninvasive debugging is useful if you do not want to end the target application at the end of the session, and the target application is running on Microsoft Windows NT or Windows 2000 (these operating systems do not support detaching from a target that the debugger has actually attached to). Noninvasive debugging is also useful if the target application has stopped responding and cannot open the break thread that you need to attach.

Now we need a breakpoint

It's important to note that when you attach the debugger to your process, you'll actually suspend all of its threads with the result that the process will be "hung", completely unable to serve any incoming request from your users; this is the reason why generally it's not a good idea to attach to a production process, unless this is the only way left... Once attached, the debugger will stop on a thread waiting for your input. Depending on what you're looking for, you might need to set a breakpoint somewhere in the application (like you can do in Visual Studio) and then let the application run until you hit the breakpoint.

The most simple command to use is:

sxe clr

The sx* commands in Windbg control the action that the debugger takes when an exception occurs in the application that is being debugged, or when certain events occur; with this command we're telling the debugger to stop on every managed exception raised. Note that the code 0xE0434F4D means you're dealing with a COM exception which is bubbled up to the CLR.

If you need to stop on a specific managed exception, then the SOS extension comes at hand with !StopOnException:

!StopOnException helps when you want the Windows Debugger to stop on a particular managed exception, say a System.OutOfMemoryException, but continue running if other exceptions are thrown. The command can be used in two ways:

• When you just want to stop on one particular CLR exception

   At the debugger prompt, anytime after loading SOS, type:

   !StopOnException -create System.OutOfMemoryException 1

   The pseudo-register number (1) indicates that SOS can use register $t1 for
   maintaining the breakpoint. The -create parameter allows SOS to go ahead
   and set up the breakpoint as a first-chance exception. -create2 would set
   it up as a 2nd-chance exception.

• When you need more complex logic for stopping on a CLR exception

   !StopOnException can be used purely as a predicate in a larger expression.
   If you type:

   !StopOnException System.OutOfMemoryException 3

   then register $t3 will be set to 1 if the last thrown exception on the
   current thread is a System.OutOfMemoryException. Otherwise, $t3 will be set
   to 0. Using the Windows Debugger scripting language, you could chain
   such calls together to stop on various exception types. You'll have to
   manually create such predicates, for example:

   sxe -c "!soe System.OutOfMemoryException 3;
           !soe -derived System.IOException 4;
           .if(@$t3==1 || @$t4==1) { .echo 'stop' } .else {g}"

The -derived option will cause StopOnException to set the pseudo-register to
1 even if the thrown exception type doesn't exactly match the exception type
given, but merely derives from it. So, "-derived System.Exception" would catch
every exception in the System.Exception heirarchy.

The pseudo-register number is optional. If you don't pass a number, SOS will
use pseudo-register $t1.

Note that you can use !soe as a shortcut for !StopOnException.

If you want to test yourself, you can try this sample page:

<%@ Page Language="C#" %>

<%@ Import Namespace="System.Data" %>
<%@ Import Namespace="System.Data.SqlClient" %>
<%@ Import Namespace="System.IO" %>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<script runat="server">

    protected void FileNotFoundButton_Click(object sender, EventArgs e)
    {
        //The file does not exists
        StreamReader reader = new StreamReader(@"c:\temp\pippo.txt");
    }

    protected void InvalidOperation_Click(object sender, EventArgs e)
    {
        SqlConnection conn = new SqlConnection("data source=fakeServer; database=fakeDb; integrated security=sspi");
        SqlCommand cmd = new SqlCommand("select * from fakeTable", conn);
        //The connection should be opened!!!
        SqlDataReader dr = cmd.ExecuteReader();
        conn.Close();
    }

    protected void SqlException_Click(object senter, EventArgs e)
    {
        SqlConnection conn = new SqlConnection("data source=fakeServer; database=fakeDb; integrated security=sspi");
        SqlCommand cmd = new SqlCommand("select * from fakeTable", conn);
        //We open the connection... but the database we're trying to connect to does not exists!!!
        conn.Open();
        SqlDataReader dr = cmd.ExecuteReader();
        conn.Close();
    }

</script>

<html xmlns="http://www.w3.org/1999/xhtml">
<head runat="server">
    <title>Exceptions</title>
</head>
<body>
    <form id="form1" runat="server">
        <div>
            <asp:Button ID="FileNotFoundButton" runat="server" Text="File Not Found" OnClick="FileNotFoundButton_Click" />
            <br />
            <br />
            <asp:Button ID="InvalidOperationButton" runat="server" Text="Invalid Operation" OnClick="InvalidOperation_Click" />
            <br />
            <br />
            <asp:Button ID="SqlConnectionButton" runat="server" Text="Sql Exception" OnClick="SqlException_Click" />
        </div>
    </form>
</body>
</html>

Let the fun begin

Run the page, attache Windbg to your worker process, assure you're loading the correct SOS version (use .loadby sos mscorwks) and you'll break into the process; now set the breakpoint, let's say on first chance InvalidOperationExpcetion:

!soe -create System.InvalidOperationException
g

Go back to your browser a click on the FileNotFound button, you'll see something like the following:

0:015> g
(814.119c): CLR exception - code e0434f4d (first chance)
(814.119c): CLR exception - code e0434f4d (first chance)
(814.119c): CLR exception - code e0434f4d (first chance)
(814.119c): CLR exception - code e0434f4d (first chance)

As you can see Windbg tracks all the exceptions but does nothing special since we told it to break only on certain exceptions; see the e0434f4d code? That indicates an unmanaged exception. Why is it there? Well... sooner or later the CLR talks to the underlying OS (to simplify) which tries to open the file we asked for, but the file does not exist... so the OS throws an exception which is bubbled up to the CLR. The same happens for the SqlException but not with DivideByZeroException (this is a special exception I'll cover later).

But if you click the right button... here we are:

'System.InvalidOperationException hit'
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
eax=03f9d7dc ebx=e0434f4d ecx=00000000 edx=00000029 esi=03f9d868 edi=032c86c0
eip=7c812a5b esp=03f9d7d8 ebp=03f9d82c iopl=0         nv up ei pl nz na po nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00000202
KERNEL32!RaiseException+0x53:
7c812a5b 5e              pop     esi

Now we can have a look at the managed stack with !clrstack:

0:012> !clrstack
OS Thread Id: 0x12c4 (12)
ESP       EIP     
03f9d8b4 7c812a5b [HelperMethodFrame: 03f9d8b4] 
03f9d958 653d7691 System.Data.SqlClient.SqlConnection.GetOpenConnection(System.String)
03f9d964 652f568a System.Data.SqlClient.SqlConnection.ValidateConnectionForExecute(System.String, System.Data.SqlClient.SqlCommand)
03f9d970 652f0eb4 System.Data.SqlClient.SqlCommand.ValidateCommand(System.String, Boolean)
03f9d9a0 652f039a System.Data.SqlClient.SqlCommand.RunExecuteReader(System.Data.CommandBehavior, System.Data.SqlClient.RunBehavior, Boolean, System.String, System.Data.Common.DbAsyncResult)
03f9d9e0 652f0331 System.Data.SqlClient.SqlCommand.RunExecuteReader(System.Data.CommandBehavior, System.Data.SqlClient.RunBehavior, Boolean, System.String)
03f9d9fc 652eee23 System.Data.SqlClient.SqlCommand.ExecuteReader(System.Data.CommandBehavior, System.String)
03f9da38 652eebe5 System.Data.SqlClient.SqlCommand.ExecuteReader()
03f9da6c 048b0954 ASP.default_aspx.InvalidOperation_Click(System.Object, System.EventArgs)
03f9da8c 6619004e System.Web.UI.WebControls.Button.OnClick(System.EventArgs)
03f9daa0 6619023c System.Web.UI.WebControls.Button.RaisePostBackEvent(System.String)
03f9dab4 661901b8 System.Web.UI.WebControls.Button.System.Web.UI.IPostBackEventHandler.RaisePostBackEvent(System.String)
03f9dab8 6614b47c System.Web.UI.Page.RaisePostBackEvent(System.Web.UI.IPostBackEventHandler, System.String)
03f9dac0 6614b3d2 System.Web.UI.Page.RaisePostBackEvent(System.Collections.Specialized.NameValueCollection)
03f9dad0 6614e263 System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)
03f9dccc 6614d8c3 System.Web.UI.Page.ProcessRequest(Boolean, Boolean)
03f9dd04 6614d80f System.Web.UI.Page.ProcessRequest()
03f9dd3c 6614d72f System.Web.UI.Page.ProcessRequestWithNoAssert(System.Web.HttpContext)
03f9dd44 6614d6c2 System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)
03f9dd58 048b0186 ASP.default_aspx.ProcessRequest(System.Web.HttpContext)
[...]

Have a look at the objects on this thread with !dumpstackobjects:

0:012> !dso
OS Thread Id: 0x12c4 (12)
ESP/REG  Object   Name
03f9d848 013ea8c4 System.InvalidOperationException
03f9d894 013ea8c4 System.InvalidOperationException
03f9d8a4 013dc4c0 System.String    ExecuteReader
03f9d8a8 013d9c74 System.Data.ProviderBase.DbConnectionClosedNeverOpened
03f9d8bc 013ea738 System.Char[]
03f9d8d8 013ea8c4 System.InvalidOperationException
03f9d8e0 013d9c74 System.Data.ProviderBase.DbConnectionClosedNeverOpened
03f9d8e4 013ea68c System.Text.StringBuilder
03f9d908 013ea90c System.String    System.InvalidOperationException
03f9d90c 013ea960 System.String    System.InvalidOperationException: ExecuteReader requires an open and available Connection. The connection's current state is closed.
03f9d910 013ea7a8 System.String    ExecuteReader requires an open and available Connection. The connection's current state is closed.
03f9d918 013ea7a8 System.String    ExecuteReader requires an open and available Connection. The connection's current state is closed.
03f9d93c 013ea8c4 System.InvalidOperationException
03f9d958 013dbf7c System.Data.SqlClient.SqlCommand
03f9d95c 013dc4c0 System.String    ExecuteReader
03f9d964 013dbf7c System.Data.SqlClient.SqlCommand
03f9d96c 013dbf7c System.Data.SqlClient.SqlCommand
[...]

Here we already have some details about the exception, we can dump objects (!do <address>) etc... as if we're analyzing a static dump file.

If you're hunting a permission problem, the !token -n command is very handy, it dumps the security token of the current thread (which might change over time if you're using impersonation):

0:003> !token -n
Thread is not impersonating. Using process token...
TS Session ID: 0
User: S-1-5-21-1721254763-462695806-1538882281-2553087 (User: EUROPE\carloc)
Groups: 
 00 S-1-5-21-1721254763-462695806-1538882281-513 ConvertSidToFriendlyName on failed with error code 0x800006fd
    Attributes - Mandatory Default Enabled 
 01 S-1-1-0 (Well Known Group: localhost\Everyone)
    Attributes - Mandatory Default Enabled 

[...]

Primary Group: S-1-5-21-1721254763-462695806-1538882281-513 ConvertSidToFriendlyName on failed with error code 0x800006fd
Privs: 
 00 0x000000017 SeChangeNotifyPrivilege           Attributes - Enabled Default 
 01 0x000000008 SeSecurityPrivilege               Attributes - 
 02 0x000000011 SeBackupPrivilege                 Attributes - 
 03 0x000000012 SeRestorePrivilege                Attributes - 
 04 0x00000000c SeSystemtimePrivilege             Attributes - 
 05 0x000000013 SeShutdownPrivilege               Attributes - 
 06 0x000000018 SeRemoteShutdownPrivilege         Attributes - 
 07 0x000000009 SeTakeOwnershipPrivilege          Attributes - 
 08 0x000000014 SeDebugPrivilege                  Attributes - Enabled 
 09 0x000000016 SeSystemEnvironmentPrivilege      Attributes - 
 10 0x00000000b SeSystemProfilePrivilege          Attributes - 
 11 0x00000000d SeProfileSingleProcessPrivilege   Attributes - 
 12 0x00000000e SeIncreaseBasePriorityPrivilege   Attributes - 
 13 0x00000000a SeLoadDriverPrivilege             Attributes - Enabled 
 14 0x00000000f SeCreatePagefilePrivilege         Attributes - 
 15 0x000000005 SeIncreaseQuotaPrivilege          Attributes - 
 16 0x000000019 SeUndockPrivilege                 Attributes - Enabled 
 17 0x00000001c SeManageVolumePrivilege           Attributes - 
 18 0x00000001d SeImpersonatePrivilege            Attributes - Enabled Default 
 19 0x00000001e SeCreateGlobalPrivilege           Attributes - Enabled Default 
Auth ID: 0:13927
Impersonation Level: Anonymous
TokenType: Primary

If you just want to have a look at the exceptions thrown by your application without necessarily breaking in the debugger or taking a dump, Windbg can print some tracking information and let the application continue:

0:006> sxe -c "!pe;!clrstack;gc" clr
0:006> g
(14c4.10a4): CLR exception - code e0434f4d (first chance)
Exception object: 013d8a20
Exception type: System.IO.FileNotFoundException
Message: Could not find file 'c:\temp\pippo.txt'.
InnerException: <none>
StackTrace (generated):
<none>
StackTraceString: <none>
HResult: 80070002
OS Thread Id: 0x10a4 (3)
ESP       EIP     
036dd6c8 7c812a5b [HelperMethodFrame: 036dd6c8] 
036dd76c 7948d83d System.IO.__Error.WinIOError(Int32, System.String)
036dd798 79395557 System.IO.FileStream.Init(System.String, System.IO.FileMode, System.IO.FileAccess, Int32, Boolean, System.IO.FileShare, Int32, System.IO.FileOptions, SECURITY_ATTRIBUTES, System.String, Boolean)
036dd890 793983c8 System.IO.FileStream..ctor(System.String, System.IO.FileMode, System.IO.FileAccess, System.IO.FileShare, Int32, System.IO.FileOptions)
036dd8bc 793a3189 System.IO.StreamReader..ctor(System.String, System.Text.Encoding, Boolean, Int32)
036dd8e0 79497d71 System.IO.StreamReader..ctor(System.String)
036dd8f4 048b08ce ASP.default_aspx.FileNotFoundButton_Click(System.Object, System.EventArgs)
036dd90c 6619004e System.Web.UI.WebControls.Button.OnClick(System.EventArgs)
036dd920 6619023c System.Web.UI.WebControls.Button.RaisePostBackEvent(System.String)
036dd934 661901b8 System.Web.UI.WebControls.Button.System.Web.UI.IPostBackEventHandler.RaisePostBackEvent(System.String)
036dd938 6614b47c System.Web.UI.Page.RaisePostBackEvent(System.Web.UI.IPostBackEventHandler, System.String)
036dd940 6614b3d2 System.Web.UI.Page.RaisePostBackEvent(System.Collections.Specialized.NameValueCollection)
036dd950 6614e263 System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)
036ddb4c 6614d8c3 System.Web.UI.Page.ProcessRequest(Boolean, Boolean)
036ddb84 6614d80f System.Web.UI.Page.ProcessRequest()
036ddbbc 6614d72f System.Web.UI.Page.ProcessRequestWithNoAssert(System.Web.HttpContext)
036ddbc4 6614d6c2 System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)
036ddbd8 048b0186 ASP.default_aspx.ProcessRequest(System.Web.HttpContext)
036ddbe4 65fe6bfb System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
036ddc18 65fe3f51 System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)
036ddc58 65fe7733 System.Web.HttpApplication+ApplicationStepManager.ResumeSteps(System.Exception)
036ddca8 65fccbfe System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest(System.Web.HttpContext, System.AsyncCallback, System.Object)
036ddcc4 65fd19c5 System.Web.HttpRuntime.ProcessRequestInternal(System.Web.HttpWorkerRequest)
036ddcf8 65fd16b2 System.Web.HttpRuntime.ProcessRequestNoDemand(System.Web.HttpWorkerRequest)
036ddd04 6600545c System.Web.HttpRuntime.ProcessRequest(System.Web.HttpWorkerRequest)
036ddd14 033f34cd Microsoft.VisualStudio.WebHost.Request.Process()
036ddd28 033f31c2 Microsoft.VisualStudio.WebHost.Host.ProcessRequest(Microsoft.VisualStudio.WebHost.Connection)
[...]

Another useful command to break on managed exception is !CheckCurrentException, but you'll likely use it in a Windbg script or adplus configuration file:

!CheckCurrentException checks if the current exception is the one specified and stores 1 or 0 in the pseudo register supplied (1 = $t1 register).  This is useful when scripting a config file to use with Adplus to get a dump only when a certain type of managed exception happens.  The config file would look like:

<ADPlus>

    <!-- Configuring ADPlus to log all first chance exceptions -->
    <!-- Will still create full dump for any type of second chance exceptions -->

  <Exceptions>
     <Config>
        <!-- This is for the CLR exception -->
       <Code> clr </Code>
       <Actions1> Log </Actions1>
       <CustomActions1> !cce System.Configuration.ConfigurationException 1; j ($t1 = 1) '.dump /ma /u c:\dumps\mydump.dmp' ; ''  </CustomActions1>
       <ReturnAction1> gn  </ReturnAction1>
       <Actions2> Log;FullDump </Actions2>
     </Config>
  </Exceptions>

</ADPlus>

Remarks: This will dump on a System.Configuration.ConfigurationException only.  It will
make a dump in the c:/dumps directory when that happens.

!bp instead sets a breakpoint on a managed function:

!Bp sets a breakpoint on the given function.  It will break even if the function isn't JIT'ed yet

!Bp <module name> <method name>
!Bp <module name>!<method name>

!sos.bp MyApp!MyFunction

And if you reach an interesting point you can always manually capture a dump with .dump /ma <dumpname>

What's next?

Debugging live you can do almost everything you would do with a static dump file, but if you're an hard core debugger you can experiment changing variable values, pointers, memory addresses contents and see the results directly in the application you're attached to, a sort of "live memory programming"... for a small sample have a look at this posts from Roberto to cheat with Minesweeper . There is a rumor here in Italy of an Engineer whom while onsite for an urgent problem fixed it directly in memory and then had just to correct and deploy the new binary, but the customer was already happily using his application without further annoyances: true or legend? Well, I know this guy and I tend to thing this is a true story .

Another interesting and powerful feature of Windbg is the ability to see registers, source code, locals, disassembly etc... (similar to what we have in Visual Studio), this is also available with a static dump file but is especially useful when debugging live... but I guess that will be the subject of another post

Carlo