Never doubt thy debugger

How to export IIS .config and .key files at the command line

2013-01-09T09:50:17Z

Handy for a batch maintenance operation a customer had to run on a few hundreds servers: they basically needed to do what’s described in this article but in a script rather than from the IIS GUI.

From Shared Configuration (Appendix 3):

By default, IIS includes two main providers for securing properties. These providers are located in the applicationHost.config file's <configProtectedData> configuration section and are defined in the <providers> element. The AesProvider is specific to dealing with encryption and decryption for properties that are in the system.webServer section. The IISWASOnlyRsaProvider is specific to dealing with encryption and decryption for properties that are in the system.applicationHost section. These keys are in the iisConfigurationKey and iisWasKey key containers and are machine-specific. In a Web farm scenario, if encryption is required, then a key from one machine - usually the one that created the applicationHost.config file - is exported and brought into the other machines so that secure properties can be decrypted and used by the Web server

So just copy the .config files from “C:\Windows\System32\Inetsrv\config” and use aspnet_regiis –px to export the Keys:

aspnet_regiis -px "iisConfigurationKey" "D:\iisConfigurationKey.xml" -pri

To import the Keys on another machine run the following:

aspnet_regiis -pi "iisConfigurationKey" "D:\iisConfigurationKey.xml"

The parameter after the -pi is the name of the key container to be imported. In this case, it is the "iisConfigurationKey" key container. The other key container that IIS uses is the "iisWasKey" key container.

Carlo

The difference between the right word and the almost right word is the difference between lightning and a lightning bug.

Mark Twain

Error 502 - Web server received an invalid response while acting as a gateway or proxy server

2013-01-09T09:30:56Z

I saw this error some time ago while working on an eCommerce solution. When the Web server (while acting as a gateway or proxy) contacted the upstream content server, it received an invalid response from the content server. Redirect/move happens like it should, but after approximatively 2 minutes after posting an order the user receives an http error 502.
The order is correctly processed by SAP but the web applications is built in a way that the user is getting a confirmation page when the order is processed: this page is never presented to the user due to the 502 error.

FREB gave us further details about the error: 502.3, 2147954417 which translates into ERROR_INTERNET_OPERATION_CANCELLED.

This lead me to this post which describes a different error but based on the same principle and the solution still applies to our scenario: we increased the proxy timeout to 5 minutes to accommodate all reasonably long running requests and the problem was solved.

appcmd.exe set config -section:system.webServer/proxy /timeout:"00:05:00" /commit:apphost

Carlo

It's not the size of the dog in the fight, it's the size of the fight in the dog.

Mark Twain

Moving to Windows 2008: smartcard slow performance?

2010-11-15T11:00:00Z

Suppose you have a web application which authenticates clients through a smartcard certificate and everything is working fine, then you decide to move your web application to Windows Server 2008 (which is a good decision in many ways ) but your users are suddenly complaining about slow performance, pages are loading slower than they used to do and you cannot find any explanation on the network infrastructure, everything seems to be properly configured on the client and on the server… what is happening?

Well, as you can imagine this is what happened to a customer I worked with recently. A closer look at the client showed that every time a new object in the page was loaded (images, script files, css…) the smartcard was checked for the certificate, resulting in a major performance loss.

To make the story short, the behavior is controlled by client certificate negotiation, specifically by SSLAlwaysNegoClientCert which in IIS 7 has been moved to http.sys (see http://learn.iis.net/page.aspx/110/changes-between-iis-60-and-iis-7-security/); you can configure the <access> element as described or using netsh with a command such as the following:

netsh http add sslcert ipport=0.0.0.0:$port certstorename=MY certhash=$Certhash appid=$Appnr sslctlidentifier=$CTLlijst sslctlstorename=CA clientcertnegotiation=enable

Carlo

Quote of the day:
Human beings are perhaps never more frightening than when they are convinced beyond doubt that they are right. - Laurens Van der Post

Automatic postback does not fire for TextBox control

2010-10-29T16:40:00Z

I recently come across an interesting request which read like this:

A simple webapplication with one Textbox and one label. The Textbox has Autopostback=”true” and has an event listener attached to the TextChanged event. In the TextChanged event the label’s text is set to the text in the textbox. A javascript function “formats” the value of the textbox on the keyUp event. The ”problem” The postback is not fired for a textbox if a javascript function sets the value of the textbox in the keyDown or keyUp event. This only occurs in IE (IE 8, IE 7, FF 3.6.8, Opera 10.6 and Chrome 5 has been tested). No javascript errors or any kind of error message, the postback is just not being fired.

And here’s a sample page to reproduce the problem:

<%@ Page Language="vb" AutoEventWireup="false" ValidateRequest="false" EnableEventValidation="false" EnableViewStateMac="false" %>

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head runat="server">
       <title></title>

       <script type="text/javascript" language="javascript">
        function FormatTextBox(obj) {
            //This is just a dummy function that adds $ as the first character.
            var value = obj.value

            if (value.length > 0 && value.charAt(0) != '$')
                value = '$' + value;

            obj.value = value; // This line causes the autopostback not to fire. This only happens if the value of the input 
                               // is set in the keyup or keydown event and only in IE.
        }
       </script>

       <script language="vbscript" runat="server">
             'Just to have something happening during the autopostback.
             Private Sub TextBox1_TextChanged(ByVal sender As Object, ByVal e As System.EventArgs) Handles TextBox1.TextChanged
                    Label1.Text = TextBox1.Text
             End Sub
       </script>

</head>
<body>
       <form id="form1" runat="server">
       <div>
             <asp:TextBox ID="TextBox1" runat="server" onkeyup="return FormatTextBox(this);" AutoPostBack="true"></asp:TextBox>
             <asp:Label ID="Label1" runat="server" Text="Label"></asp:Label>
       </div>
       </form>
</body>
</html>

Type something in the textbox then hit TAB (or click somewhere on the white page) and you’ll see nothing happens, while you should have a postback and the Label should reflect the text you typed (plus the “$” sign added by the FormatTextBox function). Interestingly enough if you type only one character in the TextBox you’ll have the problem, if you type in two or more characters then everything will work. Another interesting effect: if you type the dollar sign (“$”) which happens to be the sign added by javascript, you’ll never have an automatic postback no matter how many “$” you’ll type in.

Why is it happening? Well, any time you type a new character into the TextBox, IE compares the new value to some internal 'initial' value that it it keeping to see if the text has changed. However, when you set the value of the TextBox programmatically, IE makes the assumption that you know what just happened and that it doesn't have to fire the onChange event. So it resets its internal tracking value to be the same as the value you just programmatically set. Then when you tab out of focus, the two values are the same and no onChange event gets fired.

So, how to deal with it? The easiest solution is to use the OnBlur event or,

if you really need to hook on the onkeyup event, is to track the state of the TextBox on your own and force the onchange event to fire, even if you programatically set the TextBox value. Use a TextBox like this:

<asp:TextBox ID="TextBox1" runat="server" onkeyup="FormatTextBoxDirty(this)" 
            onblur="if (this.dirty){this.onchange();}" AutoPostBack="true" />

and javascript like this:

function FormatTextBoxDirty(obj) {
        obj.dirty = false;  // This prevents firing onchange twice in the event that we don't modify the value
        if (obj.value.length > 0 && obj.value.charAt(0) != '$') {
            obj.value = '$' + obj.value;
            obj.dirty = true;
        }
    }

Here’s the working page:

<%@ Page Language="vb" AutoEventWireup="false" ValidateRequest="false" EnableEventValidation="false" EnableViewStateMac="false" %>

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head id="Head1" runat="server">
    <title></title>
    <script type="text/javascript" language="javascript">
        function FormatTextBox(obj) {
            //This is just a dummy function that adds $ as the first character.
            var value = obj.value

            if (value.length > 0 && value.charAt(0) != '$')
                value = '$' + value;

            obj.value = value; // This line causes the autopostback not to fire. This only happens if the value of the input 
                               // is set in the keyup or keydown event and only in IE.
        }

        function FormatTextBoxDirty(obj) {
            obj.dirty = false;  // This prevents firing onchange twice in the event that we don't modify the value
            if (obj.value.length > 0 && obj.value.charAt(0) != '$') {
                obj.value = '$' + obj.value;
                obj.dirty = true;
            }
        }

    </script>
    <script language="vbscript" runat="server">
        'Just to have something happening during the autopostback.
        Private Sub TextBox1_TextChanged(ByVal sender As Object, ByVal e As System.EventArgs) Handles TextBox1.TextChanged
            Label1.Text = TextBox1.Text
        End Sub
    </script>
</head>
<body>
    <form id="form1" runat="server">
    <div>
        <asp:TextBox ID="TextBox1" runat="server" onkeyup="FormatTextBoxDirty(this)" onblur="if (this.dirty){this.onchange();}"
            AutoPostBack="true" />
        <asp:Label ID="Label1" runat="server" Text="Label"></asp:Label>
    </div>
    </form>
</body>
</html>

P.s.
Thanks to Steve Molloy for this help on this matter.

Carlo

Quote of the day:
Minds are like parachutes: they only work when they are open – Anonymous

Sometimes they come back

2010-07-27T19:58:00Z

Stephen King fans like me for sure know Night Shift, a collection of 19 short stories I first read almost (oh, my…) almost 20 years ago. In Italian the title of this collection has been translated as “A volte ritornano” which literally means Sometimes they come back. This is exactly what I thought yesterday when I got yet another version of the “503 Service Unavailable” error: after I have already written about this error here and here, I’m writing the third post about it. I’m wondering if I’ll end up with a 19 post collection as Night Shift has…

Anyway, here’s the new story: the application pools of this web server were failing with the following error

(eventid 281) The worker process for application pool 'MSExchangeOWAAppPool' encountered an error 'Unrecognized attribute 'precondition' trying to read global module configuration data from file '\\?\C:\inetpub\temp\apppools\MSExchangeOWAAppPool.config', line number '281'. Worker process startup aborted.

As you can see the precondition keyword is mistyped (it should be preCondition, capital “C”) and if we changed the .config file mentioned in the error message we were able to start our application pool but randomly (most of the time after a reboot, though) the problem came back. Interesting to note is the path of the .config file mentioned in the error: C:\inetpub\temp\AppPools\xxxxxx.config

This is where Configuration Isolation stores the application pool specific configuration files:

IIS worker processes do not have Read access to applicationHost.config. How, then, are they able to read any of the configuration set in this file?

The answer lies in the configuration isolation feature provided by IIS 7.0, which is always on by default. Instead of enabling IIS worker processes to read applicationHost.config directly when reading the configuration file hierarchy, IIS generates filtered copies of this file and uses these copies as a replacement of applicationHost.config when configuration is read inside the IIS worker process.

The reason for doing this is to prevent IIS worker processes from application pool A to be able to read configuration information in applicationHost.config that is intended for application pool B. Because applicationHost.config may contain sensitive information, such as the user name and password for custom application pool identities, as well as user name and password for virtual directories, allowing all application pools to access applicationHost.config would break application pool isolation.

WAS is responsible for generating the temporary application pool configuration files that each IIS worker process uses as a replacement of applicationHost.config. These files are placed by default in the %SystemDrive%\Inetpub\Temp\Apppools directory and are named AppPoolName.config. As mentioned earlier, these files are configured to allow access only to the IIS worker processes in the corresponding application pool, by using the IIS APPPOOL\AppPoolName Application Pool SID.

Note This process occurs automatically each time applicationHost.config is changed and therefore does not require any manual action from the administrator outside of normal configuration procedures.

Each application pool configuration file contains the configuration in applicationHost.config, with the following information removed:

All application pool definitions in the system.applicationHost/applicationPools configuration section. Only WAS is required to read this configuration section.

Any Web site definitions in the system.applicationHost/sites configuration section for sites that do not have applications in the current application pool.

Any configuration in location tags for specific Web sites, applications, or URLs that do not reside inside the applications in the current application pool.

Caution All application definitions (and their virtual directory definitions, possibly containing user name and password credentials) for any site that has at least one application in the current application pool will be present in the application pool configuration file. To disable this behavior and include only the application definitions for applications in the application pool, set the IsolationWholeSiteInclude DWORD value to 0 in the HKLM\System\CurrentControlSet\Services\WAS\Parameters key and perform an IISRESET. This may break applications in sites with applications in multiple application pools when they attempt to map physical paths for URLs in other applications.

Keep in mind that global configuration settings set in the applicationHost.config (without using location tags to apply them to specific Web sites, applications, or URLs) are not filtered. Each application pool configuration file will contain all of these settings.

Configuration isolation is a key part of the application pool isolation strategy in IIS 7.0. It is enabled by default to provide configuration isolation for server-level configuration in applicationHost.config. For strategies on achieving proper application pool isolation, see the section titled "Isolating Applications" earlier in this chapter.

Caution Configuration stored in .NET Framework machine.config and root web.config files is not isolated. Only configuration stored in applicationHost.config is isolated.

So, this means the mistyped entry very likely originates from applicationHost.config… We do not know what (or who ) changed this since I have the same roles and features installed on my machine as the customer had on his server, but after fixing the typo in the centralized configuration file the problem disappeared.

Carlo

Quote of the day:
To avoid situations in which you might make mistakes may be the biggest mistake of all. - Peter McWilliams

My GridView does not want to paginate anymore!

2010-07-23T14:15:35Z

Lots of stuff kept me busy over the last few months and this blog has been a bit too quiet… let me try to revamp it a bit (new graphic layout will come later).

I recently worked on an interesting case where a simple GridView with pagination enabled was not working as expected: every time we clicked a different page (note, page referred to the GridView pages, not .aspx pages…) we were quickly brought back to the first page: no errors logged anywhere (IIS logs, Fiddler, event viewer… nothing really useful there). Then we decided to make a quick test: we bypassed the ISA proxy and everything started working fine, no way to reproduce the problem again; needless to say, as soon as we reconfigured the proxy in IE the problem came back.

My colleague Alvar works also on ISA cases and suggested the article POST requests that do not have a POST body may be sent to a Web server that is published in ISA Server 2006 and this actually did the trick, problem solved! Thanks Alvar

Carlo

Quote of the day:
Great people talk about ideas, average people talk about things, and small people talk about wine. - Fran Lebowitz

Make your own keyboard for free

2010-04-28T20:23:40Z

I’ve been using the new Arc Keyboard for a few weeks now and I’m very pleased with it, I like it especially for typing. But (there is always a “but” )… Unfortunately I have to say this keyboard is not very practical when working with Windbg or Visual Studio (at least with the Italian layout) where you have to type special characters such as tilde (~) or braches ( { } ). Well, it is possible to get braches with the key combination ATL GR+SHIFT+è for the open one and ALT GR+SHIFT++ or ALT+123 and ALT+125 on the numpad if available and despite the weird key combination I got used to press it with only one hand and my typing flow was acceptable, even on a laptop where the numpad cannot be used. Anyway I’ve not been able to find how to get the tilde sign on an Italian keyboard without using the numpad combination (ALT+126), this means I cannot use neither my laptop nor the Arc Keyboard for coding… Should I use two different keyboards, then? No way

Bing came to the rescue and pointed me to the Keyboard Layout Creator that, among other things, allows to modify and extend an existing keyboard layout adding new keys and key combinations, and it’s easy to use, too .

So I started loading the Italian keyboard layout and decided to use the ALT GR key as my function key for my new key combinations: it’s easy to check which keys are already assigned and which are free:

Now just double-click a key on the keyboard to get this dialog:

Now just add the new key in the box with the corresponding control key combination (I chose CTRL+ALT+<key>) to assign it; after some quick tests I came up with the following layout which I’m using on all my machines now:

From the Project menu is possible to validate and test the new keyboard layout and also the Build DLL and Setup Package that creates a MSI setup package to install the new keyboard. Once installed, the new keyboard can be enabled from the Text Services and Input Languages dialog (click on Change keyboard on Region and Language dialog, Keyboarda and Languages tab), choose your custom keyboard layout as the Default input language:

Hope you’ll enjoy your new keyboard layout as I’m doing with mine .

Carlo

Quote of the day: Read, every day, something no one else is reading. Think, every day, something no one else is thinking. Do, every day, something no one else would be silly enough to do. It is bad for the mind to be always part of unanimity. - Christopher Morley

IDEVDataCollector source code available for download

2010-03-13T22:58:23Z

Download it from the IDEVDataCollector page .

Don’t expect too sophisticated coding techniques (at least it gets the job done), let me know what you think, I’m open for suggestions and feature requests .

Carlo

Quote of the day: ...when you have eliminated the impossible, whatever remains, however improbable, must be the truth. - Sir Arthur Conan Doyle

IDEVDataCollector adds IIS6 support

2010-03-09T15:04:16Z

It took me some time (mostly because of other demanding tasks and commitments I have for my daily job) but finally I have just updated IDEVDataCollector to support IIS 6, it’s available at the tool main page: http://blogs.msdn.com/carloc/pages/idevdatacollector.aspx.

Give it a try and let me know what you think

P.s.
Yes, I’m planning to release the source code too, stay tuned!

Carlo

Quote of the day:
Clothes make the man. Naked people have little or no influence on society. - Mark Twain

Which files to collect? Here’s a tool for you

2010-02-01T16:07:27Z

Over the last couple of months among other things I’ve been working on a tool to automatically the logs and information in my team we usually ask at the beginning of a new Support Call: all standard things such as IIS logs, event logs, .config files etc… Not all of those files are necessary for every case (as usual it depends on the problem at hand), but experience tells me the more information we have about the environment, the more chances we have to spot small inconsistencies, configuration problems etc… that sometimes can lay unnoticed and interfere with our work.

So, enter our guest:

IDevDataCollector

I tried to keep the UI as simple and organized as possible, utilization is very easy, just run it and click the “Collect” button, sit there a couple of minutes (how much exactly will depend on the logs collected) and you’re done . Please note that I’m still working on it, this is just a preview release but I’d love to have your feedback/bug reports/feature requests etc…, keeping in mind that the scope of this tool is to facilitate troubleshooting and data collection for IIS/ASP.NET related problems, I am not interested (and probably do not have the time) to build an omnicomprehensive tool to replace everything else we have today .

So, here is the list of what it collects so far:

IIS
- IIS logs
- IIS configuration (content of C:\Windows\System32\Inetsrv\config for IIS 7, metabase.xml for IIS 6)
- HttpErr logs
- Freb logs (IIS 7 only)
ASP.NET
- .config files and .browser files under .NET Framework installation folder(s): all versions available on the machine
- .config files for the sites/vdirs configured in IIS (the list of sites and applications is dynamically read from the config store)
- ASP.NET setup logs
- ACLs for the sites and applications defined in IIS (the list of sites and applications is dynamically read from the config store)
- Result from browsing get_aspx_version.aspx (the page does not exist, this is made on purpose to get the 404 error page from IIS)
Event Logs
- Application
- Security (not enabled by default)
- System
Environment
- System information (uses msinfo32.exe, might take a few minutes to complete)

Important: it currently works only with IIS 7 (hence on Vista/Windows 7/Windows 2008 and Windows 2008 R2).

Already on the “to do” list:

Enable Windows 2003 (IIS6) support
Support IIS7 shared configuration
Collect local policy information for the worker process account
Collect custom event logs (in case one is used to store custom application messages)
Improve performances and enhance application logging
Implement a log report (such a summary HTML page or similar) for easy access and review to the collected information
Automatically compress the collected logs for an easier transfer
Suggestions?

The dedicated page is http://blogs.msdn.com/carloc/pages/idevdatacollector.aspx, let me know what if think!

Carlo

Quote of the day:
In archaeology you uncover the unknown. In diplomacy you cover the known. - Thomas Pickering

Internet Explorer wants to install Silverlight over and over and over again…

2010-01-19T15:33:00Z

I just realized I have not posted anything here over the last couple of months which is definitely not a good thing to keep a blog alive… It’s been a busy period, I’ve been working on the Italian Developer Support Blog with my colleague Raffaele, of course working with our customers on Support Cases, working on a few internal organizational changes (still in progress) and I’ve also been working on a tool to automatically collect logs and information about IIS, ASP.NET and the server itself we’ll (hopefully ) use in our everyday work when troubleshooting customer’s problems.

One of the toughest problems I worked on in this period was actually on my machine… for some reason I could no longer use Silverlight on my office desktop, every time Internet Explorer proposed to install the component but either the setup failed with an error indicating that Silverlight was already installed on my machine (which it really was!) or some other error that inevitably lead to an aborted setup. This seemed to be a machine-wide setting (not just my account) because I got the same problem no matter which account I used to log-on to my machine.

If you quickly need to get your Silverlight working you can do as I did: use the “Reset Internet Explorer Settings” button in the “Advanced” tab to restore some IE settings to their default RTM state; this will not delete your personal data (cookies, forms, InPrivate filters, accelerators etc…) unless you explicitly flag the “Delete personal settings” checkbox:

Anyway I am curios and I managed to collect some data before and after the reset to compare and get the differences and identify which setting is causing the problem (of course I’ll post it when ready), I just wanted to save you and headache in case you have this same issue.

Carlo

Quote of the day:
In this world nothing can be said to be certain, except death and taxes. - Benjamin Franklin

Missing ASP.NET Tab (this time on Windows 2000)

2009-10-14T14:00:27Z

I already wrote a post on this matter some time ago, but this week worked on the same problem on a Windows 2000 Server, of course 32 bit.

Running aspnet_regiis -i did not help so we tried to run the script Tom provides in his post, but we got the error here below:

Line 41 is “Set objIIS = GetObject("IIS://" & strComputer & "/W3SVC/AppPools")”, this path does not exist in IIS 5 metabase, hence the error. To adapt it to IIS 5 we removed the script section that checks for Enable32BitAppOnWin64.

We run the script again and got this message:

PASS: The current value of Restrict_Run is: 0
PASS: Successfully registered mmcaspext.dll
PASS: Successfully registered AspNetMMC
Microsoft (R) .NET Framework Assembly Registration Utility 2.0.50727.1433
Copyright (C) Microsoft Corporation 1998-2004. All rights reserved.
Types registered successfully
Assembly exported to 'C:\WINNT\Microsoft.NET\Framework\v2.0.50727\AspNetMMCExt.tlb', and the type library was registered successfully
PASS: Successfully registered Assembly
Running aspmantst.exe to test the ASP.NET Tab snap-in
Attempting to create the object as CLSCTX_LOCAL_SERVER.
Create failed. hr: 80070005. QIhr: 80070005. GLE: 1008
Attempting to create the object as CLSCTX_REMOTE_SERVER to <machine_name>.
Create failed. hr: 80070005. QIhr: 80070005. GLE: 1008
Attempting to create the object as CLSCTX_INPROC_SERVER.
Object created. Attempting to call method.
Method succeded. Result: 1.1.4322.0,C:\WINNT\Microsoft.NET\Framework\v1.1.4322\aspnet_isapi.dll,2.0.50727.0,C:\WINNT\Microsoft.NET\Framework\v2.0.50727\aspnet_isapi.dll

Code 80070005 means “Access Denied”, so we do not have enough permission to complete the configuration… The object we are trying to create, which represents the ASP.NET Ta in IIS Manager, is listed in dcomcnfg as Microsoft.Aspnet.Snapin.AspNetManagementUtility.2. In a default Windows 2000 SP4 installation, the Default Access Permission ACL is empty: when you log in as an administrator, if the ACL list is empty then the owner changes to whatever account you are logging on with: this account is the owner and implicitly has access to the COM instantiation works fine. But if the ACL list is not empty (it contains any account or group) the owner does not change at logon and the automatic mechanism just described does not work.

Likely the same missing permission we were already facing, Microsoft.Aspnet.Snapin.AspNetManagementUtility.2 did not show up in our application list… so we add a look at the same kind of permission, but this time we checked the “Default Security” tab:

And here we were! On another working machine this customer had the ACL list was empty, while for some reason on the failing machine the list was not empty, we had an account listed there… in such a situation there are two options:

Clear the ACL list
Add the Administrators group and any other account you need to explicitly grand Access Permissions to

In our case we cleared the list, run the script again (this time every step completed successfully) and the customer got his ASP.NET Tab available in IIS Manager

Carlo

Quote of the day:
If you develop an ear for sounds that are musical it is like developing an ego. You begin to refuse sounds that are not musical and that way cut yourself off from a good deal of experience. - John Cage

Input validation and another StackOverflow

2009-10-06T14:55:51Z

Here’s another interesting crash problem I worked on a few days ago: the application pool was randomly crashing and the following errors were logged:

Event Type:      Error
Event Source:   .NET Runtime 2.0 Error Reporting
Event Category: None
Event ID:          1000
Date:                21/08/2009
Time:                12:23:12
User:                N/A
Computer:        <computername>
Description: Faulting application w3wp.exe, version 6.0.3790.3959, stamp 45d691cc, faulting module kernel32.dll, version 5.2.3790.4062, stamp 462643a7, debug? 0, fault address 0x0000000000027d8d.

Event Type:      Warning
Event Source:   W3SVC
Event Category:           None
Event ID:          1011
Date:                21/08/2009
Time:                12:23:14
User:                N/A
Computer:        <computername>
Description: A process serving application pool <AppPoolName> suffered a fatal communication error with the World Wide Web Publishing Service. The process id was xxx. The data field contains the error number.

Even if with some difficulties (we had to disable Error Reporting as described here) we managed to capture a coupe of dumps for this crash; dumping the exception was not very helpful:

0:027> !gle
LastErrorValue: (Win32) 0x3f0 (1008) - An attempt was made to reference a token that does not exist.
LastStatusValue: (NTSTATUS) 0xc000007c - An attempt was made to reference a token that doesn't exist. This is typically done by referencing the token associated with a thread when the thread is not impersonating a client.

0:027> !printexception
There is no current managed exception on this thread

But dumping the very long stack gave some good hint at least about a potential problem (a stack overflow):

0:027> kpL
Child-SP          RetAddr           Call Site
00000000`05fcae90 00000642`7f895333 kernel32!RaiseException+0x73
00000000`05fcaf60 00000642`78390730 mscorwks!`string'+0x81ef3
00000000`05fcb130 00000642`780a9522 mscorlib_ni!System.OrdinalComparer.Equals(<HRESULT 0x80004001>)+0x70
00000000`05fcb170 00000642`7809822a mscorlib_ni!System.Collections.Hashtable.KeyEquals(<HRESULT 0x80004001>)+0x42
00000000`05fcb1a0 00000642`748f8976 mscorlib_ni!System.Collections.Hashtable.get_Item(<HRESULT 0x80004001>)+0x15a
00000000`05fcb260 00000642`748ff9bc System_ni!System.Collections.Specialized.NameObjectCollectionBase.FindEntry(<HRESULT 0x80004001>)+0x16
00000000`05fcb290 00000642`80580f48 System_ni!System.Collections.Specialized.NameValueCollection.Get(<HRESULT 0x80004001>)+0xc
00000000`05fcb2c0 00000642`8058108d MyTest!MyClass.Web.Pages.BasePage.get_Language(<HRESULT 0x80004001>)+0x48
00000000`05fcb300 00000642`8058108d MyTest!MyClass.Web.Pages.BasePage.get_Language(<HRESULT 0x80004001>)+0x18d
00000000`05fcb340 00000642`8058108d MyTest!MyClass.Web.Pages.BasePage.get_Language(<HRESULT 0x80004001>)+0x18d
00000000`05fcb380 00000642`8058108d MyTest!MyClass.Web.Pages.BasePage.get_Language(<HRESULT 0x80004001>)+0x18d
00000000`05fcb3c0 00000642`8058108d MyTest!MyClass.Web.Pages.BasePage.get_Language(<HRESULT 0x80004001>)+0x18d
00000000`05fcb400 00000642`8058108d MyTest!MyClass.Web.Pages.BasePage.get_Language(<HRESULT 0x80004001>)+0x18d
00000000`05fcb440 00000642`8058108d MyTest!MyClass.Web.Pages.BasePage.get_Language(<HRESULT 0x80004001>)+0x18d
00000000`05fcb480 00000642`8058108d MyTest!MyClass.Web.Pages.BasePage.get_Language(<HRESULT 0x80004001>)+0x18d
00000000`05fcb4c0 00000642`8058108d MyTest!MyClass.Web.Pages.BasePage.get_Language(<HRESULT 0x80004001>)+0x18d
00000000`05fcb500 00000642`8058108d MyTest!MyClass.Web.Pages.BasePage.get_Language(<HRESULT 0x80004001>)+0x18d
00000000`05fcb540 00000642`8058108d MyTest!MyClass.Web.Pages.BasePage.get_Language(<HRESULT 0x80004001>)+0x18d
00000000`05fcb580 00000642`8058108d MyTest!MyClass.Web.Pages.BasePage.get_Language(<HRESULT 0x80004001>)+0x18d
00000000`05fcb5c0 00000642`8058108d MyTest!MyClass.Web.Pages.BasePage.get_Language(<HRESULT 0x80004001>)+0x18d
00000000`05fcb600 00000642`8058108d MyTest!MyClass.Web.Pages.BasePage.get_Language(<HRESULT 0x80004001>)+0x18d
00000000`05fcb640 00000642`8058108d MyTest!MyClass.Web.Pages.BasePage.get_Language(<HRESULT 0x80004001>)+0x18d
00000000`05fcb680 00000642`8058108d MyTest!MyClass.Web.Pages.BasePage.get_Language(<HRESULT 0x80004001>)+0x18d
00000000`05fcb6c0 00000642`8058108d MyTest!MyClass.Web.Pages.BasePage.get_Language(<HRESULT 0x80004001>)+0x18d
00000000`05fcb700 00000642`8058108d MyTest!MyClass.Web.Pages.BasePage.get_Language(<HRESULT 0x80004001>)+0x18d
00000000`05fcb740 00000642`8058108d MyTest!MyClass.Web.Pages.BasePage.get_Language(<HRESULT 0x80004001>)+0x18d
00000000`05fcb780 00000642`8058108d MyTest!MyClass.Web.Pages.BasePage.get_Language(<HRESULT 0x80004001>)+0x18d
00000000`05fcb7c0 00000642`8058108d MyTest!MyClass.Web.Pages.BasePage.get_Language(<HRESULT 0x80004001>)+0x18d
00000000`05fcb800 00000642`8058108d MyTest!MyClass.Web.Pages.BasePage.get_Language(<HRESULT 0x80004001>)+0x18d
00000000`05fcb840 00000642`8058108d MyTest!MyClass.Web.Pages.BasePage.get_Language(<HRESULT 0x80004001>)+0x18d
00000000`05fcb880 00000642`8058108d MyTest!MyClass.Web.Pages.BasePage.get_Language(<HRESULT 0x80004001>)+0x18d
00000000`05fcb8c0 00000642`8058108d MyTest!MyClass.Web.Pages.BasePage.get_Language(<HRESULT 0x80004001>)+0x18d
[...]

Using the err.exe utility to lookup the exit code of the process when it crashed, we can confirm we had a stack overflow:

Event Type: Warning
Event Source: W3SVC
Event Category: None
Event ID: 1009
Date: 21/08/2009
Time: 13:44:11
User: N/A
Computer: <computername>
Description: A process serving application pool <AppPoolName> terminated unexpectedly.
The process id was '5392'. The process exit code was '0x800703e9'.

C:\>err 0x800703e9
# for hex 0x800703e9 / decimal -2147023895
COR_E_STACKOVERFLOW corerror.h
# Is raised by the EE when the execution stack overflows as
# it is attempting to ex
# as an HRESULT: Severity: FAILURE (1), FACILITY_WIN32 (0x7), Code 0x3e9
# for hex 0x3e9 / decimal 1001
ERROR_STACK_OVERFLOW winerror.h
# Recursion too deep; the stack overflowed.
# 2 matches found for "0x800703e9"

In the IIS log the request that lead to the crash was like this:

GET	/MyApp/test.aspx?currentPreferences=2&ParentId=1&ClientDateTime=Wed%20Aug%2019%202009%2009:09:57%20GMT+0100%20(GMT%20Standard%20Time)&TimeZone=-60&Language=undefined&Currency=undefined&OrgUnitId=undefined

Basically some user was using a robot to scrape the web site and the Language parameter (along with a few others) was not correctly set; in this case the customer’s code was not properly validating those parameters (using a browser they are always valid) and this cause the endless recursion that ultimately exhausted the stack.

Needless to say, improving the parameter validation logic resolved the problem

Carlo

Quote of the day:
Seeing ourselves as others see us would probably confirm our worst suspicions about them. - Franklin P. Adams

New opening: Italian Dev Support blog

2009-09-11T07:20:32Z

Yesterday my Italian colleagues in Developer Support and I officially inaugurated our new team blog: if you can read Italian then join us at http://blogs.msdn.com/itasupport

Carlo

Quote of the day:
Everything that irritates us about others can lead us to an understanding of ourselves. - Carl Jung

Hosting on a UNC share is not supported for the Windows XP Platform

2009-09-06T12:43:16Z

If you have a complex solution, with a lot of folders and subfolders, and you are hosting and debugging it on a remote IIS server (not on your local machine), you may get an error similar to the following:

An error occurred loading a configuration file: Failed to start monitoring changes to '<path>' because the network BIOS command limit has been reached. For more information on this error, please refer to Microsoft knowledge base article 810886. Hosting on a UNC share is not supported for the Windows XP Platform

The article mentioned in the error message (810886) suggests to change the MaxCmds and MaxMpxCt values, but when I had the chance to work on this problem this solution does not always work (anyway I suggest you to give it a try before discarding it).

You may find another message in your event log, like the following:

Event Type: Warning
Event Source: Srv
Event Category: None
Event ID: 2021
Date:
Time:
User: N/A
Computer:
Description: The server was unable to allocate a work item x times in the last 60 seconds

If you do, give the article 317249 a try (increase the MaxWorkItems value).

The solution I find most useful is the one described in 911272: if your .NET build is at least 2.0.50727.62 you already have the fix mentioned in the article but you may need to activate it:

Registry information
To enable this hotfix, you must add the following DWORD value at the following registry key:
HKLM\Software\Microsoft\ASP.NET\FCNMode
The following table lists possible values for the FCNMode DWORD value and the behavior that is associated with each value.

Value Behavior

Does not exist This is the default behavior. For each subdirectory, the application will create an object that will monitor the subdirectory

0 or greater than 2 This is the default behavior. For each subdirectory, the application will create an object that will monitor the subdirectory

1 The application will disable File Change Notifications (FCNs)

2 The application will create one object to monitor the main directory. The application will use this object to monitor each subdirectory

I suggest to set FCNMode to 2, so you’ll still have File Change Notifications in place.

Carlo

Quote of the day:
The greatest challenge to any thinker is stating the problem in a way that will allow a solution. - Bertrand Russell

An authentication error occurred while communicating with the web server

2009-08-30T20:25:20Z

I’m back after the summer break (3 weeks spent trying to master the “Do It Yourself” art at home ) and my second day in office I got an interesting problem with remote debugging: after installing the Service Pack 2 for .NET 2.0, when trying to debug our web application (hosted on a remote IIS) we were getting the following error: “Unable to start debugging on the web server. An authentication error occurred while communicating with the web server”. Note, you can get the same error if you install the Service Pack 1 for .NET 3.5: the reason is simple, if you install the SP1 for .NET 3.5 you’ll automatically get the SP2 for .NET 2.0. Another point: you are using host headers in your site.

Anyway, the error is caused by a security change: in .NET 3.5 SP1 (and .NET 2.0 SP2) now defaults to specifying the host name used in the request URL in an SPN in the NTLM authentication package. The NTLM authentication process includes a challenge issued by the destination machine and sent back to the client machine. When Windows receives a challenge it generated itself, authentication will fail unless the connection is a loop back connection. When a Web Site is configured with a host header, the host name is neither the machine name nor the loop back IP address nor the machine's IP address, so the authentication request will fail.

There are two possible solutions:

The recommended approach is to map your host header name to the loop back address in the registry. Here are the steps:
- Click Start > Run, type regedit and click OK
- In Registry Editor, select the registry key “HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa\MSV1_0”
- Right-click MSV1_0 > New > Multi-String Value
- Type “BackConnectionHostNames” and press ENTER
- Right-click “BackConnectionHostNames” > Modify
- In the Value data box, type the host name or the host names for the sites that are on the local computer, then click OK
- Quit the Registry Editor and reset IIS
A less secure work around is to disable the loop back check, as described in http://support.microsoft.com/kb/896861

If you want more information on this matter you have have a look at http://blogs.msdn.com/lukaszp/archive/2008/07/18/reporting-services-http-401-unauthorized-host-headers-require-your-attention.aspx.

Carlo

Quote of the day:
If fifty million people say a foolish thing, it is still a foolish thing. - Anatole France

Security advisory bulletin 973882 (July 28, 2009)

2009-07-29T09:40:04Z

My colleague Feliciano has just made me aware of an extraordinary security bulletin we have just released for a new vulnerability discovered in ATL libraries which could allow remote code execution; here are the information I am aware of so far:

Microsoft Security Advisory (973882) English
Rilascio straordinario di sicurezza Microsoft del 28 luglio 2009: prime indicazioni e raccomandazioni (Feliciano’s post, if you can read Italian)

The updates to install:

MS09-034
MS09-035 (for developers, update for Visual Studio)

Carlo

Quote of the day:
Those who dream by day are cognizant of many things which escape those who dream only by night. - Edgar Allan Poe

ASP.NET 1.1 not configurable in IIS Manager?

2009-07-28T13:26:44Z

If you search the Internet you’ll likely find a good number of articles and blog posts about how to configure ASP.NET 1.1 on IIS 7, here are a couple of good examples:

Something I have not found written elsewhere is why it is not possible configure ASP.NET properties from the IIS Manager interface? Even if you correctly configure the application pool (32 bit, Classic pipeline), use the workaround to avoid the error on applicationHost.config, add the IgnoreSectionHandler to avoid ASP.NET 1.1 runtime exceptions due to the not recognized IIS 7 configuration tags and finally the ASP.NET 1.1 pages works fine and are served correctly, you’ll not have the “ASP.NET” icon group in IIS Manager:

As you can imagine, if you move the application to an application pool where you are running ASP.NET 2.0, the icon group appears where you expect it to be:

Well, it turns out that ASP.NET 1.1 configuration is not supported by InetMgr.exe (the IIS Manager), you have to use appcmd.exe using /commit:machine (for machine.config) or /commit:webroot (for the root web.config) to check if the changes work and they end up where they are supposed to. Note, there are some limitations because of the lack of a complete schema; if you feel brave enough you can create your own custom schema, I do not think (this is my opinion, I do not have an official work on this) that Microsoft will provide such schema. Or if you are even braver you can edit the .config files directly in Notepad, but this sounds like business for real geeks… how much geek do you feel?

Carlo

Quote of the day:
The belief in a supernatural source of evil is not necessary; men alone are quite capable of every wickedness. - Joseph Conrad

Thread exit may kill your Session

2009-07-24T14:38:58Z

Every now and then we got a call for an application which, randomly and without a specific pattern or apparent reason, shuts down all user’s sessions. Since ASP.NET 2.0 we can use Web Events to have more information on what is happening at the runtime level and this time the message we had in the event log was pretty clear:

Event code: 1002
Event message: Application is shutting down. Reason: Configuration changed.
Event time:
Event time (UTC):
Event ID:
Event sequence:
Event occurrence: 1
Event detail code:

Application information:
Application domain: /LM/W3SVC/1045621189/Root-2-1742334186915428
Trust level: Full
Application Virtual Path: /
Application Path: C:\Inetpub\wwwroot\
Machine name: <machinename>

Process information:
Process ID: 1234
Process name: w3wp.exe
Account name: NT AUTHORITY\NETWORK SERVICE

We used Process Monitor to track access to the relevant .config files which are usually the cause for this kind of problem, but with no luck.

To make a long story short, we had a look at the events around the time when the session was lost and we noticed it usually followed some exception tracking logic the customer had used, so we had a look at the source code and found something interesting: the customer had some code to write and maintain a custom log on a text file and this work was done on a background thread. This code was using the ConfigurationManager class to read some settings from the web.config.

Now the interesting part: the background thread was executed in a different AppDomain than the ASP.NET application and it needed to initialize this new AppDomain to use the ConfigurationManager; as part of this initialization we create a new File Change Notification object to monitor machine.config and web.config files for changes. When the background thread exited after his job was done, the File Change Notification object encountered this error:

0x800703E3 ( -2147023901 ) = ERROR_OPERATION_ABORTED
The I/O operation has been aborted because of either a thread exit or an application request

Note, that is not an exception; but this change notification event was “head” by all ASP.NET applications which is the expected behavior, and all AppDomains were restarted: the end users were therefore redirected to the login page .

How do you avoid this situation? Well, we made a small change to the exception handling code to not use ConfigurationManager in the background thread.

Carlo

Quote of the day:
When a person can no longer laugh at himself, it is time for others to laugh at him. - Thomas Szasz

StackOverflow continued: DataSource or DataSourceID?

2009-07-23T16:49:26Z

I have already wrote here about a StackOverflowException I recently came across; after a while I got another quite similar problem, still a StackOverflowException with a very long recursion.

Thread 36 is the one where the exception was thrown:

0:000> !threads
ThreadCount: 21
UnstartedThread: 0
BackgroundThread: 21
PendingThread: 0
DeadThread: 0
Hosted Runtime: no
                                      PreEmptive   GC Alloc           Lock
       ID OSID ThreadOBJ    State     GC       Context       Domain   Count APT Exception
  17    1  950 000f2580   1808220 Enabled  00000000:00000000 000eed48     0 MTA (Threadpool Worker)
  27    2 1bfc 00100018      b220 Enabled  00000000:00000000 000eed48     0 MTA (Finalizer)
  28    3  300 00117068    80a220 Enabled  00000000:00000000 000eed48     0 MTA (Threadpool Completion Port)
  29    4 1cf4 0011a458      1220 Enabled  00000000:00000000 000eed48     0 Ukn
   9    5  ec0 001702e0       220 Enabled  00000000:00000000 000eed48     0 Ukn
  31    6 2414 00171f68   180b220 Enabled  00000000:00000000 000eed48     0 MTA (Threadpool Worker)
  32    7 1f24 00172338   180b220 Enabled  00000000:00000000 000eed48     0 MTA (Threadpool Worker)
  33    8  97c 001729e0   180b220 Enabled  00000000:00000000 000eed48     0 MTA (Threadpool Worker)
  34    9 13bc 00173088   180b220 Enabled  00000000:00000000 000eed48     0 MTA (Threadpool Worker)
  35    a 22f8 00173a50   180b220 Enabled  00000000:00000000 000eed48     0 MTA (Threadpool Worker)
  36    b 1e60 00174540   180b220 Disabled 12445a64:124475b8 0011ab48     1 MTA (Threadpool Worker) System.StackOverflowException (1a0a00a4)
  37    c 1c8c 00174cf8   180b220 Enabled  00000000:00000000 000eed48     0 MTA (Threadpool Worker)
  15    d 1934 0017d040   880a220 Enabled  00000000:00000000 000eed48     0 MTA (Threadpool Completion Port)
   7    e 2028 0017ece8       220 Enabled  00000000:00000000 000eed48     0 Ukn
   8    f  dbc 0016f280       220 Enabled  00000000:00000000 000eed48     0 Ukn
   4   10 2298 05007008       220 Enabled  00000000:00000000 000eed48     0 Ukn
   6   11 20c0 05004278       220 Enabled  00000000:00000000 000eed48     0 Ukn
   3   12  c9c 05003b70       220 Enabled  00000000:00000000 000eed48     0 Ukn
   5   13 1a84 04ff7a60       220 Enabled  00000000:00000000 000eed48     0 Ukn
   2   14 253c 0d378ee0       220 Enabled  00000000:00000000 000eed48     0 Ukn
  42   15 1760 0d38b3d8       220 Enabled  00000000:00000000 000eed48     0 Ukn

Again, here is the managed stack of the thread:

0:036> !clrstack
OS Thread Id: 0x1e60 (36)
ESP       EIP     
04e94acc 77e42004 [FaultingExceptionFrame: 04e94acc] 
04e95058 77e42004 [HelperMethodFrame: 04e95058] 
04e950c4 65230cc1 System.Data.RBTree`1+TreePage[[System.__Canon, mscorlib]]..ctor(Int32)
04e950d4 65230abe System.Data.RBTree`1[[System.__Canon, mscorlib]].AllocPage(Int32)
04e950f8 652309fc System.Data.RBTree`1[[System.__Canon, mscorlib]].InitTree()
04e95108 65230961 System.Data.DataRowCollection..ctor(System.Data.DataTable)
04e95118 652307c2 System.Data.DataTable..ctor()
04e95134 65230535 System.Data.Common.DataTableMapping.GetDataTableBySchemaAction(System.Data.DataSet, System.Data.MissingSchemaAction)
04e95150 6523037c System.Data.ProviderBase.SchemaMapping..ctor(System.Data.Common.DataAdapter, System.Data.DataSet, System.Data.DataTable, System.Data.ProviderBase.DataReaderContainer, Boolean, System.Data.SchemaType, System.String, Boolean, System.Data.DataColumn, System.Object)
04e9519c 6522fbac System.Data.Common.DataAdapter.FillMappingInternal(System.Data.DataSet, System.Data.DataTable, System.String, System.Data.ProviderBase.DataReaderContainer, Int32, System.Data.DataColumn, System.Object)
04e951d0 6522fc1e System.Data.Common.DataAdapter.FillMapping(System.Data.DataSet, System.Data.DataTable, System.String, System.Data.ProviderBase.DataReaderContainer, Int32, System.Data.DataColumn, System.Object)
04e95218 6522f9e6 System.Data.Common.DataAdapter.FillFromReader(System.Data.DataSet, System.Data.DataTable, System.String, System.Data.ProviderBase.DataReaderContainer, Int32, Int32, System.Data.DataColumn, System.Object)
04e95270 6522f942 System.Data.Common.DataAdapter.Fill(System.Data.DataSet, System.String, System.Data.IDataReader, Int32, Int32)
04e952b4 65230105 System.Data.Common.DbDataAdapter.FillInternal(System.Data.DataSet, System.Data.DataTable[], Int32, Int32, System.String, System.Data.IDbCommand, System.Data.CommandBehavior)
04e9530c 65230010 System.Data.Common.DbDataAdapter.Fill(System.Data.DataSet, Int32, Int32, System.String, System.Data.IDbCommand, System.Data.CommandBehavior)
04e95350 6559401d System.Data.Common.DbDataAdapter.Fill(System.Data.DataSet, System.String)
04e95384 6678d11f System.Web.UI.WebControls.SqlDataSourceView.ExecuteSelect(System.Web.UI.DataSourceSelectArguments)
04e95424 6668f498 System.Web.UI.DataSourceView.Select(System.Web.UI.DataSourceSelectArguments, System.Web.UI.DataSourceViewSelectCallback)
04e95434 66792c0f System.Web.UI.WebControls.DataBoundControl.PerformSelect()
04e9544c 6679285e System.Web.UI.WebControls.BaseDataBoundControl.DataBind()
04e95458 667dcf35 System.Web.UI.WebControls.GridView.DataBind()
04e9545c 660e1ac3 System.Web.UI.WebControls.BaseDataBoundControl.EnsureDataBound()
04e95488 668ecdcd System.Web.UI.WebControls.BaseDataBoundControl.set_RequiresDataBinding(Boolean)
04e95498 667da23b System.Web.UI.WebControls.GridView.set_PageIndex(Int32)
04e954a8 0ad3e096 ASP.search_aspx.Gridview1_DataBound(System.Object, System.EventArgs)
04e959a8 667928ce System.Web.UI.WebControls.BaseDataBoundControl.OnDataBound(System.EventArgs)
04e959bc 66792b45 System.Web.UI.WebControls.DataBoundControl.OnDataSourceViewSelectCallback(System.Collections.IEnumerable)
04e959cc 6668f4a4 System.Web.UI.DataSourceView.Select(System.Web.UI.DataSourceSelectArguments, System.Web.UI.DataSourceViewSelectCallback)
04e959dc 66792c0f System.Web.UI.WebControls.DataBoundControl.PerformSelect()
04e959f4 6679285e System.Web.UI.WebControls.BaseDataBoundControl.DataBind()
04e95a00 667dcf35 System.Web.UI.WebControls.GridView.DataBind()
04e95a04 660e1ac3 System.Web.UI.WebControls.BaseDataBoundControl.EnsureDataBound()
04e95a30 668ecdcd System.Web.UI.WebControls.BaseDataBoundControl.set_RequiresDataBinding(Boolean)
04e95a40 667da23b System.Web.UI.WebControls.GridView.set_PageIndex(Int32)
04e95a50 0ad3e096 ASP.search_aspx.Gridview1_DataBound(System.Object, System.EventArgs)
04e95f50 667928ce System.Web.UI.WebControls.BaseDataBoundControl.OnDataBound(System.EventArgs)
04e95f64 66792b45 System.Web.UI.WebControls.DataBoundControl.OnDataSourceViewSelectCallback(System.Collections.IEnumerable)
04e95f74 6668f4a4 System.Web.UI.DataSourceView.Select(System.Web.UI.DataSourceSelectArguments, System.Web.UI.DataSourceViewSelectCallback)
04e95f84 66792c0f System.Web.UI.WebControls.DataBoundControl.PerformSelect()
04e95f9c 6679285e System.Web.UI.WebControls.BaseDataBoundControl.DataBind()
04e95fa8 667dcf35 System.Web.UI.WebControls.GridView.DataBind()
04e95fac 660e1ac3 System.Web.UI.WebControls.BaseDataBoundControl.EnsureDataBound()
04e95fd8 668ecdcd System.Web.UI.WebControls.BaseDataBoundControl.set_RequiresDataBinding(Boolean)
04e95fe8 667da23b System.Web.UI.WebControls.GridView.set_PageIndex(Int32)
04e95ff8 0ad3e096 ASP.search_aspx.Gridview1_DataBound(System.Object, System.EventArgs)

[...]
04ebea70 0ad3e096 ASP.search_aspx.Gridview1_DataBound(System.Object, System.EventArgs)
04ebef70 667928ce System.Web.UI.WebControls.BaseDataBoundControl.OnDataBound(System.EventArgs)
04ebef84 66792b45 System.Web.UI.WebControls.DataBoundControl.OnDataSourceViewSelectCallback(System.Collections.IEnumerable)
04ebef94 6668f4a4 System.Web.UI.DataSourceView.Select(System.Web.UI.DataSourceSelectArguments, System.Web.UI.DataSourceViewSelectCallback)
04ebefa4 66792c0f System.Web.UI.WebControls.DataBoundControl.PerformSelect()
04ebefbc 6679285e System.Web.UI.WebControls.BaseDataBoundControl.DataBind()
04ebefc8 667dcf35 System.Web.UI.WebControls.GridView.DataBind()
04ebefcc 660e1ac3 System.Web.UI.WebControls.BaseDataBoundControl.EnsureDataBound()
04ebeff8 660e1a57 System.Web.UI.WebControls.BaseDataBoundControl.OnPreRender(System.EventArgs)
04ebf008 667e07da System.Web.UI.WebControls.GridView.OnPreRender(System.EventArgs)
04ebf030 660abc21 System.Web.UI.Control.PreRenderRecursiveInternal()
04ebf048 660abc7c System.Web.UI.Control.PreRenderRecursiveInternal()
04ebf060 660abc7c System.Web.UI.Control.PreRenderRecursiveInternal()
04ebf078 660abc7c System.Web.UI.Control.PreRenderRecursiveInternal()
04ebf090 660abc7c System.Web.UI.Control.PreRenderRecursiveInternal()
04ebf0a8 660a7c4b System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)
04ebf200 660a77a4 System.Web.UI.Page.ProcessRequest(Boolean, Boolean)
04ebf238 660a76d1 System.Web.UI.Page.ProcessRequest()
04ebf270 660a7666 System.Web.UI.Page.ProcessRequestWithNoAssert(System.Web.HttpContext)
04ebf27c 660a7642 System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)
04ebf290 0ad37afe ASP.search_aspx.ProcessRequest(System.Web.HttpContext)
04ebf2a0 660adb16 System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
04ebf2d4 6608132c System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)
04ebf314 6608c5c3 System.Web.HttpApplication+ApplicationStepManager.ResumeSteps(System.Exception)
04ebf364 660808ac System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest(System.Web.HttpContext, System.AsyncCallback, System.Object)
04ebf380 66083e1c System.Web.HttpRuntime.ProcessRequestInternal(System.Web.HttpWorkerRequest)
04ebf3b4 66083ac3 System.Web.HttpRuntime.ProcessRequestNoDemand(System.Web.HttpWorkerRequest)
04ebf3c4 66082c5c System.Web.Hosting.ISAPIRuntime.ProcessRequest(IntPtr, Int32)
04ebf5d8 79f68cde [ContextTransitionFrame: 04ebf5d8] 
04ebf60c 79f68cde [GCFrame: 04ebf60c] 
04ebf768 79f68cde [ComMethodFrame: 04ebf768]

After the DataBound event hander is executed, we always call set_PageIndex and go through the binding logic over and over. What does it mean?

Well, either use Reflector or download the .NET Framework source (or the SSCLI, Shared Source Common Language Infrastructure), if you have a look at the implementation of set_PageIndex you’ll see something like this:

public virtual void set_PageIndex(int value)
{
    if (value < 0)
    {
        throw new ArgumentOutOfRangeException("value");
    }
    if (this.PageIndex != value)
    {
        this._pageIndex = value;
        if (base.Initialized)
        {
            base.RequiresDataBinding = true;
        }
    }
}

Going deeper we see this:

protected bool RequiresDataBinding()
{
    get
    {
        return this._requiresDataBinding;
    }
    set
    {
        if (((value && this._preRendered) && ((this.DataSourceID.Length > 0) && (this.Page != null))) && !this.Page.IsCallback)
        {
            this._requiresDataBinding = true;
            this.EnsureDataBound();
        }
        else
        {
            this._requiresDataBinding = value;
        }
    }
}



 

 
protected virtual void EnsureDataBound()
{
    try
    {
        this._throwOnDataPropertyChange = true;
        if (this.RequiresDataBinding && ((this.DataSourceID.Length > 0) || this._requiresBindToNull))
        {
            this.DataBind();
            this._requiresBindToNull = false;
        }
    }
    finally
    {
        this._throwOnDataPropertyChange = false;
    }
}

Here is the source code of the faulting page:

Sub Gridview1_DataBound(ByVal sender As Object, ByVal e As EventArgs)
    Try
        If GridView1.Rows.Count > 0 Then
            Dim pagerRow As GridViewRow = GridView1.BottomPagerRow
                
            [...]

            Dim CurrentPage As String
     
            If Request.QueryString("page") <> "" Then
                CurrentPage = Request.QueryString("page")
            Else
                CurrentPage = 1
            End If
    
            Try
                GridView1.PageIndex = CurrentPage - 1
            Catch
                CurrentPage = 1
                GridView1.PageIndex = CurrentPage - 1
            End Try
    
            [...]  

    End Try
End Sub

So, if we change the PageIndex value, under some circumstances we call the DataBind method, the OnDataBound event is fired, the DataBound event hander is executed, PageIndex is changed again and so on until the stack is exhausted.

Bottom line here is: do not change the PageIndex value inside the DataBound event handler!

DataSource or DataSourceID?

One of the conditions to implicitly call DataBind() is to have a value set for DataSourceID:

protected bool RequiresDataBinding()
{    
    get    
    {        
        return this._requiresDataBinding;    
    }    
    set    
    {        
        if (((value && this._preRendered) && ((this.DataSourceID.Length > 0) && (this.Page != null))) && !this.Page.IsCallback)            
        {            
            this._requiresDataBinding = true;            
            this.EnsureDataBound();        
        }        
        else        
        {            
            this._requiresDataBinding = value;        
        }    
    }
}

Honestly I have not tried it (I’ll do and update the post if needed), but I think if we use DataSource rather than DataSourceID we’ll not incur in this problem.

Carlo

Quote of the day:
There is nothing more dreadful than imagination without taste. - Johann Wolfgang von Goethe

SharePoint and Office Live? Access forbidden!

2009-07-06T22:50:21Z

I recently came across an odd problem the customer got on his SharePoint server: when the clients had the Office Live add-on installed they were constantly refused with a 403 error. The problem is well described here.

If you have this problem, there is a fix for you: 970946.

If you do not want to install the fix, you might consider to filter the UserAgent string you receive on the server (well, you should filter it before the web request is processed on the server) as one of my customers is testing: if the UserAgent string contains the word “Mozilla” and “Office”* or “non-browser” or “FrontPage”*, just remove the unwanted token.

Carlo

Quote of the day:
All things are difficult before they are easy. - Thomas Fuller

Three errors, one cause: me!

2009-06-03T14:29:17Z

A few days ago I was working on a repro I got from a customer as I very often do, and for some reason after a while my IIS started to behave oddly, pages where not served, the application pool was stuck doing “something” and whenever I tried to open the IIS Manager I got the error “There was an error when trying to connect. Do you want to retype your credentials and try again? Cannot create a file when that file already exists”:

Needless to say that being IIS on my local machine and I have not made any changes to the security/ACLs/DCOM etc… there should be no need to retype my credentials; and by the way, even retyping them did not help.

By the way, if I tried to remotely connect to another machine of mine, I got the message “The type initializer for ‘Microsoft.Web.Management.Remoting.HttpChannel’ threw an exception”:

My next step has been to try to restart the World Wide Web Publishing Service: it was (oddly?) stopped, but again I got an error “Windows could not start the World Wide Web Publishing Service service on Local Computer. Error 1068: The dependency service or group failed to start”:

The W3SVC service has a dependency on WAS (Windows Process Activation Service), I tried to start it manually and got “Windows could not start the Windows Process Activation Service service on Local Computer. Error 183: Cannot create a file when that file already exists.”:

That is the same message I got from IIS Manager, so I thought to have a look at the event log to find out more: I was wrong! I got “MMC could not create the snap-in. The snap-in might not have been installed correctly”:

I quickly found out that virtually every MMC snap-in was broken on my machine (“Server Manager” “Certificates”, “Local Users and Groups”, just name one) but why?!?

Well… on Windows 2008, MMC is based on .NET as of course is IIS and when the problem first appeared I was tampering my machine.config file to try different ThreadPool settings…

The golden rule in computing reads “Always back-up sensitive data!”… after restoring my clean machine.config all the problems above went away!

For curiosity I had a deeper look at it and found the following lines:

<system.web>

        <processModel autoConfig="true"/>

        <processModel autoConfig="false" minWorkerThreads="100" 
            minIoThreads="100" maxWorkerThreads="404" maxIoThreads="404"/>

        <httpRuntime minFreeThreads="152" minLocalRequestFreeThreads="152"/>

[...]

Easy to spot it now, eh? For my repro I had set autoConfig=”false” but then I decided to set it to “true”, for some reason I added a new <processModel> element with its own autoConfig=”true” and forgot to remove the “old” <processModel>… At least I learnt something new from my foolishness!

Note: there might be other reasons if you get the errors above, but always keep a clean backup of your configuration files at hand, just in case…

Carlo

Quote of the day:
We are continually faced with a series of great opportunities brilliantly disguised as insoluble problems. - John W. Gardner

Logparser automated within Windbg

2009-05-08T19:50:12Z

The .shell command in Windbg allows to pipe the output of a debugger command to an external process and automatically print its output back inside the debugger window; a useful example is the command FIND, for example if we want to parse the stack for every thread and find every call where the word “isapi” is involved:

0:036> .shell -ci "~*kpL1000" find /i "isapi"
013cff78 1004f94e ISAPI_Rewrite!TerminateFilter+0x3cef
013cffb0 1004f9f3 ISAPI_Rewrite!TerminateFilter+0x4462e
013cffec 00000000 ISAPI_Rewrite!TerminateFilter+0x446d3
0144ff78 1004f94e ISAPI_Rewrite+0x7763
0144ffb0 1004f9f3 ISAPI_Rewrite!TerminateFilter+0x4462e
0144ffec 00000000 ISAPI_Rewrite!TerminateFilter+0x446d3
.shell: Process exited

Incidentally also LogParser (one of my favorite debugging tools) can accept data to be parsed from the input stream using the STDIN keyword, so for example refactoring a script I posted some time ago we can find out if there are any duplicated assemblies in our application pool that should be moved to the GAC:

0:000> .shell -ci "!peb" logparser "select extract_filename(text) as Duplicated_Assemblies, count(Duplicated_Assemblies) as Hits from STDIN where index_of(text, 'temporary asp.net files') > 0 group by Duplicated_Assemblies having count(Duplicated_Assemblies) > 1" -i:textline -o:nat -rtp:-1
Duplicated_Assemblies        Hits 
---------------------------- ----
errormanager.dll             2
winformsui.dll               2
externallibraryinterface.dll 2
ptshopengine.dll             2
schemas.dll                  2
dbengine.dll                 2
flowservice.dll              2

Statistics:
-----------
Elements processed: 182
Elements output:    7
Execution time:     0.02 seconds

.shell: Process exited

Following the same principle, we can find out if there are strong named assemblies in our /bin folder as follows:

0:000> .shell -ci "!dumpdomain" find /i "shared domain"
Shared Domain: 0x793f2aa8
.shell: Process exited



0:000> .shell -ci "!dumpdomain 0x793f2aa8" logparser "SELECT DISTINCT EXTRACT_FILENAME(text) as Strong_Named_Assemblies_In_/bin FROM STDIN WHERE INDEX_OF(to_lowercase(text), 'temporary asp.net files') > 0" -i:TEXTLINE -o:NAT -RTP:-1
Strong_Named_Assemblies_In_/bin 
-----------------------------------------
crypto.dll
radplaceholder.dll
scms.dll
sqldac.dll
scontrollibrary.dll
spell.dll
editor.dll
scms.resources.dll

Statistics:
-----------
Elements processed: 164
Elements output:    8
Execution time:     0.01 seconds

.shell: Process exited

Instead of typing the whole command you can save it in a text file and execute it directly within Windbg with a command like “$><c:\debuggers\snassemblies.txt”.

Carlo

Quote of the day:
Setting a good example for children takes all the fun out of middle age. - William Feather

StackOverflowException and DataBind()

2009-04-27T20:05:07Z

The application pool for this site was getting disabled quite frequently and we found quite a few entries like the following in the event log:

Event Type: Error
Event Source: W3SVC
Event Category: None
Event ID: 1002
Date: 19/11/2008
Time: 15:20:23
User: N/A
Computer: <computername>
Description: Application pool 'DefaultAppPool' is being automatically disabled due to a series of failures in the process(es) serving that application pool

In this case the customer already had some debugging skills so when he called CSS he already had a few dumps available to analyze; they were some process shutdown dumps (taken on Kernel32!TerminateProcess), strangely we had no second chance dumps (maybe because of one of these reasons?) but they’ve been a good point to start from anyway.

First, let’s have a look at the exceptions:

0:000> !dumpheap -type Exception -stat 

[...]

Statistics:
      MT    Count    TotalSize Class Name
79333ed4        1           12 System.Text.DecoderExceptionFallback
79333e90        1           12 System.Text.EncoderExceptionFallback
79330d44        1           72 System.ExecutionEngineException
79330cb4        1           72 System.StackOverflowException
79330c24        1           72 System.OutOfMemoryException
7931ffd4        1           72 System.NullReferenceException
6610c7fc        1           84 System.Web.HttpUnhandledException
79330dd4        2          144 System.Threading.ThreadAbortException
79318afc        2          144 System.InvalidOperationException
7931740c        5          160 System.UnhandledExceptionEventHandler
Total 16 objects

Remember, a few special exceptions are loaded when the AppDomain is first created (see here) so let’s try to see if there is still something significant on the threads:

0:000> !threads
ThreadCount: 21
UnstartedThread: 0
BackgroundThread: 21
PendingThread: 0
DeadThread: 0
Hosted Runtime: no
                                      PreEmptive   GC Alloc           Lock
       ID OSID ThreadOBJ    State     GC       Context       Domain   Count APT Exception
  17    1  950 000f2580   1808220 Enabled  00000000:00000000 000eed48     0 MTA (Threadpool Worker)
  27    2 1bfc 00100018      b220 Enabled  00000000:00000000 000eed48     0 MTA (Finalizer)
  28    3  300 00117068    80a220 Enabled  00000000:00000000 000eed48     0 MTA (Threadpool Completion Port)
  29    4 1cf4 0011a458      1220 Enabled  00000000:00000000 000eed48     0 Ukn
   9    5  ec0 001702e0       220 Enabled  00000000:00000000 000eed48     0 Ukn
  31    6 2414 00171f68   180b220 Enabled  00000000:00000000 000eed48     0 MTA (Threadpool Worker)
  32    7 1f24 00172338   180b220 Enabled  00000000:00000000 000eed48     0 MTA (Threadpool Worker)
  33    8  97c 001729e0   180b220 Enabled  00000000:00000000 000eed48     0 MTA (Threadpool Worker)
  34    9 13bc 00173088   180b220 Enabled  00000000:00000000 000eed48     0 MTA (Threadpool Worker)
  35    a 22f8 00173a50   180b220 Enabled  00000000:00000000 000eed48     0 MTA (Threadpool Worker)
  36    b 1e60 00174540   180b220 Disabled 12445a64:124475b8 0011ab48     1 MTA (Threadpool Worker) System.StackOverflowException (1a0a00a4)
  37    c 1c8c 00174cf8   180b220 Enabled  00000000:00000000 000eed48     0 MTA (Threadpool Worker)
  15    d 1934 0017d040   880a220 Enabled  00000000:00000000 000eed48     0 MTA (Threadpool Completion Port)
   7    e 2028 0017ece8       220 Enabled  00000000:00000000 000eed48     0 Ukn
   8    f  dbc 0016f280       220 Enabled  00000000:00000000 000eed48     0 Ukn
   4   10 2298 05007008       220 Enabled  00000000:00000000 000eed48     0 Ukn
   6   11 20c0 05004278       220 Enabled  00000000:00000000 000eed48     0 Ukn
   3   12  c9c 05003b70       220 Enabled  00000000:00000000 000eed48     0 Ukn
   5   13 1a84 04ff7a60       220 Enabled  00000000:00000000 000eed48     0 Ukn
   2   14 253c 0d378ee0       220 Enabled  00000000:00000000 000eed48     0 Ukn
  42   15 1760 0d38b3d8       220 Enabled  00000000:00000000 000eed48     0 Ukn

Dumping the managed stack for thread 36, the recursion is quite obvious:

0:036> !clrstack
OS Thread Id: 0x1e60 (36)
ESP       EIP     
04e94acc 77e42004 [FaultingExceptionFrame: 04e94acc] 
04e95058 77e42004 [HelperMethodFrame: 04e95058] 
04e950c4 65230cc1 System.Data.RBTree`1+TreePage[[System.__Canon, mscorlib]]..ctor(Int32)
04e950d4 65230abe System.Data.RBTree`1[[System.__Canon, mscorlib]].AllocPage(Int32)
04e950f8 652309fc System.Data.RBTree`1[[System.__Canon, mscorlib]].InitTree()
04e95108 65230961 System.Data.DataRowCollection..ctor(System.Data.DataTable)
04e95118 652307c2 System.Data.DataTable..ctor()
04e95134 65230535 System.Data.Common.DataTableMapping.GetDataTableBySchemaAction(System.Data.DataSet, System.Data.MissingSchemaAction)
04e95150 6523037c System.Data.ProviderBase.SchemaMapping..ctor(System.Data.Common.DataAdapter, System.Data.DataSet, System.Data.DataTable, System.Data.ProviderBase.DataReaderContainer, Boolean, System.Data.SchemaType, System.String, Boolean, System.Data.DataColumn, System.Object)
04e9519c 6522fbac System.Data.Common.DataAdapter.FillMappingInternal(System.Data.DataSet, System.Data.DataTable, System.String, System.Data.ProviderBase.DataReaderContainer, Int32, System.Data.DataColumn, System.Object)
04e951d0 6522fc1e System.Data.Common.DataAdapter.FillMapping(System.Data.DataSet, System.Data.DataTable, System.String, System.Data.ProviderBase.DataReaderContainer, Int32, System.Data.DataColumn, System.Object)
04e95218 6522f9e6 System.Data.Common.DataAdapter.FillFromReader(System.Data.DataSet, System.Data.DataTable, System.String, System.Data.ProviderBase.DataReaderContainer, Int32, Int32, System.Data.DataColumn, System.Object)
04e95270 6522f942 System.Data.Common.DataAdapter.Fill(System.Data.DataSet, System.String, System.Data.IDataReader, Int32, Int32)
04e952b4 65230105 System.Data.Common.DbDataAdapter.FillInternal(System.Data.DataSet, System.Data.DataTable[], Int32, Int32, System.String, System.Data.IDbCommand, System.Data.CommandBehavior)
04e9530c 65230010 System.Data.Common.DbDataAdapter.Fill(System.Data.DataSet, Int32, Int32, System.String, System.Data.IDbCommand, System.Data.CommandBehavior)
04e95350 6559401d System.Data.Common.DbDataAdapter.Fill(System.Data.DataSet, System.String)
04e95384 6678d11f System.Web.UI.WebControls.SqlDataSourceView.ExecuteSelect(System.Web.UI.DataSourceSelectArguments)
04e95424 6668f498 System.Web.UI.DataSourceView.Select(System.Web.UI.DataSourceSelectArguments, System.Web.UI.DataSourceViewSelectCallback)
04e95434 66792c0f System.Web.UI.WebControls.DataBoundControl.PerformSelect()
04e9544c 6679285e System.Web.UI.WebControls.BaseDataBoundControl.DataBind()
04e95458 667dcf35 System.Web.UI.WebControls.GridView.DataBind()
04e9545c 660e1ac3 System.Web.UI.WebControls.BaseDataBoundControl.EnsureDataBound()
04e95488 668ecdcd System.Web.UI.WebControls.BaseDataBoundControl.set_RequiresDataBinding(Boolean)
04e95498 667da23b System.Web.UI.WebControls.GridView.set_PageIndex(Int32)
04e954a8 0ad3e096 ASP.search_aspx.Gridview1_DataBound(System.Object, System.EventArgs)
04e959a8 667928ce System.Web.UI.WebControls.BaseDataBoundControl.OnDataBound(System.EventArgs)
04e959bc 66792b45 System.Web.UI.WebControls.DataBoundControl.OnDataSourceViewSelectCallback(System.Collections.IEnumerable)
04e959cc 6668f4a4 System.Web.UI.DataSourceView.Select(System.Web.UI.DataSourceSelectArguments, System.Web.UI.DataSourceViewSelectCallback)
04e959dc 66792c0f System.Web.UI.WebControls.DataBoundControl.PerformSelect()
04e959f4 6679285e System.Web.UI.WebControls.BaseDataBoundControl.DataBind()
04e95a00 667dcf35 System.Web.UI.WebControls.GridView.DataBind()
04e95a04 660e1ac3 System.Web.UI.WebControls.BaseDataBoundControl.EnsureDataBound()
04e95a30 668ecdcd System.Web.UI.WebControls.BaseDataBoundControl.set_RequiresDataBinding(Boolean)
04e95a40 667da23b System.Web.UI.WebControls.GridView.set_PageIndex(Int32)
04e95a50 0ad3e096 ASP.search_aspx.Gridview1_DataBound(System.Object, System.EventArgs)
04e95f50 667928ce System.Web.UI.WebControls.BaseDataBoundControl.OnDataBound(System.EventArgs)
04e95f64 66792b45 System.Web.UI.WebControls.DataBoundControl.OnDataSourceViewSelectCallback(System.Collections.IEnumerable)
04e95f74 6668f4a4 System.Web.UI.DataSourceView.Select(System.Web.UI.DataSourceSelectArguments, System.Web.UI.DataSourceViewSelectCallback)
04e95f84 66792c0f System.Web.UI.WebControls.DataBoundControl.PerformSelect()
04e95f9c 6679285e System.Web.UI.WebControls.BaseDataBoundControl.DataBind()
04e95fa8 667dcf35 System.Web.UI.WebControls.GridView.DataBind()
04e95fac 660e1ac3 System.Web.UI.WebControls.BaseDataBoundControl.EnsureDataBound()
04e95fd8 668ecdcd System.Web.UI.WebControls.BaseDataBoundControl.set_RequiresDataBinding(Boolean)
04e95fe8 667da23b System.Web.UI.WebControls.GridView.set_PageIndex(Int32)
04e95ff8 0ad3e096 ASP.search_aspx.Gridview1_DataBound(System.Object, System.EventArgs)

[...]
04ebea70 0ad3e096 ASP.search_aspx.Gridview1_DataBound(System.Object, System.EventArgs)
04ebef70 667928ce System.Web.UI.WebControls.BaseDataBoundControl.OnDataBound(System.EventArgs)
04ebef84 66792b45 System.Web.UI.WebControls.DataBoundControl.OnDataSourceViewSelectCallback(System.Collections.IEnumerable)
04ebef94 6668f4a4 System.Web.UI.DataSourceView.Select(System.Web.UI.DataSourceSelectArguments, System.Web.UI.DataSourceViewSelectCallback)
04ebefa4 66792c0f System.Web.UI.WebControls.DataBoundControl.PerformSelect()
04ebefbc 6679285e System.Web.UI.WebControls.BaseDataBoundControl.DataBind()
04ebefc8 667dcf35 System.Web.UI.WebControls.GridView.DataBind()
04ebefcc 660e1ac3 System.Web.UI.WebControls.BaseDataBoundControl.EnsureDataBound()
04ebeff8 660e1a57 System.Web.UI.WebControls.BaseDataBoundControl.OnPreRender(System.EventArgs)
04ebf008 667e07da System.Web.UI.WebControls.GridView.OnPreRender(System.EventArgs)
04ebf030 660abc21 System.Web.UI.Control.PreRenderRecursiveInternal()
04ebf048 660abc7c System.Web.UI.Control.PreRenderRecursiveInternal()
04ebf060 660abc7c System.Web.UI.Control.PreRenderRecursiveInternal()
04ebf078 660abc7c System.Web.UI.Control.PreRenderRecursiveInternal()
04ebf090 660abc7c System.Web.UI.Control.PreRenderRecursiveInternal()
04ebf0a8 660a7c4b System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)
04ebf200 660a77a4 System.Web.UI.Page.ProcessRequest(Boolean, Boolean)
04ebf238 660a76d1 System.Web.UI.Page.ProcessRequest()
04ebf270 660a7666 System.Web.UI.Page.ProcessRequestWithNoAssert(System.Web.HttpContext)
04ebf27c 660a7642 System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)
04ebf290 0ad37afe ASP.search_aspx.ProcessRequest(System.Web.HttpContext)
04ebf2a0 660adb16 System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
04ebf2d4 6608132c System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)
04ebf314 6608c5c3 System.Web.HttpApplication+ApplicationStepManager.ResumeSteps(System.Exception)
04ebf364 660808ac System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest(System.Web.HttpContext, System.AsyncCallback, System.Object)
04ebf380 66083e1c System.Web.HttpRuntime.ProcessRequestInternal(System.Web.HttpWorkerRequest)
04ebf3b4 66083ac3 System.Web.HttpRuntime.ProcessRequestNoDemand(System.Web.HttpWorkerRequest)
04ebf3c4 66082c5c System.Web.Hosting.ISAPIRuntime.ProcessRequest(IntPtr, Int32)
04ebf5d8 79f68cde [ContextTransitionFrame: 04ebf5d8] 
04ebf60c 79f68cde [GCFrame: 04ebf60c] 
04ebf768 79f68cde [ComMethodFrame: 04ebf768]

The next step is to find out why the OnDataBound event is firing over and over; here is the source code for the faulting page (simplified to show only the relevant bits):

Sub Page_Load(ByVal Sender As Object, ByVal E As EventArgs)

Dim cmd As String = "select * from products"

If Not IsPostBack Then

' when there is a blank search, display all products
If SearchString = "" Then

SqlDataSource1.SelectCommand = cmd
GridView1.DataBind()

Else
' when there are keywords (not a blank search)

' some logic to change the sql query
GridView1.DataBind()

If GridView1.Rows.Count < 3 Then ' displays suggestions if results are less then 3
For i = 0 To Keywords.Count - 1
' some logic to change the sql query
GridView1.DataBind()
Next

For i = 0 To FoundNewKeywords.Count - 1
For s = 0 To Keywords.Count - 1
' some logic to change the sql query
GridView1.DataBind()
Next

For j = i To FoundNewKeywords.Count - 1
' some logic to change the sql query
GridView1.DataBind()

If j = FoundNewKeywords.Count - 1 And FoundNewKeywords.Count > 2 Then
For m = j - 1 To 1 Step -1
' some logic to change the sql query
GridView1.DataBind()
Next

If i < 1 Then
For m = 2 To j - 1
' some logic to change the sql query
GridView1.DataBind()
Next
End If
End If
Next
Next
End If
End If
End If
End Sub

Sub Gridview1_DataBound(ByVal sender As Object, ByVal e As EventArgs)
' Some custom logic here
End Sub

From a high level perspective the intention was to start with a “return everything” query and refine it step by step taking into account keywords specified by the user and add logic to suggest other records that might interest the user. From an operation standpoint calling DataBind() multiple times means the whole data binding logic is executed over and over, such as the custom DataBound event handler: that way, depending on what we’re doing, we could exhaust the stack space which is what was happening in this case.

The solution is simple: there is no need to call DataBind() many times, just call it once after we have refined our query.

Carlo

Quote of the day:
To avoid situations in which you might make mistakes may be the biggest mistake of all. - Peter McWilliams

Where’s my dump gone?!?

2009-04-14T20:45:37Z

Have you ever found yourself patiently waiting for a problem to reproduce with your debugger ready, and when it happens you just find there are no signs of your dump anywhere? If you are using adplus you likely had the text logs, but nothing more…

In such situations it is possible that the OS is terminating the process before the dump is fully written. If you are tying to dump an IIS process and you want to save some time and headaches, try this small cunning: temporarily disable the “Enable pinging” and “Enable rapid-fail protection” flags in the Health tab for your application pool:

Sometimes can be useful to tell IIS to not kill a failing worker process but rather leave it orphaned:

Features of Worker Process Isolation Mode

Orphaning Worker Processes

You can configure worker process isolation mode to orphan a worker process that the WWW service deems to be failing. The WWW service usually terminates a failing worker process and replaces it. If you enable orphaning, the WWW service allows a failing worker process to continue running, but separates it from the application pool (making it an orphan) and starts a new worker process in its place. You can configure the WWW service to run a command on the orphaned worker process — for example, launching a debugger.

For more information about orphaning worker processes, including how to configure this feature, see Running IIS 6.0 as an Application Server in this book and OrphanWorkerProcess Metabase Property metabase property.

In IIS 6 there the following three metabase properties can be handy to run some custom actions in case of an application pool failure:

OrphanWorkerProcess: The OrphanWorkerProcess property, when set to true, notifies the World Wide Web Publishing Service (WWW Service) not to terminate a worker process that fails to respond to pings, but to instead orphan the worker process in the application pool if the worker process suffers fatal errors
OrphanActionParams: The OrphanActionParams property specifies command-line parameters for the executable specified by the OrphanActionExe Metabase Property. To specify the process ID of the orphaned process, use %1%.
OrphanActionExe: The OrphanActionExe property specifies an executable to run when the World Wide Web Publishing Service (WWW service) orphans a worker process. You can use the OrphanActionParams Metabase Property to send parameters to the executable

The article How to generate a dump file when ASP.NET deadlocks in IIS 6.0 explains how to use the properties above and despite its title, you can use this technique to run the custom actions you need/want not only in case of an ASP.NET deadlock but in every circumstance that fits your needs.

In IIS 7 you can easily use the Advanced Settings dialog or your application pool:

Finally, while w3wp.exe cannot run without Data Execution Prevention, if you are debugging a custom process you may want to add it to the exclusion list:

Carlo

Quote of the day:
If the only tool you have is a hammer, you tend to see every problem as a nail. - Abraham Maslow

Value	Behavior
Does not exist	This is the default behavior. For each subdirectory, the application will create an object that will monitor the subdirectory
0 or greater than 2	This is the default behavior. For each subdirectory, the application will create an object that will monitor the subdirectory
1	The application will disable File Change Notifications (FCNs)
2	The application will create one object to monitor the main directory. The application will use this object to monitor each subdirectory