March, 2008

  • Never doubt thy debugger

    Unable to use SQL Server because ASP.NET version 2.0 Session State is not installed on the SQL server


    I run across this error today while setting up a repro for a problem I'm working on; the first time I through I did something wrong between the mess of things which case across my hands today (maybe I've used the wrong .sql script? smile_thinking), but after removing the database from Sql Server and re-running correct InstallSqlState.sql, I got the same message once again smile_sarcastic.

    Ok then, time for a (hopefully) quick research on this error and finally I found out a few things which might cause it.

    Running  both ASP.NET and classic ASP in the same application pool?

    If the application pool is also running classic ASP pages, and those classic ASP pages use .NET 2.0 components, and those classic ASP pages which use .NET components are called before any ASP.NET 1.1 page, then we'll load CLR 2.0 (first come, first serve smile_regular) and of course it will look for his specific ASPState version.

    Ok, there are a lot of "if" in this case, but it's still a possibility... not my scenario, through.

    Where are your session tables?

    If you use aspnet_regsql wizard, session tables are not added by default so you need to run the following command:

    aspnet_regsql.exe -S <servername> -E -ssadd -sstype p 

    Again, that was not my case.

    Are you sure you can run it? 

    Of course we still need permissions to access the database... so make sure the account used in your connection string can connect to the database has EXEC permission on the following stored procedures in ASPState database:

    • TempGetAppID
    • TempGetStateItem
    • TempGetStateItemExclusive
    • TempReleaseStateItemExclusive
    • TempInsertStateItemLong
    • TempInsertStateItemShort
    • TempUpdateStateItemLong
    • TempUpdateStateItemShort
    • TempUpdateStateItemShortNullLong
    • TempUpdateStateItemLongNullShort
    • TempRemoveStateItem
    • TempResetTimeout

    Well... to make things easier in my sample, I just granted NETWORK SERVICE dbo permission on ASPState and I got my repro up an running.

    Need self irony and a joke

    What can I say? The error message is not very helpful in this situation (ok, it's misleading smile_omg), and this reminded me a joke I heard a few years ago before joining Microsoft... it's something like the jokes we have in Italy against our Carabinieri (one of Italian police forces); but I warn you, I'm not a good storyteller...

    There is a group of friends in trouble on their helicopter, the radio is broken, there is a thick fog and they don't know where they are. Suddenly they see the shape of a skyscraper and try to attract the attention of people inside their offices; someone on the helicopter writer down on a peace of paper: "Radio broken, need help, where are we?". Immediately the people inside the office start to talk each other and finally someone writes the answer on a sheet of paper: "You're in an helicopter at about 30 feet from the ground, and in this foggy day you'd better stay at home". The pilot of the helicopter at first looks puzzled, then smiles, nods thankfully and flight back to the airport where they land safely. Other passengers were amazed and asked the pilot how he had been able to find their way home so quickly
    "Easy", he replied "we were at Microsoft's building"
    "How did you know?"
    "We asked a simple and direct question and they gave a vague and completely useless answer!"

    Ok, I told you I'm not a good storyteller...smile_teeth


    Quote of the day:
    I have a rock garden. Last week three of them died. - Richard Diran
  • Never doubt thy debugger

    MaxConnection does not have effect on ASP.NET 2.0


    An interesting change which you might have missed in .NET 2.0 is about the autoConfig setting for the <processModel> element. The customer was running out of available connections while accessing a remote web service and was trying to apply what’s suggested in Contention, poor performance, and deadlocks when you make Web service requests from ASP.NET applications but apparently the change had not effect: a memory dump shown still 12 available connections even if he set MaxConnection=96. Interesting to note that reverting the application to .NET 1.1 the problem disappeared.

    The different behavior is determined by the autoConfig value, which affects only ASP.NET applications and not Console, WinForm etc… (while the <processModel> threading configuration in .NET 1.x affected all .NET apps running on the machine):

    Specifies whether to automatically configure the following settings to achieve optimal performance based on the machine configuration:

    The values are set according to the KB article at
    This attribute does not affect the .NET Framework client applications; only ASP.NET applications.
    The autoConfig attribute can be one of the following values.

    • True: Indicates that ASP.NET automatically configures the attributes in the preceding list to achieve optimal performance based on the machine configuration.
    • False: Indicates that ASP.NET should use the explicitly defined values for the attributes in the preceding list.

    The default in the Machine.config file is True, unless there is a previously existing configuration.

    If you don’t like this auto threading feature you can simply set autoConfig=”false” or set System.Net.ServicePointManager.DefaultConnectionLimit=<your value here>



    Quote of the day:
    The male is a domestic animal which, if treated with firmness, can be trained to do most things. - Jilly Cooper
  • Never doubt thy debugger

    And when you think there's nothing left to do... you can still debug live


    There are circumstances where taking a dump is not possible or simply not convenient; imagine a situation where you're hunting a nasty exception you don't know where it comes from and you need to follow some complex steps to get there, or when you have dozens of that exception (let's say an Access Violation) but you're interested in a specific one you get only when on click on a specific button, or after you paginate long enough through a GridView (and of course you don't want dozens of full dumps, 1 Gb each), or if you're hunting for some security related problem and you need to inspect the security token your thread is using in a particular moment (something I did here) etc... For sure if you ask Doug he'll have dozens of good answers to this question smile_regular 

    Getting started

    Let's assume you already have correctly setup your Windbg workspace, symbols etc..., then you can go from File > Attach to a process... (or hit F6) to get the Attach to process dialog below:

    Attach to process

    Note that if you expand the tree for your target process you'll see some information about the process' session, this is useful if for example you have multiple instances of ASP.NET worker process which loads different versions of the runtime, or if you're interested in a specific application pool (check this post) etc...

    The Noninvasive flag is important. With noninvasive debugging, you do not have as many debugging actions, however you can minimize the debugger's interference with the target application. In noninvasive debugging, the debugger does not actually attach to the target application, but it rather suspends all of the target's threads and has access to the target's memory, registers, and other such information. However, the debugger cannot control the target, so commands like "g" (Go, more on this later) do not work. If you try to execute commands that are not permitted during noninvasive debugging, you receive an error message that states "The debugger is not attached, so process execution cannot be monitored".

    When you end the debugging session, the debugger releases the target application and the application continues running. You should close the session by using q (Quit), .detach (Detach from Process), or WinDbg's Debug > Detach Debuggee or Debug > Stop Debugging command. If you close the debugging session by closing the debugger window or by using the Exit command on the File menu in WinDbg, your target application typically stops responding. Noninvasive debugging is useful if you do not want to end the target application at the end of the session, and the target application is running on Microsoft Windows NT or Windows 2000 (these operating systems do not support detaching from a target that the debugger has actually attached to). Noninvasive debugging is also useful if the target application has stopped responding and cannot open the break thread that you need to attach.

    Now we need a breakpoint

    It's important to note that when you attach the debugger to your process, you'll actually suspend all of its threads with the result that the process will be "hung", completely unable to serve any incoming request from your users; this is the reason why generally it's not a good idea to attach to a production process, unless this is the only way left... Once attached, the debugger will stop on a thread waiting for your input. Depending on what you're looking for, you might need to set a breakpoint somewhere in the application (like you can do in Visual Studio) and then let the application run until you hit the breakpoint.

    The most simple command to use is:

    sxe clr

    The sx* commands in Windbg control the action that the debugger takes when an exception occurs in the application that is being debugged, or when certain events occur; with this command we're telling the debugger to stop on every managed exception raised. Note that the code 0xE0434F4D means you're dealing with a COM exception which is bubbled up to the CLR.

    If you need to stop on a specific managed exception, then the SOS extension comes at hand with !StopOnException:

    !StopOnException helps when you want the Windows Debugger to stop on a particular managed exception, say a System.OutOfMemoryException, but continue running if other exceptions are thrown. The command can be used in two ways:

    • When you just want to stop on one particular CLR exception

       At the debugger prompt, anytime after loading SOS, type:

       !StopOnException -create System.OutOfMemoryException 1

       The pseudo-register number (1) indicates that SOS can use register $t1 for
       maintaining the breakpoint. The -create parameter allows SOS to go ahead
       and set up the breakpoint as a first-chance exception. -create2 would set
       it up as a 2nd-chance exception.

    • When you need more complex logic for stopping on a CLR exception

       !StopOnException can be used purely as a predicate in a larger expression.
       If you type:

       !StopOnException System.OutOfMemoryException 3

       then register $t3 will be set to 1 if the last thrown exception on the
       current thread is a System.OutOfMemoryException. Otherwise, $t3 will be set
       to 0. Using the Windows Debugger scripting language, you could chain
       such calls together to stop on various exception types. You'll have to
       manually create such predicates, for example:

       sxe -c "!soe System.OutOfMemoryException 3;
               !soe -derived System.IOException 4;
               .if(@$t3==1 || @$t4==1) { .echo 'stop' } .else {g}"

    The -derived option will cause StopOnException to set the pseudo-register to
    1 even if the thrown exception type doesn't exactly match the exception type
    given, but merely derives from it. So, "-derived System.Exception" would catch
    every exception in the System.Exception heirarchy.

    The pseudo-register number is optional. If you don't pass a number, SOS will
    use pseudo-register $t1.

    Note that you can use !soe as a shortcut for !StopOnException.

    If you want to test yourself, you can try this sample page:

    <%@ Page Language="C#" %>
    <%@ Import Namespace="System.Data" %>
    <%@ Import Namespace="System.Data.SqlClient" %>
    <%@ Import Namespace="System.IO" %>
    <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "">
    <script runat="server">
        protected void FileNotFoundButton_Click(object sender, EventArgs e)
            //The file does not exists
            StreamReader reader = new StreamReader(@"c:\temp\pippo.txt");
        protected void InvalidOperation_Click(object sender, EventArgs e)
            SqlConnection conn = new SqlConnection("data source=fakeServer; database=fakeDb; integrated security=sspi");
            SqlCommand cmd = new SqlCommand("select * from fakeTable", conn);
            //The connection should be opened!!!
            SqlDataReader dr = cmd.ExecuteReader();
        protected void SqlException_Click(object senter, EventArgs e)
            SqlConnection conn = new SqlConnection("data source=fakeServer; database=fakeDb; integrated security=sspi");
            SqlCommand cmd = new SqlCommand("select * from fakeTable", conn);
            //We open the connection... but the database we're trying to connect to does not exists!!!
            SqlDataReader dr = cmd.ExecuteReader();
    <html xmlns="">
    <head runat="server">
        <form id="form1" runat="server">
                <asp:Button ID="FileNotFoundButton" runat="server" Text="File Not Found" OnClick="FileNotFoundButton_Click" />
                <br />
                <br />
                <asp:Button ID="InvalidOperationButton" runat="server" Text="Invalid Operation" OnClick="InvalidOperation_Click" />
                <br />
                <br />
                <asp:Button ID="SqlConnectionButton" runat="server" Text="Sql Exception" OnClick="SqlException_Click" />

    Let the fun begin

    Run the page, attache Windbg to your worker process, assure you're loading the correct SOS version (use .loadby sos mscorwks) and you'll break into the process; now set the breakpoint, let's say on first chance InvalidOperationExpcetion:

    !soe -create System.InvalidOperationException

    Go back to your browser a click on the FileNotFound button, you'll see something like the following:

    0:015> g
    (814.119c): CLR exception - code e0434f4d (first chance)
    (814.119c): CLR exception - code e0434f4d (first chance)
    (814.119c): CLR exception - code e0434f4d (first chance)
    (814.119c): CLR exception - code e0434f4d (first chance)

    As you can see Windbg tracks all the exceptions but does nothing special since we told it to break only on certain exceptions; see the e0434f4d code? That indicates an unmanaged exception. Why is it there? Well... sooner or later the CLR talks to the underlying OS (to simplify) which tries to open the file we asked for, but the file does not exist... so the OS throws an exception which is bubbled up to the CLR. The same happens for the SqlException but not with DivideByZeroException (this is a special exception I'll cover later).

    But if you click the right button... here we are:

    'System.InvalidOperationException hit'
    First chance exceptions are reported before any exception handling.
    This exception may be expected and handled.
    eax=03f9d7dc ebx=e0434f4d ecx=00000000 edx=00000029 esi=03f9d868 edi=032c86c0
    eip=7c812a5b esp=03f9d7d8 ebp=03f9d82c iopl=0         nv up ei pl nz na po nc
    cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00000202
    7c812a5b 5e              pop     esi

    Now we can have a look at the managed stack with !clrstack:

    0:012> !clrstack
    OS Thread Id: 0x12c4 (12)
    ESP       EIP     
    03f9d8b4 7c812a5b [HelperMethodFrame: 03f9d8b4] 
    03f9d958 653d7691 System.Data.SqlClient.SqlConnection.GetOpenConnection(System.String)
    03f9d964 652f568a System.Data.SqlClient.SqlConnection.ValidateConnectionForExecute(System.String, System.Data.SqlClient.SqlCommand)
    03f9d970 652f0eb4 System.Data.SqlClient.SqlCommand.ValidateCommand(System.String, Boolean)
    03f9d9a0 652f039a System.Data.SqlClient.SqlCommand.RunExecuteReader(System.Data.CommandBehavior, System.Data.SqlClient.RunBehavior, Boolean, System.String, System.Data.Common.DbAsyncResult)
    03f9d9e0 652f0331 System.Data.SqlClient.SqlCommand.RunExecuteReader(System.Data.CommandBehavior, System.Data.SqlClient.RunBehavior, Boolean, System.String)
    03f9d9fc 652eee23 System.Data.SqlClient.SqlCommand.ExecuteReader(System.Data.CommandBehavior, System.String)
    03f9da38 652eebe5 System.Data.SqlClient.SqlCommand.ExecuteReader()
    03f9da6c 048b0954 ASP.default_aspx.InvalidOperation_Click(System.Object, System.EventArgs)
    03f9da8c 6619004e System.Web.UI.WebControls.Button.OnClick(System.EventArgs)
    03f9daa0 6619023c System.Web.UI.WebControls.Button.RaisePostBackEvent(System.String)
    03f9dab4 661901b8 System.Web.UI.WebControls.Button.System.Web.UI.IPostBackEventHandler.RaisePostBackEvent(System.String)
    03f9dab8 6614b47c System.Web.UI.Page.RaisePostBackEvent(System.Web.UI.IPostBackEventHandler, System.String)
    03f9dac0 6614b3d2 System.Web.UI.Page.RaisePostBackEvent(System.Collections.Specialized.NameValueCollection)
    03f9dad0 6614e263 System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)
    03f9dccc 6614d8c3 System.Web.UI.Page.ProcessRequest(Boolean, Boolean)
    03f9dd04 6614d80f System.Web.UI.Page.ProcessRequest()
    03f9dd3c 6614d72f System.Web.UI.Page.ProcessRequestWithNoAssert(System.Web.HttpContext)
    03f9dd44 6614d6c2 System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)
    03f9dd58 048b0186 ASP.default_aspx.ProcessRequest(System.Web.HttpContext)

    Have a look at the objects on this thread with !dumpstackobjects:

    0:012> !dso
    OS Thread Id: 0x12c4 (12)
    ESP/REG  Object   Name
    03f9d848 013ea8c4 System.InvalidOperationException
    03f9d894 013ea8c4 System.InvalidOperationException
    03f9d8a4 013dc4c0 System.String    ExecuteReader
    03f9d8a8 013d9c74 System.Data.ProviderBase.DbConnectionClosedNeverOpened
    03f9d8bc 013ea738 System.Char[]
    03f9d8d8 013ea8c4 System.InvalidOperationException
    03f9d8e0 013d9c74 System.Data.ProviderBase.DbConnectionClosedNeverOpened
    03f9d8e4 013ea68c System.Text.StringBuilder
    03f9d908 013ea90c System.String    System.InvalidOperationException
    03f9d90c 013ea960 System.String    System.InvalidOperationException: ExecuteReader requires an open and available Connection. The connection's current state is closed.
    03f9d910 013ea7a8 System.String    ExecuteReader requires an open and available Connection. The connection's current state is closed.
    03f9d918 013ea7a8 System.String    ExecuteReader requires an open and available Connection. The connection's current state is closed.
    03f9d93c 013ea8c4 System.InvalidOperationException
    03f9d958 013dbf7c System.Data.SqlClient.SqlCommand
    03f9d95c 013dc4c0 System.String    ExecuteReader
    03f9d964 013dbf7c System.Data.SqlClient.SqlCommand
    03f9d96c 013dbf7c System.Data.SqlClient.SqlCommand

    Here we already have some details about the exception, we can dump objects (!do <address>) etc... as if we're analyzing a static dump file.

    If you're hunting a permission problem, the !token -n command is very handy, it dumps the security token of the current thread (which might change over time if you're using impersonation):

    0:003> !token -n
    Thread is not impersonating. Using process token...
    TS Session ID: 0
    User: S-1-5-21-1721254763-462695806-1538882281-2553087 (User: EUROPE\carloc)
     00 S-1-5-21-1721254763-462695806-1538882281-513 ConvertSidToFriendlyName on failed with error code 0x800006fd
        Attributes - Mandatory Default Enabled 
     01 S-1-1-0 (Well Known Group: localhost\Everyone)
        Attributes - Mandatory Default Enabled 
    Primary Group: S-1-5-21-1721254763-462695806-1538882281-513 ConvertSidToFriendlyName on failed with error code 0x800006fd
     00 0x000000017 SeChangeNotifyPrivilege           Attributes - Enabled Default 
     01 0x000000008 SeSecurityPrivilege               Attributes - 
     02 0x000000011 SeBackupPrivilege                 Attributes - 
     03 0x000000012 SeRestorePrivilege                Attributes - 
     04 0x00000000c SeSystemtimePrivilege             Attributes - 
     05 0x000000013 SeShutdownPrivilege               Attributes - 
     06 0x000000018 SeRemoteShutdownPrivilege         Attributes - 
     07 0x000000009 SeTakeOwnershipPrivilege          Attributes - 
     08 0x000000014 SeDebugPrivilege                  Attributes - Enabled 
     09 0x000000016 SeSystemEnvironmentPrivilege      Attributes - 
     10 0x00000000b SeSystemProfilePrivilege          Attributes - 
     11 0x00000000d SeProfileSingleProcessPrivilege   Attributes - 
     12 0x00000000e SeIncreaseBasePriorityPrivilege   Attributes - 
     13 0x00000000a SeLoadDriverPrivilege             Attributes - Enabled 
     14 0x00000000f SeCreatePagefilePrivilege         Attributes - 
     15 0x000000005 SeIncreaseQuotaPrivilege          Attributes - 
     16 0x000000019 SeUndockPrivilege                 Attributes - Enabled 
     17 0x00000001c SeManageVolumePrivilege           Attributes - 
     18 0x00000001d SeImpersonatePrivilege            Attributes - Enabled Default 
     19 0x00000001e SeCreateGlobalPrivilege           Attributes - Enabled Default 
    Auth ID: 0:13927
    Impersonation Level: Anonymous
    TokenType: Primary

    If you just want to have a look at the exceptions thrown by your application without necessarily breaking in the debugger or taking a dump, Windbg can print some tracking information and let the application continue:

    0:006> sxe -c "!pe;!clrstack;gc" clr
    0:006> g
    (14c4.10a4): CLR exception - code e0434f4d (first chance)
    Exception object: 013d8a20
    Exception type: System.IO.FileNotFoundException
    Message: Could not find file 'c:\temp\pippo.txt'.
    InnerException: <none>
    StackTrace (generated):
    StackTraceString: <none>
    HResult: 80070002
    OS Thread Id: 0x10a4 (3)
    ESP       EIP     
    036dd6c8 7c812a5b [HelperMethodFrame: 036dd6c8] 
    036dd76c 7948d83d System.IO.__Error.WinIOError(Int32, System.String)
    036dd798 79395557 System.IO.FileStream.Init(System.String, System.IO.FileMode, System.IO.FileAccess, Int32, Boolean, System.IO.FileShare, Int32, System.IO.FileOptions, SECURITY_ATTRIBUTES, System.String, Boolean)
    036dd890 793983c8 System.IO.FileStream..ctor(System.String, System.IO.FileMode, System.IO.FileAccess, System.IO.FileShare, Int32, System.IO.FileOptions)
    036dd8bc 793a3189 System.IO.StreamReader..ctor(System.String, System.Text.Encoding, Boolean, Int32)
    036dd8e0 79497d71 System.IO.StreamReader..ctor(System.String)
    036dd8f4 048b08ce ASP.default_aspx.FileNotFoundButton_Click(System.Object, System.EventArgs)
    036dd90c 6619004e System.Web.UI.WebControls.Button.OnClick(System.EventArgs)
    036dd920 6619023c System.Web.UI.WebControls.Button.RaisePostBackEvent(System.String)
    036dd934 661901b8 System.Web.UI.WebControls.Button.System.Web.UI.IPostBackEventHandler.RaisePostBackEvent(System.String)
    036dd938 6614b47c System.Web.UI.Page.RaisePostBackEvent(System.Web.UI.IPostBackEventHandler, System.String)
    036dd940 6614b3d2 System.Web.UI.Page.RaisePostBackEvent(System.Collections.Specialized.NameValueCollection)
    036dd950 6614e263 System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)
    036ddb4c 6614d8c3 System.Web.UI.Page.ProcessRequest(Boolean, Boolean)
    036ddb84 6614d80f System.Web.UI.Page.ProcessRequest()
    036ddbbc 6614d72f System.Web.UI.Page.ProcessRequestWithNoAssert(System.Web.HttpContext)
    036ddbc4 6614d6c2 System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)
    036ddbd8 048b0186 ASP.default_aspx.ProcessRequest(System.Web.HttpContext)
    036ddbe4 65fe6bfb System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
    036ddc18 65fe3f51 System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)
    036ddc58 65fe7733 System.Web.HttpApplication+ApplicationStepManager.ResumeSteps(System.Exception)
    036ddca8 65fccbfe System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest(System.Web.HttpContext, System.AsyncCallback, System.Object)
    036ddcc4 65fd19c5 System.Web.HttpRuntime.ProcessRequestInternal(System.Web.HttpWorkerRequest)
    036ddcf8 65fd16b2 System.Web.HttpRuntime.ProcessRequestNoDemand(System.Web.HttpWorkerRequest)
    036ddd04 6600545c System.Web.HttpRuntime.ProcessRequest(System.Web.HttpWorkerRequest)
    036ddd14 033f34cd Microsoft.VisualStudio.WebHost.Request.Process()
    036ddd28 033f31c2 Microsoft.VisualStudio.WebHost.Host.ProcessRequest(Microsoft.VisualStudio.WebHost.Connection)

    Another useful command to break on managed exception is !CheckCurrentException, but you'll likely use it in a Windbg script or adplus configuration file:

    !CheckCurrentException checks if the current exception is the one specified and stores 1 or 0 in the pseudo register supplied (1 = $t1 register).  This is useful when scripting a config file to use with Adplus to get a dump only when a certain type of managed exception happens.  The config file would look like:


        <!-- Configuring ADPlus to log all first chance exceptions -->
        <!-- Will still create full dump for any type of second chance exceptions -->

            <!-- This is for the CLR exception -->
           <Code> clr </Code>
           <Actions1> Log </Actions1>
           <CustomActions1> !cce System.Configuration.ConfigurationException 1; j ($t1 = 1) '.dump /ma /u c:\dumps\mydump.dmp' ; ''  </CustomActions1>
           <ReturnAction1> gn  </ReturnAction1>
           <Actions2> Log;FullDump </Actions2>


    Remarks: This will dump on a System.Configuration.ConfigurationException only.  It will
    make a dump in the c:/dumps directory when that happens.

    !bp instead sets a breakpoint on a managed function:

    !Bp sets a breakpoint on the given function.  It will break even if the function isn't JIT'ed yet

    !Bp <module name> <method name>
    !Bp <module name>!<method name>

    !sos.bp MyApp!MyFunction

    And if you reach an interesting point you can always manually capture a dump with .dump /ma <dumpname>

    What's next?

    Debugging live you can do almost everything you would do with a static dump file, but if you're an hard core debugger you can experiment changing variable values, pointers, memory addresses contents and see the results directly in the application you're attached to, a sort of "live memory programming"... for a small sample have a look at this posts from Roberto to cheat with Minesweeper smile_wink. There is a rumor here in Italy of an Engineer whom while onsite for an urgent problem fixed it directly in memory and then had just to correct and deploy the new binary, but the customer was already happily using his application without further annoyances: true or legend? Well, I know this guy and I tend to thing this is a true story smile_nerd.

    Another interesting and powerful feature of Windbg is the ability to see registers, source code, locals, disassembly etc... (similar to what we have in Visual Studio), this is also available with a static dump file but is especially useful when debugging live... but I guess that will be the subject of another post smile_nerd


    Quote of the day:
    Whenever I hear anyone arguing for slavery, I feel a strong impulse to see it tried on him personally. - Abraham Lincoln
  • Never doubt thy debugger

    Web Garden, AppDomain, statics… when Sql Server is not enough to save your session


    Things have gone a bit crazy lately, we’ve been under a huge workload and the time left for blogging was virtually non existent… but the good thing is that I’ve been working on a few interesting cases I hope I'll have time to blog about, and this is the first one.

    The problem

    The application was running on a Windows 2003 cluster and the well known solution to avoid session problems in a multi server environment is to store user sessions out-of-process, either using a State Server or Sql Server, and in this case we were using Sql Server; this is also the right approach if you are using Web Garden for your application pool and this is the situation we were in (web garden + state on Sql Server). But some session-wide arrays the customer was using were suddenly empty as soon as the customer increased the number of worker processes above 1 for the application pool, which is exactly what is not supposed to happen with <sessionState mode=”SqlServer” … />. At the beginning I thought to a configuration problem on the affected servers but the usual quick check on the metabase shown that was not the case, so the next logical step was to get a repro from the customer and continue from there.

    Well, despite mi attempts I’ve not been able to reproduce the problem but the customer confirmed he could, so were be back again to some machine-specific issue? I was still not convinced and decided to have a closer look to the source code of that repro; the essential part was made by a couple of static arrays the customer was filling on the Page Load event and then binding to a couple of DropDownList controls: changing the selection on those controls (handled by the SelectedIndexChange event) was changing some other UI parts on the page and here is where the customer was getting an IndexOutOfRangeException, clear sign the array was unexpectedly empty.

    Ok, so this time I decided to have a closed look at the internal behavior on that code within Visual Studio, and while I was inspecting some variables in the debugger I suddenly got the exception the customer reported, a Javascript alert which was created within a catch clause.

    This made me think; not that IIS7 (I’m using Windows 2008 on my main workstation) was not respecting the fact that there was a debugger attached to one of its pools (Johan quickly blogged about this issue here), but rather about the fact that due to its health monitoring mechanism IIS7 recycled the application pool I was debugging, and as a consequence the application thrown the exception. But neither the customer reported a process crash not the event log had any relevant entry… Moreover the repro was not even using neither Session() nor Cache() objects, so this could not be a “classical” session lost problem. From my past experience I turned my attention to the static arrays (there is only one “instance” of a static object for the whole application, and if someone changes the array that affects everyone using it), but that was not our case since the customer and I were able to reproduce the problem on a single machine, single user. But static objects have another peculiarity: they are scoped to the AppDomain.

    If you have the detective instinct you might have guessed where I am going from there… The static arrays are “loaded automatically by the .NET Framework common language runtime (CLR) when the program or namespace that contains the class is loaded” (, and the first time a user requested the page the array was filled with some data coming from a database. To avoid useless travels to the database, the arrays were filled only if we were not in a postback (IsPostBack=false, so we are in a GET request); when we post the page back (so we’re in a POST) the initialization code does not run because IsPostBack is true and the code assumes the array is already filled and ready to use, and if we’re lucky everything works fine. This is true until IIS decides (for its own internal philosophical reasons) that our POST will be served by the same w3wp.exe instance which already served our first GET. But… what happens if we’re sent to a new w3wp.exe instance? Remember everything was working fine with 1 process per application pool and the problem appeared only increasing the number of processes per pool (Web Garden >= 2)…

    Well… we will be still issuing a POST, which will be served by a new w3wp.exe instance which will load (for the first time) the CLR and initialize the application (the AppDomain and its static fields), the requested page will run… and IsPostBack will still equals to true. So we’ll not query the database to fill the array, we’ll simply assume the array is valid (and the array object is actually valid, we’re not getting a NullReferenceException, it’s just empty) and we’ll get the IndexOutOfRangeException when trying to read it.

    A bit of theory

    The main subject here is isolation; this post from Chris Brumme (see also this one for more details on AppDomains) explains the very basic of this matter:

    By default, static fields are scoped to AppDomains.  In other words, each AppDomain gets its own copy of all the static fields for the types that are loaded into that AppDomain.  This is independent of whether the code was loaded as domain-neutral or not.  Loading code as domain neutral affects whether we can share the code and certain other runtime structures.  It is not supposed to have any effect other than performance

    Processes are isolated by definition, it’s just the OS architecture which works that way (and not only on Windows, but also on other systems); if you think at how messages are exchanged in the Windows OS, there is a complex infrastructure which works at kernel level to make the communication between processes possible (Windows is a system based on messages, which are sent to windows and processes through low level system APIs), you can understand that isolation is essential for security and stability of the entire OS and for the application running on top of it. If an application could easily read/write data belonging to other processes, it would be a perfect environment for hackers and viruses… we need the opposite (of course with some flexibility left to run our applications). Since an AppDomain is loaded within a given process, and the main purpose of an AppDomain is isolation, e.g. prevent that one application can affect other applications running within the same process, change its data etc…, it’s also clear that when you load an AppDomain within a specific process, it cannot be accessed from AppDomains loaded in other processes (of course unless you want it explicitly, then you can use .NET Remoting to create this kind of communication).

    How should be memory, objects, memory addresses, resources, threads etc… shared between different processes? And if a process throws an exception which affect that shared memory, it will kill all other processes which are using that same shared memory region… this is the opposite of isolation and safety, what we must have an our operating systems. And think to a cluster/NLB environment: how could we share those static objects (again, memory, threads, resources and so on) across different machines? How could a static variable travel across processes and machines?

    So, an AppDomain is specific to a process, static fields are specific to an AppDomain… Sql Server (or any kind of state server) is not the solution for the problem the customer reported; this is rather an design issue. Also the customer enabled Web Garden to have performance benefits and this is actually the idea behind Web Garden, but in specific circumstances; you lose more performance than you gain in 90% of cases and it cannot overcome the system architecture, where again processes are isolated from each other. Web Garden only increases performance in cases where you don't rely on cache and where your application is not CPU intensive; in essence it really has a measurable positive impact on performance is if you have a site like that is mostly static content and completely stateless. In most cases it causes performance to be worse because we have to maintain one cache per process and because of the overhead of having multiple processes.


    So, the best solution for this problem is to not use Web Garden; alternatively, if the customer likes the idea to have multiple processes serving the application pool, then some re-coding is needed to change those static arrays into “simple” private objects and store them in Session() or Cache(), which is perfectly maintained using Sql Server. Bottom line is: does not use statics with Web Garden, or be sure you always check if your data objects are correctly initialized and do not simply assume they are in a good shape because you’re in a postback.



    Quote of the day:
    Never explain: your friends do not need it and your enemies will not believe you anyway. - Elbert Hubbard
Page 1 of 1 (4 items)