• Never doubt thy debugger

    Does hot deployment really means "HOT"?


    Every now and then we get calls from customer having troubles with ASP.NET deployment, usually in cluster/NLB environment but sometimes also in single servers; what they usually have in common is the complexity of the web application, maybe made of dozens (if not hundreds) of pages, controls etc... Typical questions are:

    • I am deploying my new pages and dlls but then suddenly the application is restarted, and my user lose their work and have to login again
    • After the deployment the application is slow at startup
    • In my NLB I randomly get inconsistent behaviors and weird exceptions we never got before
    • We are using a tool to synchronize files and folders, and during the sync for a few seconds we receive a lot of errors telling that some dlls are not available, but after some more seconds everything is running fine again

    There are a few variations to the above, but you got the point. I think this might be due to a misunderstanding (of should I say the documentation is not clear enough?) about deployment capabilities of ASP.NET and what is recommended for complex/full loaded applications; the point is that ASP.NET supports hot deployment meaning that whenever you update a "core" component (the content of the /bin folder, for instance) the runtime finishes serving queued requests and then reload the application to reflect your changes. But this also means that a number of things are happening, (roughly) including recompiling the the new assemblies, re-creating the AppDomain to host the application, go through security checks and load the modules into the AppDomain.

    You can easily imagine that for a complex application those steps might require some time during which we application is not able to respond to incoming user's requests. If we then consider a cluster/NBL environment, where every request from your users might be served by a different server... if you're in the middle of your deployment a user might be "bounced" from a server with the new dlls already installed and another with the old ones still in place, bad thinks will happen then... smile_confused

    A few days ago one of those cases came in, and the customer was reporting a deadlock problem in his w3wp.exe, with the following entry in his event log:

    Event Type:     Warning 
    Event Source:   W3SVC-WP 
    Event Category: None 
    Event ID:  2262 
    Date:      18/10/2007 
    Time:      15:46:10 
    User:      N/A 
    Computer:  <computername> 
    ISAPI 'c:\windows\microsoft.net\framework\v2.0.50727\aspnet_isapi.dll' reported itself as unhealthy for the following reason: 'Deadlock detected'. 
    Event Type:     Warning 
    Event Source:   W3SVC 
    Event Category: None 
    Event ID:  1013 
    Date:      18/10/2007 
    Time:      15:47:43 
    User:      N/A 
    Computer:  <computername> 
    A process serving application pool 'DefaultAppPool' exceeded time limits during shut down. The process id was '5780'. 

    We captured a dump as described in How to generate a dump file when ASP.NET deadlocks in IIS 6.0, and the first thing to note is that almost all the threads in the process were waiting on a stack exactly like the following:

    [HelperMethodFrame: 06f9e968] System.Threading.Monitor.Enter(System.Object) 
    System.Web.Compilation.CompilationLock.GetLock(Boolean ByRef) 
    System.Web.Compilation.BuildManager.GetVPathBuildResultInternal(System.Web.VirtualPath, Boolean, Boolean, Boolean) 
    System.Web.Compilation.BuildManager.GetVPathBuildResultWithNoAssert(System.Web.HttpContext, System.Web.VirtualPath, Boolean, Boolean, Boolean) 
    System.Web.Compilation.BuildManager.GetVirtualPathObjectFactory(System.Web.VirtualPath, System.Web.HttpContext, Boolean, Boolean) 
    System.Web.Compilation.BuildManager.CreateInstanceFromVirtualPath(System.Web.VirtualPath, System.Type, System.Web.HttpContext, Boolean, Boolean) 
    System.Web.UI.PageHandlerFactory.GetHandlerHelper(System.Web.HttpContext, System.String, System.Web.VirtualPath, System.String) 
    System.Web.UI.PageHandlerFactory.System.Web.IHttpHandlerFactory2.GetHandler(System.Web.HttpContext, System.String, System.Web.VirtualPath, System.String) 
    System.Web.HttpApplication.MapHttpHandler(System.Web.HttpContext, System.String, System.Web.VirtualPath, System.String, Boolean) 
    System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef) 
    System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest(System.Web.HttpContext, System.AsyncCallback, System.Object) 
    System.Web.Hosting.ISAPIRuntime.ProcessRequest(IntPtr, Int32) 
    [ContextTransitionFrame: 06f9ee00] 
    [GCFrame: 06f9ee50] 
    [ComMethodFrame: 06f9efa8] 

    As you can see we are in the middle of a massive compilation, most likely triggered by the application update the customer mentioned.

    In the meantime there was one AppDomains which was shutting down because it reached its allowed compilation limit before recycling (again, this sound really like a consequence of the massive compilation going on in the meantime in other threads):

    HttpRuntime 0x143fcc24: 
    _shutDownStack:    at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo) 
       at System.Environment.get_StackTrace() 
       at System.Web.HttpRuntime.ShutdownAppDomain() 
       at System.Web.Hosting.HostingEnvironment.ShutdownThisAppDomainOnce() 
       at System.Web.Hosting.HostingEnvironment.InitiateShutdownWorkItemCallback(Object state) 
       at System.Threading._ThreadPoolWaitCallback.WaitCallback_Context(Object state) 
       at System.Threading.ExecutionContext.runTryCode(Object userData) 
       at System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode code, CleanupCode backoutCode, Object userData) 
       at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) 
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) 
       at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(Object state) 
    _shutDownMessage: Recompilation limit of 15 reached 
    HostingEnvironment caused shutdown 
    _shutdownInProgress: 1 
    _requestQueue: 0x144255a0 
    _appDomainAppPath: C:\Inetpub\wwwroot 
    _appDomainAppId: /LM/W3SVC/235991876/Root 

    Imagine a situation where while the server is still receiving incoming request from the clients, the application's files are being update and the runtime needs to go through the initialization steps: it's a risky situation where it's easy to find yourself stuck in the middle of something, likely with performance and locking problems and eventually with a hang or even worse a deadlock.

    Despite the fact that ASP.NET supports “hot deployment” (i.e. you can override application files and the runtime will automatically load the new version at next request), but this is not recommended in “hot” production environments and in particular in a cluster/NBL configuration, where there is always the risk that a request is served first by the updated server, while the next one might be served by the second server which is not updated yet, causing unpredictable behaviors; in such situations it’s always a good idea to schedule your deployments in a “maintenance window” where you can temporarily take offline the application, update everything you need and put it back online when done (if you’re just updating application’s files should be a matter of a couple of minutes, just the time needed for the transfer and restart the services).

    Depending on your needs you may also want to edit your web.config and increase the number of compilations allowed before recycling the AppDomain by setting a higher value for numRecompilesBeforeAppRestart (I suggest to not set it higher than 50)

        tempDirectory = "" 
        debug = "false" 
        strict = "false" 
        explicit = "true" 
        batch = "true" 
        urlLinePragmas = "false" 
        batchTimeout = "900"  
        maxBatchSize = "1000" 
        maxBatchGeneratedFileSize = "1000" 
        numRecompilesBeforeAppRestart = "15" 
        defaultLanguage = "vb" 
        assemblyPostProcessorType = "" 



    Quote of the day:
    First you're an unknown, then you write one book and you move up to obscurity. - Martin Myers
  • Never doubt thy debugger

    Visual Studio designer, CodeDom and InitializeComponent()


    The code problem

    A few days ago I had the chance to work on a case not concerning ASP.NET or IIS, but rather an interaction with Visual Studio designer with classes from System.CodeDom namespace. The customer was developing a custom control meant for other developers to use it in their projects, and upon adding the control to the form, the former had to automatically modify the some codebehind files for example to add some custom methods and register itself in the InitializeComponent method to add events etc... The problem was that we were able to add new methods to the form, but despite our attempts nothing was added to InitializeComponent smile_thinking

    Initiallize component

    Here's the chunk of code we were using:

    private void AddCode(Form form)
       IDesignerHost host = null;
       host = (IDesignerHost)form.Site.GetService(typeof(IDesignerHost));
       //Add method "Form1_Load" on Form1
       CodeMemberMethod member = new CodeMemberMethod();
       member.Name = "Form1_Load";
       member.Parameters.Add(new CodeParameterDeclarationExpression("System.Object", "sender"));
       member.Parameters.Add(new CodeParameterDeclarationExpression("System.EventArgs", "e"));
       CodeSnippetExpression sn;
       sn = new CodeSnippetExpression("MessageBox.Show(\"Hello world\")");
       member.Attributes = MemberAttributes.Private;
       CodeTypeDeclaration typedecl = (CodeTypeDeclaration)form.Site.GetService(typeof(CodeTypeDeclaration));
       //This code will add the following line to the "InitializeMethod" method
       // this.Load += new System.EventHandler(this.Form1_Load);
       member = new CodeMemberMethod();
       foreach (CodeTypeMember typememb in typedecl.Members)
           if (typememb.Name == "InitializeComponent")
           { member = (CodeMemberMethod)typememb; }
       CodeDelegateCreateExpression createDelegate1;
       createDelegate1 = new CodeDelegateCreateExpression(new CodeTypeReference("System.EventHandler"), new CodeThisReferenceExpression(), "Form1_Load");
       CodeAttachEventStatement attach = new CodeAttachEventStatement(new CodeThisReferenceExpression(), "Load", createDelegate1);
       //Add and remove a label because otherwise the code to add the method seems to stay "inactive,
       //while in this way it works
       Label lbl = (Label)host.CreateComponent(typeof(Label));

    private void InitializeComponent()

    Interestingly beside InitializeComponent Visual Studio was showing the yellow line which means that the code has been changed and still needs to be saved, so something was actually happening... so I tried to add the event in another method and it worked fine; an also adding a MessageBox inside the foeach loop to display LinePragma.FileName and LinePragma.LineNumber showed we were working on the appropriate code file and code line (the InitializeComponent declaration)... So there is a problem with InitializeCompoent itself?

    Well, yes.

    I remember I read about it long time ago when I was getting my hand dirty with the emerging .NET technology (it was back in 2001 I think), but then I moved to web applications and this went somewhere in the back of my mind... the InitializeComponent is a very important method for Visual Studio, which protects it from custom manipulations which might compromise good working of the designer, and this is actually explained in a KB article: Code comments that you add in the InitializeComponent method are lost when you add a control in Visual Studio .NET or in Visual Studio 2005.

    And that was actually the exact situation we where into: for some reason the code was not added as expected unless we were adding and removing a control to the From (a Label in our case) and this simple trick had the effect to somehow activate the code (I guess that way we were calling some specific method otherwise we were not touching), but his also had the side effect to put us in the situation described in the article. And even if our was were working fine, sooner or later we would have stumbled in this kind of Visual Studio protection anyway, because a developer would have certainly added other controls to the form anyway...

    So we had two problems here:

    1. Find a way to add event lines to the Form initialization
    2. Find a way to "refresh" the code without adding/removing fake controls to the Form

    To resolve point one we decided to create an InitializeComponent2 method still inside Form1.Designer.cs (to have it consistent with the default InitializeComponent added by Visual Studio) so we were now able to add our events and custom properties, and add a call to our InitializeComponent2 in Form1 construction, right below the standard InitializeComponent() call, and this way we had our event working as expected smile_regular.

    Chain of Events

    When using design-time controls, it's important to know what happens behind the scenes when you drop a control on your design surface—or for controls such as a Form, what happens when you create a Form that is inherited from another Form you created.

    When an object is opened in the design environment, the class that the object is inherited from (not the newly created class) is constructed by Visual Studio. Remember, a control that is created fires its constructor, which also fires the InitializeComponent() method within the base class's constructor. Once the derived class is constructed, that magically named method within your newly created class, InitializeComponent(), is parsed line-by-line by Visual Studio. The only thing magic about this method is its name. Visual Studio just knows to look for this method.

    In Visual Basic .NET, if Visual Studio can't understand the line of code, the line is removed (a nice way of saying eaten). The Visual Basic team felt it was more important to maintain the design-time environment. The C# team felt it was more important to maintain the code, so if Visual Studio can't parse InitializeComponent() you'll get the text of the exception presented to you. This includes inherited Forms, or controls placed on the design surface of another component class control.

    Any property that is part of your base object is set by the base object and its constructor. When the InitializeComponent() method runs, the values are changed to the values you have set on the property sheet. In Visual Studio, the property sheet is just a graphical representation of the InitializeComponent() method. If your base class performs some sort of functionality in the constructor, be careful; it will be executed in the design environment as well.

    You do have some control over this, however. Any class that derives from the Component class has a property called DesignMode that is set to true when the code is executing within the constructs of the Visual Studio designer. So you have the option to wrap code within an if statement. There's one more trick, however. The DeisgnMode property isn't set to true in the constructor. Remember, there's no real magic here. Visual Studio creates your object as it parses the InitializeComponent() method. Once the object is constructed, Visual Studio keeps track of the objects it creates, and simply says:

    newlyCreatedObject.DesignMode = true

    Next steps

    The argument intrigued me, so I was curious to further explore the capabilities offered by the CodeDom namespace; doing some researches on the Internet you can quite easily find articles which describes how to create a class from crash, create the source file and compile it on the fly to have the resulting assembly and this is quite a standard usage of CodeDome, here are just a couple of references:

    But I've not been able to find good articles about how to customize the current code graph, the one we're currently using (instead of creating a completely new class/component from scratch), so I made some tests myself with conflicting results (something's working the way I like, something don't...). Here's what I've come up with, it's still not perfect (continue reading to find out what's wrong) but even if far from being perfect at least it works:

    private void AddCode(Form form)
        //Hook to the designer
        IDesignerHost host = (IDesignerHost)form.Site.GetService(typeof(IDesignerHost));
        //Add method "Form1_Load" on Form1
        CodeMemberMethod member = new CodeMemberMethod();
        member.Name = "Form1_Load";
        member.Parameters.Add(new CodeParameterDeclarationExpression("System.Object", "sender"));
        member.Parameters.Add(new CodeParameterDeclarationExpression("System.EventArgs", "e"));
        CodeSnippetExpression sn = new CodeSnippetExpression("MessageBox.Show(\"Hello world\")");
        member.Attributes = MemberAttributes.Private;
        CodeTypeDeclaration typedecl = (CodeTypeDeclaration)form.Site.GetService(typeof(CodeTypeDeclaration));
        //Create an InitializeComponent2() method
        member = new CodeMemberMethod();
        member.Name = "InitializeComponent2";
        //Find the constructor and add a call to my "InitializeComponent2()" method
        //CodeConstructor ctor = new CodeConstructor();
        CodeConstructor ctor = null;
        foreach (CodeTypeMember typememb in typedecl.Members)
            if (typememb.Name == ".ctor")
                ctor = (CodeConstructor)typememb;
        CodeMethodInvokeExpression invokeExpression = new CodeMethodInvokeExpression();
        invokeExpression.Method = new CodeMethodReferenceExpression(
            new CodeThisReferenceExpression(), "InitializeComponent2");
        //This code will add the following line to the "InitializeMethod2" method
        // this.Load += new System.EventHandler(this.Form1_Load);
        member = new CodeMemberMethod();
        foreach (CodeTypeMember typememb in typedecl.Members)
            if (typememb.Name == "InitializeComponent2")
                member = (CodeMemberMethod)typememb;
        CodeDelegateCreateExpression createDelegate1 = new CodeDelegateCreateExpression(
            new CodeTypeReference("System.EventHandler"), new CodeThisReferenceExpression(), "Form1_Load");
        CodeAttachEventStatement attachStatement1 = new CodeAttachEventStatement(new CodeThisReferenceExpression(), "Load", createDelegate1);
        //Add and remove a label because otherwise the code to add the method seems to stay "inactive",
        //while in this way it works
        Label lbl = (Label)host.CreateComponent(typeof(Label));

    The debug problem

    Debugging design time controls is not part of my daily job (I've never saw one, in web development), so while working on this case I had to figure out out to debug one of those... At the beginning I just needed to check a couple of variable values so to make things simple I just added a couple of MessageBox.Show() where I needed (like I used to fill my old ASP pages with Response.Write() statements long time ago). As you can guess this is not a practical approach. And at the same time I could not imagine to be the only person facing this problem, so there must be some article outside on the Internet dealing with this issue... and here is it, just in case you need it: Walkthrough: Debugging Custom Windows Forms Controls at Design Time.

    Start external program Essentially what needs to be done is set the control's project as the startup progect, change the debug start action for the control you need to debug, setting it to "Start external program" and pointing it to the Visual Studio executable (default path is C:\Program Files\Microsoft Visual Studio 8\Common7\IDE\devenv.exe), then set a breackpoint where you want to stop in its code and press F5 to start debugging. A new instance of Visual Studio is created, and from it you can open once again the Solution which contains the control you want to debug; now simply do what you need to run the control (add it to a new form for example) and hit the breakpoint (within the first Visual Studio instance, of course). Here we are, now you can step through, inspect variables and do whatever you usually do while debugging. smile_nerd

    Note that the breakpoint icon in the first instance of Visual Studio will display the "breakpoint not hit" icon and tooltip until you cause the code to run, through the second Visual Studio instance

    Breakpoint will not currently be hit

    What's still missing

    To have a well designed component I wanted to be able to create the InitializeComponent2 method where I wanted (possibly in the Designer.cs file to have it together with the standard InitializeComponent created by Visual Studio), remove the "#line" entries in the class constructor to have the code clean and easy to read and avoid the ugly trick to ad/remove a label to the form for activate the code. Unfortunately those are three problems I've still not resolved smile_thinking.

    Creating a new method in Form1.cs is not a big deal, nor is adding some lines of code to it; but problems arise when trying to interact with the class constructor. This is not a protected method as we saw for InitializeComponent(), but no matter what I tried, there are always a few "#line" statements added:

    public Form1()
    #line 17 "C:\Temp\TestForMS\TestForMS\TestForm\Form1.cs"
    #line default
    #line hidden

    Interestingly those nasty "#line" does not appear if we create a new code constructor, I guess that's because in that case we're working with a new object we created and have "full control" on it; for example creating a constructor overload and adding a call to my InitializeComponent2() produces perfectly clean code. Based on this assumption (still to verify, anyway) I tried to replace the class constructor with a new object created for this purpose, but when adding it to the class code graph still produces those lines... smile_eyeroll. Those are C# Preprocessor Directives and luckily the compiler does not complain, so for the moment I had to give up and live with them... smile_sad; but that's just for now, hopefully I'll find a way to get rid of them.

    Ideally I wanted to create my InitializeComponent2() method to the Form1.Designer.cs file, but the default behavior is to add it to the Form1 class (in Form1.cs); the nice thing is that we have a LinePragma property which "Gets or sets the line on which the type member statement occurs", it contains a FileName and LineNumber property to clearly identify where the statement is executed. So I thought to use it to instruct the CodeDom to add my method in the code file I wanted with the following code (and quite a few variations of it):

    //Create an InitializeComponent2() method
    member = new CodeMemberMethod();
    member.Name = "InitializeComponent2";
    foreach (CodeTypeMember typememb in typedecl.Members)
        if (typememb.Name == "InitializeComponent")
            member.LinePragma = new CodeLinePragma();
            member.LinePragma.FileName = typememb.LinePragma.FileName;

    Despite my attempts, InitializeComponent2() is always created in Form1.cs... still something to look into smile_eyeroll.

    Finally, the ugly workaround (or should I say "alternative solution"? smile_tongue) of adding and removing the fake label to the form to activate the changes made to the code; I guess this invokes a sort of refresh mechanism, so I tried to dig into those calls with Reflector because I wanted to find the property to set or the method to call to activate the refresh without adding the label, but again with no luck... third (and hopefully last) point to clarify.


    CodeDom offers some interesting capabilities to control developers but also has some limitations, here's a list (likely not complete):

    • CodeCompile unit does not have space for using directives or namespace members, so they are placed now into first default namespace
    • using alias directive - no support found
    • nested namespaces - no support found (so parser is flattening namespace hierarchy)
    • variable declaration list (int i,j,k;) - no support - transformed to individual var declarations
    • pointer type - no support found
    • jagged array type (array of arrays) - CSharpCodeProvider reverses order of ranks
    • params keyword - not supported - param is omitted in parsing and param is then an ordinary array type param
    • private modifier on nested delegate is not shown by CSharpCodeProvider (all other nested types works fine)
    • unsafe modifier - no support found
    • readonly modifier - no support found
    • volatile modifier - no support found
    • explicit interface implementation - not implemented yet (I think this can be done)
    • add and remove accessors for Event - no support found
    • virtual and override modifiers do not work in CSharpCodeProvider for events
    • Operator members and Destructors - no support found
    • Expressions - no unary expressions(operations), only one dimension array, some operators not supported
    • Attribute targets - no support found
    • Attributes on accessor - no support found
    • If CompileUnit contains custom attributes in global scope, CSharpCodeProvider prints then before global using directives (because using has to be in the first namespace)

    Back to the sample treated in this post, I'll try to resolve those 3 open problems I mentioned above, but if any of you have some details to add (or a solution to share smile_wink) please leave a comment, I'll be happy to correct the post! smile_nerd



    Quote of the day:
    Addresses are given to us to conceal our whereabouts. - Saki
  • Never doubt thy debugger

    HTTP error 406 with .NET Framework 3.0


    I got a couple of cases about this problem recently... Imagine this scenario: you install the .NET Framework 3.0 on your client, and then browse an ASP.NET based web site; you get a 406 HTTP return code from the web server, which means "Client browser does not accept the MIME type of the requested page" (see IIS status codes). Uninstalling the .NET Framework 3.0 corrects the problem, and you're finally able to successfully browse the site.

    The problem proved itself in two different ways and apparently for two different reasons, but the underlying cause was actually the same. The 406 return code also means that any of the configuration limits has been reached, and digging into IIS logs we found that the problem was actually due to the length of the "Accept" header which has a limit of 256 bytes. Installing the .NET Framework 3.0 you receive support for a few additional file formats, here is how it looks the Accept header on Windows Vista (where the .NET Framework 3.0 is preinstalled):

    Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/x-ms-application, 
    application/vnd.ms-xpsdocument, application/xaml+xml, application/x-ms-xbap, application/vnd.ms-excel, 
    application/vnd.ms-powerpoint, application/msword, application/x-shockwave-flash, */*

    In this case there was a custom ISAPI extension which was actually filtering the incoming headers, and was blocking our request because it exceeded the configured length: to resolve the problem the customer simply increased the size limit configured within that specific ISAPI to let the requests to through. Anyway another solution was to reduce the number of Accept types, just the standard ones.

    The second customer was using a third party web server which we then discovered was able to accept a relatively short User-Agent string; the customer did not have any "power" on the web server to reconfigure it, so we only possibility left was reduce the header as much as possible on the client. The nice thing is that the User-Agent string is stored in a couple of registry key and even if that's not an ideal solution it was the only possibility we had, we decided to manually tweak those values; the first we looked at was HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\5.0\User Agent\Post Platform. On the right side you have a list of strings added to the User-Agent string which will be appended after the platform string (as the Post Platform name suggests); in case you need to add a custom string before the platform string, you can add a Pre Platform key. Just in case you're curious (or you're really desperate trying to reduce your User-Agent as much as possible) the platform part of the string is stored in HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings > User Agent. For further infos have a look at Understanding User-Agent strings and Internet Explorer 7 User-Agent string.

    If you need to tweak the User-Agent string be very careful and test it closely before deploying into production, since you might break client-side scripts which needs to identify the specific browser or platform version you're using; ASP.NET also relies on the User-Agent string to identify the browser and produce markup compliant with that specific platform (does it support javascript? embedding videos? is a mobile browser? does it support CSS? etc...). See Determining Browser Capabilities in ASP.NET and Control adapters.


    Quote of the day:
    Your manuscript is both good and original, but the part that is good is not original and the part that is original is not good. - Samuel Johnson
  • Never doubt thy debugger

    Procmon on x64? "/?" will help :-)


    Maybe some of you already know this trick, but if you don't... here it is smile_regular

    I've been running a Vista x64 as my main machine in office for a few months now, and when I had to analyze a Process Monitor trace received from a customer, but if the log was coming from a x86 machine (that's still the most common for customer calls we see today) I was not able to open it on my desktop, and always had to rely on my laptop (where I run Windows XP Pro) or on my second desktop (Windows 2003).

    Process Monitor invalid argument

    I took this for granted for a while, but then this morning I thought to have a look at the command line options (procmon /?) and got a nice surprise:

    Process Monitor usage

    Tried it, and (of course) I was finally able to open the 32bit trace on my 64bit machine.. smile_nerd



    Quote of the day:
    It's amazing that the amount of news that happens in the world every day always just exactly fits the newspaper. - Jerry Seinfeld
  • Never doubt thy debugger

    A Trappable error C0000005 occured in an external object, the script cannot continue running


    This is an Access Violation exception which can have different causes (MDAC bad installation, third party components such as Oracle dlls which Oracle has a fix for etc...), but in the specific case I had some time ago this was due to a corrupted Windows Scripting Host engine. This was a classic ASP application and it was crashing randomly; we captured a dump with adplus (4 dumps, actually) and the stack of the faulting thread was always the same:

    0:020> kpL2000
    ChildEBP RetAddr  
    04f5f5c8 7346520b vbscript!VarStack::SetMasterSp(long cvar = 6, class VAR ** ppvarSp = 0x04f5f6b8)+0xa3
    04f5f5e4 7346492e vbscript!CScriptRuntime::Init(class CSession * psess = 0x00000000, class FncInfo * pfin = 0x04654e28, class VAR * pvarThis = 0x00000000, int cvar = 0, class VAR * prgvar = 0x00000000, unsigned long grfcall = 0)+0x265
    04f5f6f0 73464cd2 vbscript!CScriptEntryPoint::Call(class VAR * pvarRes = 0x00000000, int cvar = 0, class VAR * prgvar = 0x00000000, class VAR * pvarThis = 0x00000000, unsigned long grfCall = 0)+0x4a
    04f5f748 73465522 vbscript!CSession::Execute(class IEntryPoint * pep = 0x04654e20, struct tagVARIANT * pvarRes = 0x00000000, int cvar = 0, struct tagVARIANT * prgvar = 0x00000000, struct tagVARIANT * pvarThis = 0x00000000, unsigned long grfscr = 0)+0xb4
    04f5f798 7346189b vbscript!COleScript::ExecutePendingScripts(struct tagVARIANT * pvarRes = 0x00000000, struct tagEXCEPINFO * pei = 0x00000000)+0x13e
    04f5f7b4 709e2f5a vbscript!COleScript::SetScriptState(tagSCRIPTSTATE ss = SCRIPTSTATE_STARTED (1))+0x150
    04f5f7e0 709e2f1a asp!CActiveScriptEngine::TryCall(unsigned short * strEntryPoint = 0x00000000)+0x19
    04f5f81c 709f412b asp!CActiveScriptEngine::Call(unsigned short * strEntryPoint = 0x00000000)+0x31
    04f5f824 709f40c8 asp!CActiveScriptEngine::MakeEngineRunnable(void)+0x7
    04f5fa80 709f4231 asp!AllocAndLoadEngines(class CHitObj * pHitObj = 0x01db20f0, class CTemplate * pTemplate = 0x022b1fe8, struct ActiveEngineInfo * pEngineInfo = 0x04f5fb0c, class CScriptingNamespace * pScriptingNamespace = 0x020f2580, int fGlobalAsa = 1)+0x449
    04f5fae0 709f4418 asp!ExecuteGlobal(class CHitObj * pHitObj = 0x01db20f0, class CIntrinsicObjects * intrinsics = 0x04f5fb68, struct ActiveEngineInfo * pEngineInfo = 0x04f5fb0c)+0x184
    04f5fb44 709e2a4d asp!Execute(class CTemplate * pTemplate = 0x022b1e90, class CHitObj * pHitObj = 0x01db20f0, class CIntrinsicObjects * intrinsics = 0x04f5fb68, int fChild = 0)+0xb7
    04f5fb98 709e271a asp!CHitObj::ViperAsyncCallback(int * pfRePosted = 0x00000000)+0x3e8
    04f5fbb4 75bd72a5 asp!CViperAsyncRequest::OnCall(void)+0x92
    04f5fbd0 7770f0eb comsvcs!CSTAActivityWork::STAActivityWorkHelper(void * pvoid = 0x05068f38)+0x32
    04f5fc1c 7770fb38 ole32!EnterForCallback+0xc4
    04f5fd7c 77710042 ole32!SwitchForCallback+0x1a3
    04f5fda8 77694098 ole32!PerformCallback+0x54
    04f5fe40 777127fd ole32!CObjectContext::InternalContextCallback+0x159
    04f5fe60 75bd7649 ole32!CObjectContext::DoCallback+0x1c
    04f5fecc 75bd79a5 comsvcs!CSTAActivityWork::DoWork(unsigned long * pdwWorkCount = 0x0012ced0)+0x12d
    04f5fee4 75bd833e comsvcs!CSTAThread::DoWork(class CSTAWork * pWork = 0x05068f38)+0x18
    04f5ff04 75bd878a comsvcs!CSTAThread::ProcessQueueWork(void)+0x37
    04f5ff84 77bcb530 comsvcs!CSTAThread::WorkerLoop(void * pv = 0x0012ceb0)+0x17c
    04f5ffb8 77e6608b msvcrt!_threadstartex(void * ptd = 0x010aebe0)+0x74
    04f5ffec 00000000 kernel32!BaseThreadStart+0x34

    The call stack shows the VBScript engine is called to run the ASP page, and is crashing just after the initialization; I'm not completely sure if this stack is specific to this problem or if we would have something similar also for the other known causes (MDAC and Oracle dlls) because unfortunately I've not had the chance to review those dumps, so just keep in mind this as a possible solution and not the only one...

    Anyway here we fixed the problem reinstalling the Windows Scripting Host engine for Windows 2003 (version 5.7, as I'm writing).




    Quote of the day:
    After I'm dead I'd rather have people ask why I have no monument than why I have one. - Cato the Elder
  • Never doubt thy debugger

    .NET Runtime 2.0 Error, Event id:5000


    This error message was reported by a customer last week; every day, randomly, his application pools (on two servers in NLB) were restarted during business hours without an apparent reason, and of course users were complaining because of lost sessions etc...; the following entry was added to the Application event log:

    Event Type: Error
    Event Source: .NET Runtime 2.0 Error Reporting
    Event Category: None
    Event ID: 5000
    Date: 9/28/2005
    Time: 3:18:02 PM
    User: N/A
    Computer: IIS-SERVER
    EventType clr20r3, P1 w3wp.exe, P2 6.0.3790.1830, P3 42435be1, P4 app_web_7437ep-9, P5, P6 433b1670, P7 9, P8 a, P9 system.exception, P10 NIL

    The first thing to do when we have a specific error message is to search the Knowledge Base for known issues/hotfixes etc..., and was quite easy to find the article Unhandled exceptions cause ASP.NET-based applications to unexpectedly quit in the .NET Framework 2.0 which I already used more than once in the past to resolve similar issues; I usually suggest the customer to first apply the solution described in "Method 2":

    Method 2

    Change the unhandled exception policy back to the default behavior that occurs in the .NET Framework 1.1 and in the .NET Framework 1.0.
    Note We do not recommend that you change the default behavior. If you ignore exceptions, the application may leak resources and abandon locks.
    To enable this default behavior, add the following code to the Aspnet.config file that is located in the following folder:


            <legacyUnhandledExceptionPolicy enabled="true" />

    That is to resolve the problem immediately and have the application running smoothly (at least smoother than before), while the second important step I always suggest is review application's code to spot places where an exception might throw exceptions outside of a try...catch block and correct it; the bottom line is that is not safe ignoring exceptions and continue execution "hoping" that everything goes right because good old Mr. Murphy and his law are always there, watching carefully on you! smile_teeth. Anyway this time the customer had already available a couple of crash dumps captured with adplus, so I just needed some analysis to have a better understanding or what exactly was causing the error... Easy, right? smile_nerd

    Ok, so the first thing I usually do when I'm in Windbg (and the environment is ready to go) is running some standard commands to get a general idea about the state of the process (threads, stack, heap status, exceptions and so on), but in this case I was interested in finding the root cause of a specific exception, and opening the dump I already had this interesting message:

    This dump file has an exception of interest stored in it.
    The stored exception information can be accessed via .ecxr.
    (eb4.e40): CLR exception - code e0434f4d (first/second chance not available)
    CLR exception type: System.FormatException
        "Input string was not in a correct format."

    Let's have a look at the exception objects still on the heap:

    Number of exceptions of this type: 2 Exception object: 1c10f964 Exception type: System.FormatException Message: Input string was not in a correct format. InnerException: <none> StackTrace (generated): SP IP Function 09E0ECA4 793A84DB System.Text.StringBuilder.FormatError() 09E0ECAC 79604D81 System.Text.StringBuilder.AppendFormat(System.IFormatProvider, System.String, System.Object[]) 09E0ECF0 79350B9E System.String.Format(System.IFormatProvider, System.String, System.Object[]) 09E0ED0C 793A6704 System.String.Format(System.String, System.Object) 09E0ED1C 01ABE4A5 MyApp.Class.Identity.Pi2.Sp.Frmx.CSessionLogin.CleanupTicket(System.DateTime, Int32, Boolean, System.Collections.ArrayList) 09E0ED90 01ABE306 MyApp.Class.Identity.Pi2.Sp.Frmx.CAuthentication.Cleanup(System.Object) 09E0EDEC 793D9C1A System.Threading._TimerCallback.TimerCallback_Context(System.Object) 09E0EDF0 793683DD System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) 09E0EE08 793D9C7F System.Threading._TimerCallback.PerformTimerCallback(System.Object)

    The exception stack shows three calls involving a TimerCallback, then a couple of custom methods and finally a few string manipulations (by the way, just in case you don't know, the stack must be read from bottom to top); the exception is coding from a StringBuilder object, while the runtime was trying to format a string but for some reason failed. This string manipulation is happening "behind the scenes", the runtime uses those objects as a consequence of what we're doing in our custom methods, so if the StringBuilder is failing is likely because we passed it some wrong arguments, or asked him to do something not allowed.

    Ok, so let's have a look at those two custom methods; we can extract the IL code by means of the !u command and the IP (Instruction Pointer) address:

    0:020> !u 01ABE4A5 
    Normal JIT generated code
    MyApp.Class.Identity.Pi2.Sp.Frmx.CSessionLogin.CleanupTicket(System.DateTime, Int32, Boolean, System.Collections.ArrayList)
    Begin 01abe3f0, size 31d
    01abe3f0 55              push    ebp
    01abe3f1 8bec            mov     ebp,esp
    01abe3f3 57              push    edi
    01abe3f4 56              push    esi
    01abe3f5 53              push    ebx
    01abe3f6 83ec50          sub     esp,50h
    01abe3f9 33c0            xor     eax,eax
    01abe45b f30f7e00        movq    xmm0,mmword ptr [eax]
    01abe45f 660fd60424      movq    mmword ptr [esp],xmm0
    01abe464 e8e7b88e77      call    mscorlib_ni!System.DateTime.Compare(System.DateTime, System.DateTime) (793a9d50)
    01abe469 8bf0            mov     esi,eax
    01abe46b 85f6            test    esi,esi
    01abe46d 7e41            jle     MyApp_Class_Identity_Pi2_Sp_Frmx!MyApp.Class.Identity.Pi2.Sp.Frmx.CSessionLogin.CleanupTicket(System.DateTime, Int32, Boolean, System.Collections.ArrayList)+0xc0 (01abe4b0)
    01abe46f 8b05e031b507    mov     eax,dword ptr ds:[7B531E0h]
    01abe475 8945a8          mov     dword ptr [ebp-58h],eax
    01abe478 b9f05e1079      mov     ecx,offset mscorlib_ni+0x45ef0 (79105ef0)
    01abe47d e89a3bf2ff      call    019e201c (JitHelp: CORINFO_HELP_NEWSFAST)
    01abe482 8bd8            mov     ebx,eax
    01abe484 8d7b04          lea     edi,[ebx+4]
    01abe487 3b0f            cmp     ecx,dword ptr [edi]
    01abe489 8b75bc          mov     esi,dword ptr [ebp-44h]
    01abe48c 83c654          add     esi,54h
    01abe48f 3b0e            cmp     ecx,dword ptr [esi]
    01abe491 f30f7e06        movq    xmm0,mmword ptr [esi]
    01abe495 660fd607        movq    mmword ptr [edi],xmm0
    01abe499 8bf3            mov     esi,ebx
    01abe49b 8bd6            mov     edx,esi
    01abe49d 8b4da8          mov     ecx,dword ptr [ebp-58h]
    01abe4a0 e833828e77      call    mscorlib_ni!System.String.Format(System.String, System.Object) (793a66d8)
    >>> 01abe4a5 8bf0            mov     esi,eax
    01abe4a7 8975b8          mov     dword ptr [ebp-48h],esi
    01abe4aa 90              nop
    01abe4ab e99f000000      jmp     MyApp_Class_Identity_Pi2_Sp_Frmx!MyApp.Class.Identity.Pi2.Sp.Frmx.CSessionLogin.CleanupTicket(System.DateTime, Int32, Boolean, System.Collections.ArrayList)+0x15f (01abe54f)
    01abe4b0 8d7dc8          lea     edi,[ebp-38h]
    01abe4b3 8d7510          lea     esi,[ebp+10h]

    We can continue disassemble the subsequent calls, here is an excerpt of StringBuilder.AppendFormat:

    79350daa 83fe7b          cmp     esi,7Bh
    79350dad 0f84d33f2b00    je      mscorlib_ni!System.Text.StringBuilder.AppendFormat(System.IFormatProvider, System.String, System.Object[])+0x2b41ce (79604d86)
    79350db3 83fe7d          cmp     esi,7Dh
    79350db6 751f            jne     mscorlib_ni!System.Text.StringBuilder.AppendFormat(System.IFormatProvider, System.String, System.Object[])+0x21f (79350dd7)
    79350db8 3b3c24          cmp     edi,dword ptr [esp]
    79350dbb 7d15            jge     mscorlib_ni!System.Text.StringBuilder.AppendFormat(System.IFormatProvider, System.String, System.Object[])+0x21a (79350dd2)
    79350dbd 3b7b04          cmp     edi,dword ptr [ebx+4]
    79350dc0 0f8326402b00    jae     mscorlib_ni!System.Text.StringBuilder.AppendFormat(System.IFormatProvider, System.String, System.Object[])+0x2b4234 (79604dec)
    79350dc6 66837c7b087d    cmp     word ptr [ebx+edi*2+8],7Dh
    79350dcc 0f84d63f2b00    je      mscorlib_ni!System.Text.StringBuilder.AppendFormat(System.IFormatProvider, System.String, System.Object[])+0x2b41f0 (79604da8)
    79350dd2 83c7ff          add     edi,0FFFFFFFFh
    79350dd5 eb15            jmp     mscorlib_ni!System.Text.StringBuilder.AppendFormat(System.IFormatProvider, System.String, System.Object[])+0x234 (79350dec)
    79350dd7 8bc5            mov     eax,ebp
    79350e74 8d4ade          lea     ecx,[edx-22h]
    79350e77 e84c2cb200      call    mscorwks!JIT_Writeable_Thunks_Buf+0x198 (79e73ac8) (JitHelp: CORINFO_HELP_GETSHARED_GCSTATIC_BASE)
    79350e7c 8b6810          mov     ebp,dword ptr [eax+10h]
    79350e7f 8b44240c        mov     eax,dword ptr [esp+0Ch]
    79350e83 2b4508          sub     eax,dword ptr [ebp+8]
    79350e86 89442410        mov     dword ptr [esp+10h],eax
    79350e8a 837c240800      cmp     dword ptr [esp+8],0
    79350e8f 7508            jne     mscorlib_ni!System.Text.StringBuilder.AppendFormat(System.IFormatProvider, System.String, System.Object[])+0x2e1 (79350e99)
    79350e91 85c0            test    eax,eax
    79350e93 0f8f3c3f2b00    jg      mscorlib_ni!System.Text.StringBuilder.AppendFormat(System.IFormatProvider, System.String, System.Object[])+0x2b421d (79604dd5)
    79350e99 8bd5            mov     edx,ebp
    79350e9b 8b4c2414        mov     ecx,dword ptr [esp+14h]
    79350e9f e85ca3ffff      call    mscorlib_ni!System.Text.StringBuilder.Append(System.String) (7934b200)
    79350ea4 837c240800      cmp     dword ptr [esp+8],0
    79350ea9 0f847afdffff    je      mscorlib_ni!System.Text.StringBuilder.AppendFormat(System.IFormatProvider, System.String, System.Object[])+0x71 (79350c29)
    79350eaf 837c241000      cmp     dword ptr [esp+10h],0
    79350eb4 0f8e6ffdffff    jle     mscorlib_ni!System.Text.StringBuilder.AppendFormat(System.IFormatProvider, System.String, System.Object[])+0x71 (79350c29)
    79350eba ff742410        push    dword ptr [esp+10h]
    79350ebe 8b4c2418        mov     ecx,dword ptr [esp+18h]
    79350ec2 ba20000000      mov     edx,20h
    79350ec7 e898670500      call    mscorlib_ni!System.Text.StringBuilder.Append(Char, Int32) (793a7664)
    79350ecc e958fdffff      jmp     mscorlib_ni!System.Text.StringBuilder.AppendFormat(System.IFormatProvider, System.String, System.Object[])+0x71 (79350c29)
    79350ed1 8b442414        mov     eax,dword ptr [esp+14h]
    79350ed5 83c428          add     esp,28h
    79350f92 8bf9            mov     edi,ecx
    79350f94 8bf2            mov     esi,edx
    79350f96 b96cfe0f79      mov     ecx,offset mscorlib_ni+0x3fe6c (790ffe6c)
    79350f9b e8cb75b300      call    mscorwks!RuntimeTypeHandle::GetRuntimeType (79e8856b)
    79350fa0 3bc6            cmp     eax,esi
    79350fa2 750a            jne     mscorlib_ni!System.Globalization.CultureInfo.GetFormat(System.Type)+0x1e (79350fae)
    79350fa4 8bcf            mov     ecx,edi
    79350fa6 8b01            mov     eax,dword ptr [ecx]
    79350fa8 ff5070          call    dword ptr [eax+70h]
    79350fab 5e              pop     esi
    79350fac 5f              pop     edi
    79350fad c3              ret
    79350fae b9f0ff0f79      mov     ecx,offset mscorlib_ni+0x3fff0 (790ffff0)
    79350fb3 e8b375b300      call    mscorwks!RuntimeTypeHandle::GetRuntimeType (79e8856b)
    79350fb8 3bc6            cmp     eax,esi
    79350fba 0f84e43e2b00    je      mscorlib_ni!System.Globalization.CultureInfo.GetFormat(System.Type)+0x2b3f14 (79604ea4)
    79350fc0 33c0            xor     eax,eax
    79350fc2 5e              pop     esi
    79350fe4 85ed            test    ebp,ebp
    79350fe6 0f84cc3e2b00    je      mscorlib_ni!System.Text.StringBuilder.Append(Char[], Int32, Int32)+0x2b3ee8 (79604eb8)
    79350fec 85f6            test    esi,esi
    79350fee 750c            jne     mscorlib_ni!System.Text.StringBuilder.Append(Char[], Int32, Int32)+0x2c (79350ffc)
    79350ff0 8bc3            mov     eax,ebx

    From this latest chunk of IL we can see that we're inside a loop (note the recurrent StringBuilder.AppendFormat calls) and we're very likely dealing with dates (note the System.Globalization.CultureInfo.GetFormat calls); the next call is where the runtime is throwing the exception:

    0:020> !u 793A84DB 
    preJIT generated code
    Begin 793a84a4, size 38
    793a84a4 50              push    eax
    793a84a5 b9b8201279      mov     ecx,offset mscorlib_ni+0x620b8 (791220b8)
    793a84aa e8e1b5ac00      call    mscorwks!JIT_Writeable_Thunks_Buf+0x160 (79e73a90) (JitHelp: CORINFO_HELP_NEWSFAST)
    793a84af 890424          mov     dword ptr [esp],eax
    793a84b2 8b1500e06879    mov     edx,dword ptr [mscorlib_ni+0x5ce000 (7968e000)]
    793a84b8 b9791b0070      mov     ecx,70001B79h
    793a84bd e82ab8ce00      call    mscorwks!JIT_StrCns (7a093cec)
    793a84c2 8bc8            mov     ecx,eax
    793a84c4 e81229b800      call    mscorwks!GetResourceFromDefault (79f2addb)
    793a84c9 8bd0            mov     edx,eax
    793a84cb 8b0c24          mov     ecx,dword ptr [esp]
    793a84ce e81dc10100      call    mscorlib_ni!System.FormatException..ctor(System.String) (793c45f0)
    793a84d3 8b0c24          mov     ecx,dword ptr [esp]
    793a84d6 e8f1bfce00      call    mscorwks!JIT_Throw (7a0944cc)
    >>> 793a84db cc              int     3

    The next step would be to have a look at the source code of the component, but I'm not the developer and I don't have it at hand... of course I could have called the customer to ask for the project, but why bother him when we can extract the modules directly from the dump? But first, we need to know where the method we're investigating is contained. The !ip2md (Instrutction Pointer to Method Definition) comes at hand; let's dump the CleanupTicket IP:

    0:020> !ip2md 01ABE4A5 
    MethodDesc: 05ecfb20
    Method Name: MyApp.Class.Identity.Pi2.Sp.Frmx.CSessionLogin.CleanupTicket(System.DateTime, Int32, Boolean, System.Collections.ArrayList)
    Class: 0668e38c
    MethodTable: 05ecfbac
    mdToken: 06000017
    Module: 05ec6508
    IsJitted: yes
    m_CodeOrIL: 01abe3f0

    Now we can dump the method table with !dumpmt to find out the name of the dll:

    0:020> !dumpmt 05ecfbac
    EEClass: 0668e38c
    Module: 05ec6508
    Name: MyApp.Class.Identity.Pi2.Sp.Frmx.CSessionLogin
    mdToken: 02000006  (C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\test\8b453812\8f89c492\assembly\dl3\4a6a6dc7\0076b989_27f1c701\MyApp.Class.Identity.Pi2.Sp.Frmx.DLL)
    BaseSize: 0x60
    ComponentSize: 0x0
    Number of IFaces in IFaceMap: 0
    Slots in VTable: 25

    Finally with the lm command we can get the complete details about this module, included the chunk of memory (initial and final addresses) where it is loaded. Note that if the file name contains dots (".") you must replace them with underscores ("_"), and you don't have to use the extension; for example System.Web.dll will be translated as System_Web:

    0:020> lmv mMyApp_Class_Identity_Pi2_Sp_Frmx
    start    end        module name
    068b0000 068be000   MyApp_Class_Identity_Pi2_Sp_Frmx   (deferred)             
        Image path: C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\test\8b453812\8f89c492\assembly\dl3\4a6a6dc7\0076b989_27f1c701\MyApp.Class.Identity.Pi2.Sp.Frmx.DLL
        Image name: MyApp.Class.Identity.Pi2.Sp.Frmx.DLL
        Using CLR debugging support for all symbols
        Has CLR image header, track-debug-data flag not set
        Timestamp:        Fri Aug 24 18:40:06 2007 (46CF09E6)
        CheckSum:         00000000
        ImageSize:        0000E000
        File version:     0.12.2792.0
        Product version:  0.12.2792.0
        File flags:       0 (Mask 3F)
        File OS:          4 Unknown Win32
        File type:        2.0 Dll
        File date:        00000000.00000000
        Translations:     0000.04b0
        InternalName:     MyApp.Class.Identity.Pi2.Sp.Frmx.dll
        OriginalFilename: MyApp.Class.Identity.Pi2.Sp.Frmx.dll
        ProductVersion:   0.12.2792.0
        FileVersion:      0.12.2792.0

    Finally... let's use !savemodule <starting address> <output name> to extract the module from the dump:

    0:020> !savemodule 068b0000 C:\temp\MyApp.Class.Identity.Pi2.Sp.Frmx.DLL
    4 sections in file
    section 0 - VA=2000, VASize=4d54, FileAddr=1000, FileSize=5000
    section 1 - VA=8000, VASize=a3, FileAddr=6000, FileSize=1000
    section 2 - VA=a000, VASize=8a8, FileAddr=7000, FileSize=1000
    section 3 - VA=c000, VASize=c, FileAddr=8000, FileSize=1000
    Wow, that sounds long and complicated, but once you get used to it it's just a matter of 20 seconds clock

    Ok, now that we have our module we can use Reflector to finally have a look at the source code of our CleanupTicket method, here it is:

    internal void CleanupTicket(DateTime dt, int iToMin, bool bNoCheck, ArrayList vArrayList)
        string str;
        if (!bNoCheck)
            if (DateTime.Compare(dt, this.m_dtE) > 0)
                str = string.Format("LEASE to expired at {0:yyy/MM/dd-HH:mm:ss:fff", this.m_dtE);
            else if (DateTime.Compare(dt, this.m_dtLastAccess.AddMinutes((double) iToMin)) > 0)
                str = string.Format("IDLE to timout expired at {0:yyy/MM/dd-HH:mm:ss:fff} + {1} min", this.m_dtLastAccess, 60);
        if (StringType.StrCmp(str, "", false) != 0)
            bfLogging.WriteFmt("CSessionLogin::CleanupTicket", "", "Ticket {0} is added to the logout bag. {1}", new object[] { this.m_strGuid, str });
        if (bNoCheck | (StringType.StrCmp(str, "", false) != 0))
        if (this.m_vHashtableCSessionLoginPseudoSp != null)
            foreach (CSessionLogin login in this.m_vHashtableCSessionLoginPseudoSp.Values)
                login.CleanupTicket(dt, iToMin, bNoCheck | (StringType.StrCmp(str, "", false) != 0), vArrayList);

    As you can see we are formatting some strings and we have a loop as we saw in the IL code, and here (and in the previous method, Cleanup, which I've not reported here just to save some space) we don't have any try...catch blocks; this means that if for any reason an exception is thrown in this code (or if we pass some wrong argument which can't be successfully converted to the underlying runtime objects like the StringBuilder) the exception will not be trapped, and as explained in the KB article this will eventually recycle the worker process smile_nerd

    In this case the developer reviewed the code to add exception handling on the sensitive lines, and they finally got rid of those unwanted worker process recycles. By the way, my colleague Tess ha a very similar post on her blog here.


    Quote of the day:
    A good novel tells us the truth about its hero; but a bad novel tells us the truth about its author. - G. K. Chesterton

  • Never doubt thy debugger

    Take control over Windbg


    If you're like me and most of my colleagues in Customer Service and Support, you'll probably spend a good deal of time in your debugger, which for me is WinDbg. So, since I'm in love with keyboard shortcuts (a colleague of mine likes kidding me stealing my mouse since I hardly use it, and every now and then uses me as his personal "shortcut reference" smile_regular) and always searching for "productivity tips" to optimize my workflow. Windbg has a few of them, here is what I use.

    Automate WinDbg startup

    I first discovered this in this post from Tess and then customized it to match my needs, especially on my Vista x64 machine; basically this is a .reg file set WinDbg as the default application for .dmp files, and adds a few entries to the right click menu to chose different options. I also have to say that on my x64 machine I actually have two separate folders for WinDbg 32bit and 64bit, so that depending on the dump I'm debugging I can use the right tool:

    Windows Registry Editor Version 5.00
    @="c:\\debuggers x86\\cdb.exe"
    @="Debug This Dump for .NET 1.1 x86"
    @="\"C:\\debuggers x86\\windbg\" -z \"%1\" -c \"$<c:\\debuggers x86\\commands_1.txt\""
    @="c:\\debuggers x86\\cdb.exe"
    @="Debug This Dump for .NET 2.0 x86"
    @="\"C:\\debuggers x86\\windbg\" -z \"%1\" -c \"$<c:\\debuggers x86\\commands_2.txt\""
    @="c:\\debuggers x64\\cdb.exe"
    @="Debug This Dump for .NET 2.0 x64"
    @="\"C:\\debuggers x64\\windbg\" -z \"%1\" -c \"$<c:\\debuggers x64\\commands_2.txt\""

    right click menu 

    With every "block" we are associating the .dmp extension to the cdb.exe icon, creating a description to add a new command to the right click menu and linking it to Windbg. Note that we're using some command line switches to tell Windbg which is the file to open (-z) and a script file to execute at startup (-c).

    This script file is actually just a text file with some commands you would normally run at the beginning of your debugging session; for instance I use it to set my symbol path, load some Windbg extensions I use quite often, and open a log file to have the debugger output automatically saved for later reference, just in case I need to quickly find some information without having to reopen Windbg and go through the various commands:

    .sympath SRV*c:\symbols*\\internalshare\symbols*http://msdl.microsoft.com/download/symbols
    .load extension1
    .load extension2
    .load extension3
    .logopen /d /t

    The /d switch causes the log file name to be chosen based on the target process along with its state, while the /t switch appends the process ID with the current date and time (so if I open multiple times the same dump I have a record for all of my debugging sessions). By the way, you can use the .chain command to list all loaded debugger extensions and their search order, i.e. if you type a command which is contained in more that one extension, the one executed is the one contained in the most recently loaded extension.

    Help online

    If you need help about a particular extension, you can type !<extensionname>.help and a list of all available commands for that extension will be printed in the output window, while if you need help on a specific command you can use !<extensionname>.help <commandname>:

    0:000> !sos.help dumpobj
    !DumpObj [-v] [-short] [-r 2] <object address>
    This command allows you to examine the fields of an object, as well as learn 
    important properties of the object such as the EEClass, the MethodTable, and 
    the size.

    Typing commands

    As I said, I really hate when I have to continuously move my hand from the keyboard to the mouse (maybe for just one click) and vice-versa, so I'm always looking for keyboard shortcuts; so within Windbg I like a couple of small details: if you have some text selected within the command output, a simple right-click will copy it into the clipboard and another right-click will paste it to the command textbox. As an alternative, if you have the mouse cursor within the command output (say after you selected a memory address) you can start typing right away without bothering clicking into the command textbox, because the cursor will automatically move there without losing a letter of what you are typing smile_regular. Small thing I know, but that's how it always begin; very small (from "Big trouble in little China" smile_wink).

    Create or connect to a remote debug session

    Sometimes could happen that you need assistance analyzing a dump (or someone asks your help), but what if you're in a team like mine spread across all over Europe and having a colleague walk down to your desk is simply not an option? Luckily Windbg allows to share your debugging session using the .server command: .server tcp:port=<portnumber> for instance if run on your Windbg (the one with the dump you want to share) will create a server session using TCP as the communication protocol, on port <portnumber>; the string highlighted in red below is the connection string you can send to your peer.

    0:000> .server tcp:port=1234
    Server started.  Client can connect with any of these command lines
    0: <debugger> -remote tcp:Port=1234,Server=<computername>

    Now, on the "client" Windbg you can go to File > Connect to remote session (or hit CTRL+R) and either browse the network to find the Windbg server you're looking for, or type tcp:port=<portnumber>,server=<machinename> and your remote colleague and you will see the same command output and will be able to run commands on the shared dump smile_nerd

    connect to remote debugging session

    While connected you can use the .clients command to list of clients connected to the server, and the .echo command to send messages from one Windbg to the other(s).

    Increase output readability

    Going through a long list of commands and outputs in the debugger window may be not that easy, so it's possible to customize the font and text colors through Edit > Options...

    windbg workspace options



    Quote of the day:
    If life is a waste of time, and time is a waste of life, why don't we all get wasted and have the time of our lives?
  • Never doubt thy debugger

    Gacutil not supported on production?


    The other day I got an interesting question from a customer:

    We are documenting how our production assemblies should be deployed on our production servers. At first we wanted to deploy shared assemblies through the gacutil.exe utility.
    By reading
    this article we realized that Microsoft's recommendation is to use gacutil in a development environment but that it should not be used to deploy assemblies in production.

    This raises the following questions: How should we deploy our assemblies in production? Why not gacutil? Any alternative to MSI?

    The customer was referring to this note box:


    The reason is explained in this article:

    In deployment scenarios, use Windows Installer 2.0 to install assemblies into the global assembly cache. Use Windows Explorer or the Global Assembly Cache tool only in development scenarios, because they do not provide assembly reference counting and other features provided when using the Windows Installer.

    The preferred installation method is a MSI package, but using the appropriate command line switches you can still use gacutil as described in http://msdn2.microsoft.com/en-us/library/ex0ss12c(vs.80).aspx:

    Gacutil.exe provides options that support reference counting similar to the reference counting scheme supported by Windows Installer. You can use Gacutil.exe to install two applications that install the same assembly; the tool keeps track of the number of references to the assembly. As a result, the assembly will remain on the computer until both applications are uninstalled. If you are using Gacutil.exe for actual product installations, use the options that support reference counting. Use the /i and /r options together to install an assembly and add a reference to count it. Use the /u and /r options together to remove a reference count for an assembly. Be aware that using the /i and /u options alone does not support reference counting. These options are appropriate for use during product development but not for actual product installations.



    Quote of the Day:
    An adventure is only an inconvenience rightly considered. An inconvenience is only an adventure wrongly considered.
    --G.K. Chesterton
  • Never doubt thy debugger

    Ok, now how do I capture my dump?


    Now we know how it started, some basic information and terminology and why symbols are important, it's now time to capture our first dump. How? When? Using which tool? It depends... smile_nerd

    From the second post of this series we already know the difference between a hang and a crash dump and depending on the problem we are troubleshooting we know which one we need. There are different ways to capture such a dump but the most used tool in CSS is adplus, and specifically in the IIS and Internet Development team (which I belong to, by the way smile_regular) we also use DebugDiag. I personally prefer adplus, I like it to be run at the command line and be highly configurable through its .cfg files, but there are circumstances (especially unmanaged memory leaks and hanging web applications where the customer can't be always monitoring the server to readily capture a manual hang dump with adplus) where I find DebugDiag quite useful. Let's see the differences.


    Adplus is essentially a VBScript (quite large, more than 5.000 lines) file which helps you capture a dump and it accepts some arguments at the command line (or it's also possible to use a .cfg file to fine control its behavior). The basic arguments you must always provide are the dump mode (-hang or -crash) and the target process(es) (-p <process ID> or -pn <process name>):

    adplus -hang -pn w3wp.exe

    Optionally we can pass multiple process names (or process IDs) simply repeating the -p(n) switch; we can also use the special switch -iis to dump all the IIS related processes (inetinfo.exe, dllhost.exe and w3wp.exe/aspnet_wp.exe). We can also chose to not dump the first chance exceptions (-NoDumpOnFirst) or chose to have a full dump also on first chance (-FullOnFirst) etc...

    Create your .cfg

    Adplus supports a xml-based configuration file to avoid retyping lengthy and detailed command lines; you can specify some options in the .cfg and pass others directly from the command line. All options have defaults, so none is mandatory. Here is the .cfg file format:

       1: <ADPlus>
       2:     <!-- Comments -->
       3:     <Settings>
       4:         <!-- defining basic settings (run mode, quiet mode, etc.) -->
       5:     </Settings>
       6:     <PreCommands>
       7:         <!-- defines a set of commands to execute before the sxe and bp commands -->
       8:     </PreCommands>
       9:     <PostCommands>
      10:         <!-- defines a set of commands to execute after the sxe and bp commands -->
      11:     </PostCommands>
      12:     <Exceptions>
      13:         <!-- commands acting on the exception actions -->
      14:     </Exceptions>
      15:     <Breakpoints>
      16:         <!-- defining breakpoints -->
      17:     </Breakpoints>
      18:     <HangActions>
      19:         <!-- defining actions for hang mode -->
      20:     </HangActions>
      21:     <LinkConfig>
      22:         <!-allows to link to another config file -->
      23:     </ LinkConfig >
      24: </ADPlus>

    The Setting section allows you to define the main settings for adplus, and most of them are equivalent to the command line switches; it's possible to include multiple instances of ProcessID, ProcessName, Spawn and Notify tags:

       1: <Settings>
       2:     <RunMode> runmode </RunMode>
       3:     <Option> IIS </Option>
       4:     <Option> Quiet </Option>
       5:     <Option> NoTlist </Option>
       6:     <Option> NoTsCheck </Option>
       7:     <Option> NoFreeSpaceChecking </Option>
       8:     <OutputDir> OutputDirectoryName </OutputDir>
       9:     <ProcessID> PID </ProcessID>
      10:     <ProcessName> ProcessName </ProcessName>
      11:     <GenerateScript> ScriptName </GenerateScript>
      12:     <Spawn> Spawn Command </Spawn>
      13:     <Sympath> Symbol Path </Sympath>
      14:     <SympathPlus> Symbol Path to add </SympathPlus>
      15:     <Notify> { ComputerName | UserName } </Notify>
      16:     <Debugger> Debugger </ Debugger >
      17: </Settings>

    The Exceptions section allows to customize the actions taken when the debugger breaks on the specified exception; it's also possible to define custom exceptions to monitor:

       1: <Exceptions>
       2:         <!-- options act on all currently defined exceptions -->
       3:        <Option>  FullDumpOnFirstChance  </Option>
       4:        <Option>  MiniDumpOnSecondChance  </Option>
       5:        <Option>  NoDumpOnFirstChance  </Option>
       6:        <Option>  NoDumpOnSecondChance  </Option>
       8:         <!-Defining new exceptions -->
       9:        <NewException>
      10:            <Code> ExceptionCode </Code>
      11:            <Name> ExceptionName </Name>
      12:        </NewException>
      14:         <!-Configuring already defined exceptions -->
      15:        <Config>
      16:           <Code> { ExceptionCode | AllExceptions } </Code>
      17:           <Actions1> Actions  </Actions1> 
      18:           <CustomActions1> CustomActions  </CustomActions1> 
      19:           <Actions2> Actions  </Actions2> 
      20:           <CustomActions2> CustomActions  </CustomActions2> 
      21:           <ReturnAction1> { G | GN | GH | Q | QD } </ReturnAction1> 
      22:           <ReturnAction2> { G | GN | GH | Q | QD } </ReturnAction2> 
      23:        </Config>  
      24:     </Exceptions>

    ReturnAction1 and ReturnAction2 tags are used to define additional actions for first and second chance exceptions respectively; they work just like the debugger commands and are supported only in Windows XP and later.

    The HangActions section allows you specify actions to take when running adplus in hang mode:

       1: <HangActions>
       2:      <Option> MiniDump </Option>
       3:      <Option> FullDump </Option>
       4:      <Option> NoDump </Option>
       5:      <Option> Clear </Option>
       6:      <Actions> Actions  </Actions> 
       7:      <CustomActions> CustomActions  </CustomActions>
       8: </HangActions>

    Here is a sample configuration file (taken from WinDbg help):

       1: <ADPlus>
       2:     <Settings>
       3:         <RunMode> CRASH </RunMode>
       4:         <Option> IIS </Option>
       5:         <OutputDir> c:\MyDumps </OutputDir>
       6:     </Settings>
       8:     <PreCommands>
       9:         <Cmd> .load sos\sos.dll  </Cmd>
      10:     </PreCommands>
      12:     <!-- defining and configuring exceptions -->
      13:     <Exceptions>
      14:         <!-- First we redefine all exceptions -->
      15:         <Config>
      16:             <Code>AllExceptions</Code>
      17:             <Actions1>Log</Actions1>
      18:             <Actions2>MiniDump;Log;EventLog</Actions2>
      19:         </Config>
      20:         <!-- Next we define a special behavior for the Access Violation exception -->
      21:         <Config>
      22:             <Code> av </Code>
      23:             <Actions1>Log</Actions1>
      24:             <Actions2>FullDump;Log;EventLog</Actions2>
      25:         </Config>
      26:         <!-- Adding a custom exception -->
      27:         <NewException>
      28:             <Code> 0x80010005 </Code>
      29:             <Name> Custom_Exception_05 </Name>
      30:         </NewException>
      31:         <!-- Configuring the custom exception -->
      32:         <Config>
      33:             <Code> 0x80010005 </Code>
      34:             <Actions1>Log;Stack</Actions1>
      35:             <Actions2>FullDump;Log;EventLog</Actions2>
      36:             <CustomActions2>   !heap;!locks     </CustomActions2>
      37:         </Config>
      38:     </Exceptions>
      40:     <!-- defining breakpoints -->
      41:     <Breakpoints>
      42:         <NewBP>
      43:             <Address> MyModule!MyClass::MyMethod </Address>
      44:             <Actions> Log;Stack;MiniDump  </Actions> 
      45:             <ReturnAction> G  </ReturnAction> 
      46:         </NewBP>  
      47:     </Breakpoints>
      48: </ADPlus>

    For further details see ADPlus Configuration Files section in WinDbg help.


    This tool was originally developed by the IIS Escalation Engineer team (I'm not sure if they are still the ones behind it now) and essentially allows the user to create a set of rules to capture hang and crash dumps of your processes, also allowing you to specify on which exception to capture the dump, how many dumps to capture for each session, on which events etc... Very flexible.

    DebugDiag report There are three circumstances where I find DebugDiag really helpful and can ease my debugging life compared to adplus. First, it makes very easy to debug in a Terminal Session, it's just a matter of adding a flag in the Preferences tab. Second, creating a hang rule it's possible to monitor a specific URL and DebugDiag will automatically "ping" it to check if the site is up and running; if the server does not respond, DebugDiag will capture the dump you told it. This can be very useful in situations where you need to troubleshoot a site which hangs sporadically, maybe at night, and it's not possible to pay someone to patiently sit in front of the server waiting to run adplus at the right moment... smile_regular When you create this kind of rule you have to specify one (or more) URLs to monitor and then you have to tell DebugDiag which process or application pool to monitor; so even if you can configure the rule on your client to monitor www.yoursite.com, of course you can't dump a process which is running on a different machine so you must install and configure DebugDiag on the affected web server.

    For memory leaks and crashes you can quickly have an idea of what's in the dump opening it within DebugDiag and running the appropriate script in the Advanced Analysis tab; you'll have a simple but precise report about which exception caused the crash and in which thread, or a list of the components responsible for the higher allocation rates etc... This is not enough to have a complete analysis on the problem, but it's for sure a good point to start; of course you can still open and analyze those dumps with WinDbg as usual.

    WinDbg while live debugging

    Live debugging will be the subject of a future post, but just in case you attached WinDbg to a process and you're now wondering how to dump it...

    .dump /ma <output file>

    Dumping on a terminal session?

    If you need to dump a process on a Windows NT or Windows 2000 machine, you cannot to so through a Terminal Server session since the debugger cannot attach to a process which runs in a different Window Station, and if you try you'll likely get a message like the following:

    Win32 error 5
    "Access is denied."

    This is by design but there are a couple of solutions if you need, check You cannot debug through a Terminal Server session or "Running in crash mode remotely" topic in Windbg help.

    DebugDiag makes it easier to overcome this limitation, simply configuring it to run in service mode through its property dialog

    options & settings


    We have different tools for different tasks but also to fit specific habits of every developer: a command line script for geeks whom want to be in full control of their machine and a GUI based debugger for speedy people and final customers whom want to start troubleshooting their applications on their own. Which one to chose is mainly a matter of taste and feelings, the environment you're troubleshooting (if you already have one debugger installed on the machine, it does not make much sense to install the other as they are equivalent in most circumstances) etc... Make your choice and go with your preferred debugger, but if I can give you an advice don't forget to give the other one a try, sooner or later it might come to the rescue... smile_wink



    Quote of the Day:
    Men achieve a certain greatness unawares, when working to another aim.
    --Ralph Waldo Emerson
  • Never doubt thy debugger

    Is your UserControl sluggish at loading?


    Here's an interesting story about performance I had the chance to work on over the last couple of weeks. The object of the call was a UserControl embedded in Internet Explorer, which was very slow to load the first time you browsed the page, but then was performing quite well after that long delay (around 60 seconds); unfortunately closing and reopening the browser caused another 60 seconds delay, which was quite bothering if not frustrating for end users... As you can imagine the control needs to be downloaded, JIT compiled and loaded which of course requires some time depending on how big is the control, how fast (or slow) the Internet connection, how powerful the client etc..., but those 60 seconds where definitely too much. Moreover on Vista we were prompted to run csc.exe and under some circumstances (usually if IE was not run as Administrator) we got a FileNotFoundException.

    fiddler probing


    First thing, Fiddler showed that after downloading the first dll (the actual UserControl) we were probing for further localized resources, so we wasting about 15 seconds just for this reason (which is a quarter of the entire time spent waiting on a white IE page...). At the same time Process Monitor showed (as expected) that some time was needed to JIT the control but also to create on the fly the serialization proxy assembly, which the UserControl needed to call a remote Web Service and get the data to show to the user.

    There are different possible approaches to the problem, but in this case we had some constraints imposed by the hosting company the customer was using to test his application, so to work around those limitations we first removed the strong name from the two assemblies because it was not a strict security requirement in this case and because strong named assemblies are prone to probing (as you may know, one of the token in a strong name is the culture information). Then we specified the NeutralResourceLanguageAttribute to tell the ResourceManager to not search for resources that are included in the main assembly; when looking up resources in the same culture as the neutral resources language, the ResourceManager automatically uses the resources located in the main assembly, instead of searching for a satellite assembly with the current user interface culture for the current thread. This will improve lookup performance for the first resource you load, and can reduce your working set. As a final step we added an empty file (zero bytes) in the first places where the runtime was probing; this because the NeutralResourceLanguageAttribute resolved the problem for me (I was testing with an English OS) but not for the customer whom was using a non English OS...

    Serialization assembly

    As you can see from the screenshot here, the runtime was also probing for <assemblyname>.XmlSerializers.dll which is the proxy needed to call the remote Web Service (and which needs to be created on the fly, as Process Monitor shown); so in this case we used sgen.exe to pre-generate the serialization assembly we could put on the server in the same folder of the UserControl. This serialization assembly is downloaded by the browser and therefore there is no need to create it on the client, so we removed both the security prompt (and the exception) and the delay to create the proxy on the client.

    In short, after applying those changes the time taken from the first HTTP request to the UserControl fully loaded and functioning on the client went from about 60 seconds to just 5! smile_omg

    There is a nice article from Chris Sells on the subject, it's 5 years old but still gives some interesting insight on probing and how serve related files (like .config etc...).

    Specify a codeBase

    Doing further researches for this post I found another couple of possible solutions to the problem. First it's possible to specify a codeBase attribute in your web.config as described in http://support.microsoft.com/kb/814668/en-us (this applies also to .NET 1.1).

    Or you can add a section to the web.config file telling the ResourceManager exactly which resources are present on the machine for each assembly.  Here's an example:

       1: <?xml version="1.0"?>
       2: <configuration>
       3:     <satelliteassemblies>
       4:         <assembly name="mscorlib, Version=..., PublicKeyToken=...">
       5:             <culture>fr</culture>
       6:         </assembly>
       7:         <assembly name="UserAssembly, ...">
       8:             <culture>fr-FR</culture>
       9:             <culture>de-CH</culture>
      10:         </assembly>
      11:             <assembly name="UserAssembly2, ...">
      12:         </assembly>
      13:     </satelliteassemblies>
      14: </configuration>

    Note that the assembly references in this example are abbreviated, and they will need to supply full assembly names.



    Quote of the Day:
    For greed, all nature is too little.
    --Seneca, Roman statesman and author
  • Never doubt thy debugger

    Remember to undo your impersonation


    A couple of weeks ago I got an interesting query from a customer, whom had a problem impersonating a service user account by code; the design was a bit more complicated, though:

    • Impersonation not set in web.config
    • By code they needed to impersonate the account logged on the client issuing the HTTP request (this worked fine)
    • By code they needed to impersonate a service account they used to access a backend database (here they were getting an access denied error)
    • Switch back to the previous user, the one logged on the client (again this was working fine)

    This was quite clearly an impersonation problem, and after some debugging we found the "Access Denied" was being thrown when executing the line highlighted in red in the following snippet, way before even trying to access the network to read the backend database:

     1: If CType(LogonUser(username, domain, password, LOGON32_LOGON_INTERACTIVE, LOGON32_PROVIDER_DEFAULT, token), Boolean) Then
     2:    If DuplicateToken(token, 2, tokenDuplicate) Then
     3:    Dim identity As New WindowsIdentity(tokenDuplicate)
     4:       If System.Web.HttpContext.Current Is Nothing Then
     5:          Dim mImpersonatedContext As WindowsImpersonationContext = identity.Impersonate
     6:       Else
     7:          System.Web.HttpContext.Current.Items("ImpersonationContext") = identity.Impersonate
     8:       End If
     9:    End If
     10: [...]

    In the screenshot below you can see the "Access Denied" message when trying to display the WindowsIdentity.Name property

    Autos: access is denied

    Since I was able to repro on my machine, I attached WinDbg to the worker process and set a breakpoint on advapi32!ImpersonateLoggedOnUser and having a look at the stack and managed exceptions the problem was quite clear. !gle shows the last error for the current thread:

       1: 0:017> !gle
       2: LastErrorValue: (Win32) 0x5 (5) - Access is denied.
       3: LastStatusValue: (NTSTATUS) 0xc0000022 - {Access Denied}  A process has requested access to an object, 
       4:     but has not been granted those access rights.

    Also confirmed by the managed exceptions:

       1: Exception object: 021314ec
       2: Exception type: System.Web.HttpException
       3: Message: An error occurred while attempting to impersonate.  Execution of this request cannot continue.
       4: InnerException: <none>
       5: StackTrace (generated):
       6:     SP       IP       Function
       7:     01ECF4F0 044DCB73 System.Web.ImpersonationContext.GetCurrentToken()
       8:     01ECF534 041E3BE9 System.Web.ImpersonationContext.get_CurrentThreadTokenExists()
       9:     01ECF564 0417FB4E System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)
      10:     01ECF61C 041922CC System.Web.HttpApplication+ApplicationStepManager.ResumeSteps(System.Exception)
      11:     01ECF66C 0417EEA6 System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest(System.Web.HttpContext, System.AsyncCallback, System.Object)
      12:     01ECF688 04183DB5 System.Web.HttpRuntime.ProcessRequestInternal(System.Web.HttpWorkerRequest)
      15: Exception object: 021312a8
      16: Exception type: System.Security.SecurityException
      17: Message: Access is denied.
      19: InnerException: <none>
      20: StackTrace (generated):
      21:     SP       IP       Function
      22:     01ECF24C 79636928 System.Security.Principal.WindowsIdentity.GetCurrentInternal(System.Security.Principal.TokenAccessLevels, Boolean)
      23:     01ECF26C 79389652 System.Security.Principal.WindowsIdentity.GetCurrent()
      24:     01ECF278 06350A92 WebApplication1._Default.Page_Load(System.Object, System.EventArgs)
      25:     01ECF318 04301954 System.Web.UI.Control.OnLoad(System.EventArgs)
      26:     01ECF328 043019A0 System.Web.UI.Control.LoadRecursive()
      27:     01ECF33C 043147C4 System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)
      28:     01ECF50C 04312982 System.Web.UI.Page.ProcessRequest(Boolean, Boolean)
      29:     01ECF544 0431285F System.Web.UI.Page.ProcessRequest()
      30:     01ECF57C 0431277F System.Web.UI.Page.ProcessRequestWithNoAssert(System.Web.HttpContext)
      31:     01ECF584 04312712 System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)
      32:     01ECF598 063502F6 ASP.default_aspx.ProcessRequest(System.Web.HttpContext)
      33:     01ECF5A4 041BA93F System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
      34:     01ECF5DC 0417FAD1 System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)

    As a test we added the Job Manager account to the local Administrators group and the problem went away, so this was clearly a lack of permission for that account; specifically, the Job Manager user was not allowed to "take ownership" of the current thread, which was already impersonating the client account whom issued the HTTP request. Since we where using service account we checked the permission required in the article Process and request identity in ASP.NET, in particular the ASPNET account specific permission configurable through the Group Policy snap-in (gpedit.msc); we added the required permission, but the problem was still there.

    We then found a quite old (but still applicable) KB article which seemed to be applicable to this problem: LogonUser fails in ISAPI extensions. Here's the interesting part:


    The code inside LogonUser tries to open the process token. It fails since the authenticated user may not have access to the process token (SYSTEM if it's an inproc ISAPI.)


    As a temporary workaround, you can call RevertToSelf to return the thread to the security context of the process token before calling LogonUser.


    This behavior is by design.

    So, going back in the stack where the failing call started, here is what we find:

       1: Dim test As New customer
       2: Dim col As List(Of CustomerDetails)
       3: Dim p As WindowsPrincipal = HttpContext.Current.User
       4: Dim id As WindowsIdentity = p.Identity
       6: 'impersonate the Windows Authenticated User
       7: Dim wic As WindowsImpersonationContext = id.Impersonate()
       8: col = test.getcustomers() ' Works okay to access database
      10: 'Error Switching below, step into code
      11: CBPUser.SwitchToIOUser()   ' Access Denied, step through code to see issue arise.
      12: col = test.getcustomers()  ' Fails to access database
      14: CBPUser.SwitchFromIOUser() ' Switch back
      15: col = test.getcustomers()  ' Works okay to access database again

    I then had a look at the MSDN docs about impersonation, especially to find some sample code, like for example How To: Using impersonation and delegation in ASP.NET 2.0 and How to implement impersonation in an ASP.NET application: interesting enough all the samples in those articles always revert the impersonation calling the WindowsImpersonationContext.Undo() method (which under the covers ultimately calls RevertToSelf(), as you can guess)...

    Since testing the code in practice is easier an quicker, I added the Undo() call and run it again:

       1: Dim test As New customer
       2: Dim col As List(Of CustomerDetails)
       3: Dim p As WindowsPrincipal = HttpContext.Current.User
       4: Dim id As WindowsIdentity = p.Identity
       6: ' impersonate the Windows Authenticated User
       7: Dim wic As WindowsImpersonationContext = id.Impersonate()
       8: col = test.getcustomers() ' Works okay to access database
       9: wic.Undo()
      11: ' Error Switching below, step into code
      12: CBPUser.SwitchToIOUser()   'Works fine now, no more access denied!
      13: col = test.getcustomers()  'Works ok to access database
      15: CBPUser.SwitchFromIOUser() ' Switch back
      16: col = test.getcustomers()  ' Works okay to access database again

    Much better! smile_regular Just to be sure, Sql Profiler shown a connection with the service account which was our final goal. So the final message is: remember to always use Undo() when you're done with your code impersonation.

    Case closed Sherlock! smile_nerd



    Quote of the Day:
    Resentment is like taking poison and hoping the other person dies.
    --St. Augustine
  • Never doubt thy debugger

    My goodness, were's gone my Properties window?!?


    I had a funny half-hour this afternoon, when one of my colleagues took a new call from a customer whom had some troubles with the Properties window in his Visual Studio 2005... The customer reported a weird behavior within the Visual Studio IDE in particular with the Properties window, which was not available despite his attempts to display it both pressing the F4 keyboard shortcut and from the View > Properties Window menu command. Moreover this had the effect to remove the focus from the Visual Studio IDE but apparently nothing else was getting it, and there was a weird Format menu appearing and disappearing... smile_omg

    It was kind of fun to see Stefano on the phone with the customer, listening to his description and his face getting more and more puzzled... smile_teeth Until after some quick research in our internal docs he found a reference to a similar problem, and after a quick test he triumphally called the customer back with the solution after just 20 minutes! thumbs_up

    As I guess you know the Properties window (like other windows in Visual Studio) can be detached from the IDE and left float around the screen; the point here was that the customer somehow moved the window outside the desktop and it was not visible anymore... smile_omg How to get it back?

    • Press F4 (or the corresponding keyboard shortcut) to move the focus to the Visual Studio window you're hunting
    • Press ALT+- (ALT and minus key) to open the small window menu
    • Press M
    • Press ENTER
    • Now the window will be "attached" to the mouse pointer to move it around, but most important we can see it now! smile_angelDock it to the IDE if you wish



    Quote of the Day:
    I like pigs. Dogs look up to us. Cats look down on us. Pigs treat us as equals.
    --Winston Churchill
  • Never doubt thy debugger

    Why should we care about symbols?


    I already touched this topic a while ago, but since it's an important part of the debugging process (and your debugging techniques may vary a lot, depending if you have or not good symbols for your dump) I though would be a good idea to give some more details. And just to jump start on the topic, here's something I learnt a while ago after wasting a few hours typing commands and looking at inconsistent results... debugging with the wrong symbols could be much worse than debugging with no symbols at all.

    What are symbols?

    You can think of symbol files basically as small databases, files which contain source line information, data types, variables, functions and everything else needed to provide names for all lines of code, instead of hexadecimal addresses. They usually have .pdb (or .dbg) extension, and are matched with the actual executable code using an internal timestamp, so it's very important to generate symbols every time you build the application, also for release builds. If you'll ever have a problem with your live application and you'll need to debug it, and if you'll not have the matching symbols (matching means the symbols obtained from the same exact build of the dlls you put in production) you could be in troubles... Building once again the application to create the symbol files, even without changing your code does not really help because the timestamp will not match, and WinDbg will complain about missing symbols anyway...

    generate debugging information

    Note that in Visual Studio if you set "Release" in the "Standard" toolbar, the "Generate debugging information" checkbox will automatically be cleared, so remember to go to the Project properties and flag it again before rebuilding the project.

    Since in ASP.NET 2.0 we have a new default page architecture, there is no real need to create the .pdb files unless you're using the "Web Application Project" template which comes with the Service Pack 1 for Visual Studio 2005, in which case you can find it from the project properties, "Compile" tab.

    advanced compiler settings

    The same applies for Visual Studio 2008.

    How can we use symbols?

    When you open a dump within WinDbg and you type in a command for example to inspect the call stack, the debugger will start looking for matching symbols to show you an output as detailed as possible: but how does it decide where to look for those symbols? It will use the path(s) specified in the "Symbol Search Path" dialog you can find under the File menu (CTRL+S is the keyboard shortcut.

    symbol search path dialog Here you can specify the symbol servers where WinDbg can download the symbols from, and of course you can use has many server more than one server at a time; WinDbg will simply access those servers in the same order you put in the search path, and it goes through the end of the list until it finds a match.

    Microsoft has a public symbol server accessible through the Internet which stores public .pdb files for almost all of our products: http://msdl.microsoft.com/download/symbols.

    I work a lot with WinDbg and memory dumps in my daily job, also with my laptop when not connected to the Internet or corporate network, so I need to be able to debug while offline and in any case I don't want to waste time waiting for the tool to download the same symbols over and over again, through one dump to the other... for this reason it's also possible to create a local symbol cache (also referred as downstream store) on your hard disk. When looking for symbols, WinDbg will first of all check your local cache and if the match is found there no additional check is done against the other symbol servers, while if the match is not found WinDbg goes on as usual until it finds the right match; in this case it downloads the .pdb and stores it in your local symbol cache, so that next time it will be readily available for you.

    The symbol path is a string composed of multiple directory paths, separated by semicolons. For each directory in the symbol path, the debugger will look in three directories: for instance, if the symbol path includes the directory c:\MyDir, and the debugger is looking for symbol information for a dll, the debugger will first look in c:\MyDir\symbols\dll, then in c:\MyDir\dll, and finally in c:\MyDir. It will repeat this for each directory in the symbol path. Finally, it will look in the current directory, and then the current directory with \dll appended to it. (The debugger will append dll, exe, or sys, depending on what binaries it is debugging.)

    Here is a sample symbol path:


    The above is actually the symbol path (a bit simplified) I use on my machines: as you can see I have a local cache in C:\Symbols; if I've never downloaded a particular symbol before, WinDbg does to an internal share were we have full symbols, and if still unsuccessful I finally give a try to the public Microsoft symbol server on the Internet. If you include two asterisks in a row where a downstream store would normally be specified, then the default downstream store is used. This store will be located in the sym subdirectory of the home directory. The home directory defaults to the debugger installation directory; this can be changed by using the !homedir extension. If the DownstreamStore parameter is omitted and no extra asterisk is included (i,e. if you use srv with exactly one asterisk or symsrv with exactly two asterisks) then no downstream store will be created and the debugger will load all symbol files directly from the server, without caching them locally. Note that If you are accessing symbols from an HTTP or HTTPS site, or if the symbol store uses compressed files, a downstream store is always used. If no downstream store is specified, one will be created in the sym subdirectory of the home directory.

    The symbol server does not have to be the only entry in the symbol path. If the symbol path consists of multiple entries, the debugger checks each entry for the needed symbols; moreover the symbol path can contain several directories or symbol servers, separated by semicolons. This allows you to locate symbols from multiple locations (or even multiple symbol servers). If a binary has a mismatched symbol file, the debugger cannot locate it using the symbol server because it checks only for the exact parameters. However, the debugger may find a mismatched symbol file with the correct name, using the traditional symbol path, and successfully load it; in this case it's important to know if our symbols matches (see the next topic).

    You can set the symbol path in advance once for all within WinDbg: open an empty instance, press CTRL+S, type in the path, clock "Ok" on the dialog and close WinDbg, accepting to save the workspace if prompted to do so (next time you'll open WinDbg the value will still be there). Or you can use the .sympath command within WinDbg with a dump open.

    Another option is to set the system wide variable _NT_SYMBOL_PATH (the syntax is still the same), used by debuggers like WinDbg or also from adplus directly when it captures the dump.

    The same principle applies to the Visual Studio debugger (also have a look at the article http://support.microsoft.com/kb/311503/en-us):

    visual studio symbols options

    How can I check if my symbols matches?

    Looking at a call stack sometimes it's clear you're having a problem with unmatched symbols because WinDbg tells you something like:

     1: ChildEBP RetAddr 
     2: 0012f6dc 7c59a2d1 NTDLL!NtDelayExecution(void)+0xb
     3: 0012f6fc 7c59a29c KERNEL32!SleepEx(unsigned long dwMilliseconds = 0xfa, int bAlertable = 0)+0x32
     4: *** ERROR: Symbol file could not be found. Defaulted to export symbols for aspnet_wp.exe - 
     5: 0012f708 00442f5f KERNEL32!Sleep(unsigned long dwMilliseconds = 0x444220)+0xb
     6: WARNING: Stack unwind information not available. Following frames may be wrong.
     7: 0012ff60 00444220 aspnet_wp+0x2f5f
     8: 0012ffc0 7c5989a5 aspnet_wp!PMGetStartTimeStamp+0x676
     9: 0012fff0 00000000 KERNEL32!BaseProcessStart(<function> * lpStartAddress = 0x004440dd)+0x3d

    Unfortunately could happen to not be so lucky, and you'll find yourself wondering if the stack you are looking at is genuine or there are some small (or maybe even not so small) inconsistencies which may lead you down to a completely wrong path. In such cases, you can first of all use the lm command to find which .pdb files have been loaded:

     1: kernel32 (pdb symbols) .sympath SRV\kernel32.pdb\CE65FAF896A046629C9EC86F626344302\kernel32.pdb
     2: ntdll (pdb symbols) .sympath SRV\ntdll.pdb\36515FB5D04345E491F672FA2E2878C02\ntdll.pdb
     3: shell32 (deferred)
     4: user32 (deferred)

    As you can see in the example above, two symbols were loaded (for kernel32.dll and ntdll.dll), while shell32.dll and user32.dll were not part of the stack analyzed, so WinDbg has not loaded yet (deferred) their symbols. A bad match will look like the following:

     1: ntdll M (pdb symbols) .sympath SRV\ntdll.pdb\36515FB5D04345E491F672FA2E2878C02\ntdll.pdb

    Notice the "M" highlighted in red (could also be a "#" pound sign)? That stands or "mismatch", and indicates there is a problem with that particular module (search for "Symbol Status Abbreviations" in WinDbg help for further details). Alternatively you can use the !sym noisy and .reload command to reload symbols verbosely to have a detailed output. Look for "Symbols files and paths - Overivew" in WinDbg help for more details.

    Trick: you have the right symbol, but WinDbg does not matches it anyway...

    I'm not sure why this happens, and especially why I had this problem only with ntdll.dll (and its .pdb): I was not able to get the proper stack even with a matching symbol (and I checked more than once to be really sure)... until I got the idea to delete the ntdll.pdb folder in my local cache (if you have a dump open you must first unload the symbol from WinDbg or the file will be locked: use the .reload /u <module_name> command), then run a .reload /f <module_name> (/f forces immediate symbol load) and let WinDbg to download it again... this usually does the trick and I finally get the correct stack.

    Debugging without symbols?

    It's not impossible, but it's harder than debugging with matching symbols; the main difference is that you'll not be able to see method names, variable names etc... and generally speaking the stack will be less easily readable. To give you a quick example, here is an excerpt of the stack of a very simple application I wrote for test (it has just a button which sets the text of a label to the DateTime.Current.ToString()):

    without symbols:

     1: 5 Id: 10d4.1204 Suspend: 1 Teb: 7ffd7000 Unfrozen
     2: ldEBP RetAddr 
     3: NING: Frame IP not in any known module. Following frames may be wrong.
     4: 9f524 71a6b7f8 0x7c90eb94
     5: 9fa0c 03490657 0x71a6b7f8
     6: WARNING: Unable to verify checksum for System.dll
     7: ERROR: Module load completed but symbols could not be loaded for System.dll
     8: 9fa40 7a603543 CLRStub[StubLinkStub]@3490657(<Win32 error 318>)
     9: a8240 032908ff System!System.Net.Sockets.Socket.Accept(<HRESULT 0x80004001>)+0xc7
     10: ERROR: Module load completed but symbols could not be loaded for WebDev.WebHost.dll
     11: 9fab0 7940a67a WebDev_WebHost!Microsoft.VisualStudio.WebHost.Server.OnStart(<HRESULT 0x80004001>)+0x27
     12: WARNING: Unable to verify checksum for mscorlib.dll
     13: ERROR: Module load completed but symbols could not be loaded for mscorlib.dll
     14: bd1b4 7937d2bd mscorlib!System.Threading._ThreadPoolWaitCallback.WaitCallback_Context(<HRESULT 0x80004001>)+0x1a
     15: bd1b4 7940a7d8 mscorlib!System.Threading.ExecutionContext.Run(<HRESULT 0x80004001>)+0x81
     16: 9fae0 7940a75c mscorlib!System.Threading._ThreadPoolWaitCallback.PerformWaitCallbackInternal(<HRESULT 0x80004001>)+0x44
     17: 32010 79e79dd3 mscorlib!System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(<HRESULT 0x80004001>)+0x60
     18: 9fb04 79e79d57 0x79e79dd3
     19: 9fb84 79f71cba 0x79e79d57
     20: 9fba4 79f71c64 0x79f71cba
     21: 9fc08 79f71cf3 0x79f71c64
     22: 9fc3c 7a0b0896 0x79f71cf3
     23: 9fc9c 79f7ba4f 0x7a0b0896
     24: 9fcb0 79f7b9eb 0x79f7ba4f
     25: 9fd44 79f7b90c 0x79f7b9eb
     26: 9fd80 79ef9887 0x79f7b90c
     27: 9fda8 79ef985e 0x79ef9887
     28: 9fdc0 7a0a32da 0x79ef985e
     29: 9fe28 79ef938f 0x7a0a32da
     30: 9fe94 79f7be67 0x79ef938f
     31: 9ffb4 7c80b683 0x79f7be67
     32: 9ffec 00000000 0x7c80b683

    with matching symbols:

     1: 5 Id: 10d4.1204 Suspend: 1 Teb: 7ffd7000 Unfrozen
     2: ldEBP RetAddr 
     3: 9f4e4 7c90e9c0 ntdll!KiFastSystemCallRet
     4: 9f4e8 71a54033 ntdll!ZwWaitForSingleObject+0xc
     5: 9f524 71a6b7f8 mswsock!SockWaitForSingleObject+0x1a0
     6: 9f9bc 71ac0e2e mswsock!WSPAccept+0x21f
     7: 9f9f0 71ac103f ws2_32!WSAAccept+0x85
     8: 9fa0c 03490657 ws2_32!accept+0x17
     9: WARNING: Unable to verify checksum for System.ni.dll
     10: 9fa40 7a603543 CLRStub[StubLinkStub]@3490657(<Win32 error 318>)
     11: a8240 032908ff System_ni!System.Net.Sockets.Socket.Accept(<HRESULT 0x80004001>)+0xc7
     12: 9fab0 7940a67a WebDev_WebHost!Microsoft.VisualStudio.WebHost.Server.OnStart(<HRESULT 0x80004001>)+0x27
     13: WARNING: Unable to verify checksum for mscorlib.ni.dll
     14: bd1b4 7937d2bd mscorlib_ni!System.Threading._ThreadPoolWaitCallback.WaitCallback_Context(<HRESULT 0x80004001>)+0x1a
     15: bd1b4 7940a7d8 mscorlib_ni!System.Threading.ExecutionContext.Run(<HRESULT 0x80004001>)+0x81
     16: 9fae0 7940a75c mscorlib_ni!System.Threading._ThreadPoolWaitCallback.PerformWaitCallbackInternal(<HRESULT 0x80004001>)+0x44
     17: 32010 79e79dd3 mscorlib_ni!System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(<HRESULT 0x80004001>)+0x60
     18: 9fb04 79e79d57 mscorwks!CallDescrWorker+0x33
     19: 9fb84 79f71cba mscorwks!CallDescrWorkerWithHandler+0xa3
     20: 9fba4 79f71c64 mscorwks!DispatchCallBody+0x1e
     21: 9fc08 79f71cf3 mscorwks!DispatchCallDebuggerWrapper+0x3d
     22: 9fc3c 7a0b0896 mscorwks!DispatchCallNoEH+0x51
     23: 9fc9c 79f7ba4f mscorwks!QueueUserWorkItemManagedCallback+0x6c
     24: 9fcb0 79f7b9eb mscorwks!Thread::DoADCallBack+0x32a
     25: 9fd44 79f7b90c mscorwks!Thread::ShouldChangeAbortToUnload+0xe3
     26: 9fd80 79ef9887 mscorwks!Thread::ShouldChangeAbortToUnload+0x30a
     27: 9fda8 79ef985e mscorwks!Thread::ShouldChangeAbortToUnload+0x33e
     28: 9fdc0 7a0a32da mscorwks!ManagedThreadBase::ThreadPool+0x13
     29: 9fe28 79ef938f mscorwks!ManagedPerAppDomainTPCount::DispatchWorkItem+0xdb
     30: 9fe3c 79ef926b mscorwks!ThreadpoolMgr::ExecuteWorkRequest+0xaf
     31: 9fe94 79f7be67 mscorwks!ThreadpoolMgr::WorkerThreadStart+0x223
     32: 9ffb4 7c80b683 mscorwks!Thread::intermediateThreadProc+0x49
     33: 9ffec 00000000 kernel32!BaseThreadStart+0x37

    The difference is quite obvious... The WinDbg help file also gives a few hints:

    1. To figure out what the addresses mean, you'll need a computer which matches the one with the error. It should have the same platform (x86, Intel Itanium, or x64) and be loaded with the same version of Windows
    2. When you have the computer configured, copy the user-mode symbols and the binaries you want to debug onto the new machine
    3. Start CDB or WinDbg on the symbol-less machine
    4. If you don't know which application failed on the symbol-less machine, issue an | (Process Status) command. If that doesn't give you a name, break into KD on the symbol-less machine and do a !process 0 0, looking for the process ID given by the CDB command
    5. When you have the two debuggers set up — one with symbols which hasn't hit the error, and one which has hit the error but is without symbols — issue a k (Display Stack Backtrace) command on the symbol-less machine
    6. On the machine with symbols, issue a u (Unassemble) command for each address given on the symbol-less stack. This will give you the stack trace for the error on the symbol-less machine
    7. By looking at a stack trace you can see the module and function names involved in the call

    I got symbols from my customer: what should I do now?

    The easiest thing you could do is use symstore.exe (you'll find it in WinDbg/adplus installation folder) with a command like the following:

    symstore add /f c:\temp\SymbolsTest\Bin\*.pdb /s c:\symbols /t "Symbols Test"

    Let's have a quick look at the syntax:

    • The "add" keyword is quite self explanatory smile_wink
    • /f tells symstore which is the file you want to add; note you can use wildcards, so you can add multiple files at once
    • /s is the path to your symbols store (this will most likely be your local cache, or your shared symbol server)
    • /t a required description for the symbol to store

    Symstore will create a structure similar to the following:


    Also note the "0000Admin" folder created by symstore, which contains one file for each transaction (every "add" or "delete" operation is recorded as a transaction), as well as the logs server.txt and history.txt; the former contains a list of all transactions currently on the server, while the latter contains a chronological history of all transactions run on the machine. For further information you can see the "Using SymStore" topic in WinDbg help (debugger.chm).


    While it is possible to debug without symbols (this is true especially for managed code), remember that your life could be much easier if you (and your customers) will take care of your symbols and will generate them every time the application will be rebuilt, also in release mode.

    I deliberately simplified the argument (I just wanted to share what I learnt in my daily job and give some quick tips to get started), much more could be said and if you're interested I encourage you to read the "Symbols" section in the WinDbg help, or search the Internet where you'll find good blog posts/articles on this subject smile_nerd


  • Never doubt thy debugger

    SyncToy not working on Vista x64?


    I've been using SyncToy for quite a few months to keep in sync some folders between my laptop and the other two machines I have in office, and it always worked just great for me (I know, I should be using Groove instead but I'm not happy to have services and programs running when they want, instead of when I tell them to run... smile_nerd).

    When I switched my primary desktop in office to Vista x64, I very quickly discovered that SyncToy was crashing immediately after running it, with no error messages or clues about what is going wrong... I didn't had much time to spend debugging it and try to figure out what was going wrong (it's not a must have tool for my work, after all...) so I simply used the laptop to synchronize folders between the two desktops, too...

    Until this morning, when I had a few minutes free and decided to get back to this problem and try to fix it once for all (and write a blog post on it, too smile_wink); anyway before even opening WinDbg, a research on the Internet brought me to this blog: http://joshmouch.wordpress.com/2007/03/27/synctoy-14-and-vista-x64-error-fixed/. I tried, and it works like a charm! thumbs_up Wonderful, thanks a lot Josh!



    Quote of the Day:
    The conventional view serves to protect us from the painful job of thinking.
    --John Kenneth Galbraith
  • Never doubt thy debugger

    Something you need to know before start debugging


    It may appear as a contradiction after my previous post, but the first thing to do to start analyzing a memory dump is ask yourself: do I really need a dump?!? smile_omg

    Let me explain: when you need to troubleshoot an error there are a number of things to do before really going down the dump path, simply because not all problems can be resolved in that way... keep in mind that a dump is nothing more than a snapshot of a process at a certain point in time, we can try to understand what happened in the past but with some limitations (as long as the details we are looking for are still in memory), and of course we can't know what happened to the process after the dump has been taken; exactly like a picture you can take with your digital camera. For example if you are having problems to remotely debug your application, I hardly think a dump can add any value to your troubleshooting... while in case of a memory leak a dump is one of the first things I ask the customer to provide me (but again there are some things before this step).

    What's first, then? Well, as you can guess, the first step is to understand the problem and know the scenario where it reproduces. If you are troubleshooting your own application you should already know most of the details, but if you are a consultant and are helping one of your customers with a weird exception thrown one in a while, you must have an open an ongoing discussion with the people whom developed the application, and maybe with the IT pros whom are maintaining the application and the environment day by day.

    Let's assume we have the information we need to start, and we decided we need to capture a dump. But which kind? How? When? Moreover, are you sure you and your customer are talking the same language and using the same terms to name things? I tell you because I learnt this lesson on my own in the hard way... the customer was describing a crash in his application so we configured adplus to run in crash mode, but some some reason we were unable to get a dump when the crash reproduced; we kept trying, but after 3-4 runs we gave up. Finally it turned out that the crash the customer was reporting was "just" an exception not handled in a try...catch block shown to the final user (do you know the yellow/orange ASP.NET error page?) but the worker process was still happily serving requests for other users... smile_thinking

    Some terminology

    So, here is some basic terminology: if your customer is not expert in this area, assure he understands those terms and stick to them to avoid confusion. This still apply if you'll ever need to raise a support call with Microsoft CSS, this is the terminology you can expect to be used

    • crash: this refers to a process which for some reason (usually an unhandled exception) is terminated by the operating system. How to be sure? Check the TaskManger when the problem occurs, and if the process gets a new PID, it has been recycled. And check your event log: usually you'll have a message like "process xyz terminated unexpectedly"
    • hang: the application reach a status where it's unable to continue serve incoming requests (and maybe the users are getting a "server too busy" error) but the process does not crash. In such a situation the target process could simply sit there in memory doing nothing, and you have to restart it manually to restore normal application activities. Note that IIS has a mechanism to automatically detect the status of its worker processes, and if one of them for some reason does not respond to regular pings, after a certain timeout elapses IIS assumes the process is hanging and recycles it. In this latter case the symptom may looks like a crash, but it really isn't and you can tell because you'll not have the "process xyz terminated unexpectedly" message, but rather you'll have something like "a process serving application pool xyz has failed to respond to a ping"
    • deadlock: imagine thread 1 in your application has acquired a lock on resource A (a handle, a socket etc...) but to complete its work must also access at the same time resource B; now imagine this resource B is locked by thread 2 which in turn is waiting for resource A (remember it's locked by threads 1?)... we have a deadlock (is the same concept as the circular reference in a Excel sheet) because this situation does not have a solution, unless one of the two threads finally times out and release the resource it was locking
    • leak: we have a memory leak when a process keeps growing over time and never releases back the memory to the operating system, until it eventually throws an OutOfMemoryException and it finally crashes. In this case capturing a dump when the process is being terminated is almost certainly too late, so it's better to capture a manual dump when the process is approaching it's size limit, but before the actual crash. By the way, a leak can take only 5 minutes to cause the process to crash, or it might take some days; but the pattern is always the same, as the OOM exception and the crash at the end. The smaller (and slower) the leak, the more difficult will be to find the culprit(s) of the problem...

    Crash or Hang dump?

    So, now that we gathered all those details about the problem, which is the right approach to capture the dump we need? It depends on the problem, of course. There may be some variations depending on the circumstances, but basically we can capture either a crash or hang dump. What's the difference?

    We'll need a crash dump when we can't determine when the problem (typically a crash like the name implies, but that's not the only case) will happen, so we can configure the debugger in advance to monitor our target process and capture a dump when the process will be terminated, or when we need to capture a dump on a specific exception (as I'll discuss in another post). On the other hand, we can capture a hang dump after the problem has occurred but the process is still in memory, for example in a memory leak scenario but also when a process is burning our your CPU.

    First or second chance?

    As the name suggests, exceptions should be the exception rather than the rule; so for example it's always a good idea to check if an object is valid before trying to use it, rather than let the runtime throw an exception and trap it in a try...catch block. Anyway what happens when you have a debugger attached to a process? The debugger gets the first chance to handle the exception; If it allows the execution to continue and does not handle the exception, the application will see the exception as usual. If the application does not handle the exception, the debugger gets a second chance to see the exception; in this case the application would normally crash if the debugger was not present.

    This is more clear when you use adplus in crash mode: by default the debugger attaches to the target process and logs every exception thrown; if you try you'll very likely end up with quite a few minidumps (a few megabytes each) corresponding to every exception thrown and trapped in try...catch blocks in your code, and a second chance full dump (the same size as the process, the private bytes value you can see in TaskManager) when the process will crash.

    How much does this cost?

    I mean in terms of performance for your server which could potentially be a highly stressed production environment? Of course there is a cost, especially for a crash dump because you'll have a debugger attached to your worker process for the time needed to reproduce the problem, but it's hard to exactly tell how much; in my experience I just had one server where the debugger was really affecting the site and forced us to stop it. But it worth mentioning that the server was already beyond its capacity limit and was already performing badly, the debugger was just the last straw...

    Having a repro in a test environment is the ideal situation, since we'll be able to capture dumps, run tests and do whatever needed to resolve the problem without causing additional pain to your poor users.



    Quote of the Day:
    keep it real/keep it clean/keep it simple
Page 7 of 13 (187 items) «56789»