@TessFerrandez
The other day I came across an issue where an ASP.NET site stopped responding and didn’t start serving requests again until the W3WP.exe process was restarted.
We grabbed some memory dumps with debug diag before restarting the process to see what was going on.
Debugging the dumps:
I opened the dump in windbg, and loaded up sos (.loadby sos mscorwks).
As with all hang dumps I usually check what the threads are doing (~* kb) and I found that one thread (10) was initiating a garbage collection.
This is really not all that uncommon since GCs happen quite frequently. What was interesting about this one though was that it was sitting in mscorwks!Thread::SysSuspendForGC which basically means that it is trying to suspend all the .net threads in order to get started with the garbage collection.
Whenever you see a thread stuck in this, you are almost always running into some GC deadlock condition.
To find out why we can threads can’t be suspended we can look at the !threads output to find out if any threads (other than the GC initiator) has preemptive GC disabled. (This means that it can’t be suspended)
From the !threads output we can see that in fact thread 26 has preemptive GC disabled. I’ve talked about this in earlier posts, but in short, some technologies like fusion (used to load assemblies) will disable GC preemption to avoid getting interrupted during some critical operations.
So what is this thread doing… the MyTools.MyAssemblyLoader constructor is trying to load up an assembly using Assembly.LoadFile, so that is why preemptive GC is disabled. This is usually a very quick operation but here it is stuck waiting on some event before it can finish.
0:026> !sos.clrstack OS Thread Id: 0x5230 (26) ESP EIP 0df0d200 7c8285ec [HelperMethodFrame_PROTECTOBJ: 0df0d200] System.Reflection.Assembly.nLoadFile(System.String, System.Security.Policy.Evidence) 0df0d4b8 793f583f System.Reflection.Assembly.LoadFile(System.String) 0df0d4cc 085f704d MyTools.MyAssemblyLoader..ctor(System.String) ... 0df0ebb0 083e85b0 MyAPP.Default.Page_Load(System.Object, System.EventArgs) 0df0ed10 698a1928 System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr, System.Object, System.Object, System.EventArgs) 0df0ed20 6629769f System.Web.Util.CalliEventHandlerDelegateProxy.Callback(System.Object, System.EventArgs) 0df0ed34 66143a84 System.Web.UI.Control.OnLoad(System.EventArgs) 0df0ed44 66143ad0 System.Web.UI.Control.LoadRecursive() 0df0ed58 66155106 System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean) 0df0ef10 66154a1b System.Web.UI.Page.ProcessRequest(Boolean, Boolean) 0df0ef48 66154967 System.Web.UI.Page.ProcessRequest() 0df0ef80 66154887 System.Web.UI.Page.ProcessRequestWithNoAssert(System.Web.HttpContext) 0df0ef88 6615481a System.Web.UI.Page.ProcessRequest(System.Web.HttpContext) 0df0ef9c 083e826e ASP.default_aspx.ProcessRequest(System.Web.HttpContext) 0df0efa8 65ff27d4 System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() 0df0efdc 65fc15b5 System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef) 0df0f01c 65fd32e0 System.Web.HttpApplication+ApplicationStepManager.ResumeSteps(System.Exception) 0df0f06c 65fc0225 System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest(System.Web.HttpContext, System.AsyncCallback, System.Object) 0df0f088 65fc550b System.Web.HttpRuntime.ProcessRequestInternal(System.Web.HttpWorkerRequest) 0df0f0bc 65fc5212 System.Web.HttpRuntime.ProcessRequestNoDemand(System.Web.HttpWorkerRequest) 0df0f0c8 65fc3587 System.Web.Hosting.ISAPIRuntime.ProcessRequest(IntPtr, Int32) 0df0f278 79f35ee8 [ContextTransitionFrame: 0df0f278] 0df0f2c8 79f35ee8 [GCFrame: 0df0f2c8] 0df0f420 79f35ee8 [ComMethodFrame: 0df0f420]
Looking around at other threads, we find another thread that is also loading up an assembly from the same location so this is probably the thread that will set the event so that thread 26 can continue.
0:020> !sos.clrstack OS Thread Id: 0x5dbc (20) ESP EIP 0993edb4 7c8285ec [HelperMethodFrame_PROTECTOBJ: 0993edb4] System.Reflection.Assembly.nLoadFile(System.String, System.Security.Policy.Evidence) 0993f06c 793f583f System.Reflection.Assembly.LoadFile(System.String) 0993f080 085f704d MyTools.MyAssemblyLoader..ctor(System.String) 0993f0b0 083e8612 MyAPP.Default.Page_Load(System.Object, System.EventArgs) 0993f210 698a1928 System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr, System.Object, System.Object, System.EventArgs) 0993f220 6629769f System.Web.Util.CalliEventHandlerDelegateProxy.Callback(System.Object, System.EventArgs) 0993f234 66143a84 System.Web.UI.Control.OnLoad(System.EventArgs) 0993f244 66143ad0 System.Web.UI.Control.LoadRecursive() 0993f258 66155106 System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean) 0993f410 66154a1b System.Web.UI.Page.ProcessRequest(Boolean, Boolean) 0993f448 66154967 System.Web.UI.Page.ProcessRequest() 0993f480 66154887 System.Web.UI.Page.ProcessRequestWithNoAssert(System.Web.HttpContext) 0993f488 6615481a System.Web.UI.Page.ProcessRequest(System.Web.HttpContext) 0993f49c 083e826e ASP.default_aspx.ProcessRequest(System.Web.HttpContext) 0993f4a8 65ff27d4 System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() 0993f4dc 65fc15b5 System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef) 0993f51c 65fd32e0 System.Web.HttpApplication+ApplicationStepManager.ResumeSteps(System.Exception) 0993f56c 65fc0225 System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest(System.Web.HttpContext, System.AsyncCallback, System.Object) 0993f588 65fc550b System.Web.HttpRuntime.ProcessRequestInternal(System.Web.HttpWorkerRequest) 0993f5bc 65fc5212 System.Web.HttpRuntime.ProcessRequestNoDemand(System.Web.HttpWorkerRequest) 0993f5c8 65fc3587 System.Web.Hosting.ISAPIRuntime.ProcessRequest(IntPtr, Int32) 0993f778 79f35ee8 [ContextTransitionFrame: 0993f778] 0993f7c8 79f35ee8 [GCFrame: 0993f7c8] 0993f920 79f35ee8 [ComMethodFrame: 0993f920]
The problem here is that this thread has enabled preemptive GC again and the GC has suspended it so that it is now waiting for the GC to finish before it can finish.
In other words we have a deadlock situation caused by a race condition with two threads loading the same dll at the same time:
Thread 10 – initiating GC – suspending all threads Thread 26 – can’t be suspended until it has finished loading the dll but is waiting for thread 20 to do so Thread 20 – waiting for the GC to finish to continue loading its dll
Solutions:
This issue could be resolved by putting a lock around the Assembly.Load in the MyTools.MyAssemblyLoader constructor.
However, this issue is actually fixed in this hotfix which is included in .NET 2.0 SP1 so if you are running into this you can install SP1 for .NET Framework 2.0 to fix the problem.
Have a good one,
Tess
Awesome insight into some .NET loader internals. Thanks!
Excellent resource. I first found your blog a couple of years back when I was trying to use WinDbg to isolate a resource leak in a WinForm app (combo box leaks handles when the auto-complete isn't terminated with a keystroke). Please keep writing. Everytime I come back to bang my head on one, it sinks a little deeper.
Do you do much with COM+?
We're having trouble finding a good resource for figuring out why sessions aren't tearing down as expected. Thanks again for all that you do.
Hi Scott,
Thanks for the nice words:)
I try to do as little COM as humanly possible, but we do have some good teams in support that do COM so if you're still looking for some help you might want to start a case. Http://support.microsoft.com has the details on how to create one...