This has been an interesting case where we had an ASP.NET 2.0 application which under load was completely blocked after a few minutes. Since we were talking about a hang/deadlock (as reported from the customer), the !critlist command (you can find the command within the SieExtPub.dll extension) is a good start:
0:021> !critlist CritSec at 7a393800. Owned by thread 21. Waiting Threads: 6 7 8 9 10 11 12 13 42 72 104 107 108 109 110 111 112 113 114 115 116 CritSec at e194c. Owned by thread 32. Waiting Threads: 21
Thread 32 holds a critical section and thread 21 is waiting on it, but thread 21 is also owning another critical section, and about 20 other threads are waiting there… so apparently this is not a real deadlock (21 is waiting on 32 but 32 is not waiting on 21); but what is thread 32 doing?
0:032> kpL2000 ChildEBP RetAddr 0327fd30 7c822124 ntdll!KiFastSystemCallRet 0327fd34 77e6baa8 ntdll!NtWaitForSingleObject+0xc 0327fda4 79e718fd kernel32!WaitForSingleObjectEx+0xac 0327fde8 79e718c6 mscorwks!PEImage::LoadImage+0x199 0327fe38 79e7187c mscorwks!CLREvent::WaitEx+0x117 0327fe48 7a0e288e mscorwks!CLREvent::Wait+0x17 0327fe9c 7a086e76 mscorwks!Thread::SysSuspendForGC+0x52a 0327ff88 7a0d867b mscorwks!SVR::GCHeap::SuspendEE+0x16c 0327ffa8 7a0d8987 mscorwks!SVR::gc_heap::gc_thread_function+0x3b 0327ffb8 77e66063 mscorwks!SVR::gc_heap::gc_thread_stub+0x9b 0327ffec 00000000 kernel32!BaseThreadStart+0x34
It is waiting to start GC but it cannot, because thread 95 has PreEmptive disabled:
0:032> !threads ThreadCount: 69 UnstartedThread: 0 BackgroundThread: 69 PendingThread: 0 DeadThread: 0 Hosted Runtime: yes PreEmptive GC Alloc Lock ID OSID ThreadOBJ State GC Context Domain Count APT Exception 23 1 1288 0012efb8 1808220 Enabled 1048ebfc:1048efe8 0011f578 0 MTA (Threadpool Worker) [...] 95 35 1114 05424c30 180b222 Disabled 16577334:16579008 00162f38 1 MTA (Threadpool Worker) 96 36 1b54 05407bf8 180b220 Enabled 145124ec:14512fe8 0011f578 0 MTA (Threadpool Worker) [...]
As you might know, if a thread has PreEmptive GC disabled is because it does not want to be interrupted by the GC; loading a file is a good reason to disable PreEmptive and that’s exactly the case:
0:095> !clrstack OS Thread Id: 0x1114 (95) ESP EIP 05e5ecd0 7c82ed54 [HelperMethodFrame_PROTECTOBJ: 05e5ecd0] System.Reflection.Assembly.nLoadFile(System.String, System.Security.Policy.Evidence) 05e5ef88 793f583f System.Reflection.Assembly.LoadFile(System.String) 05e5ef9c 04674f82 MyApp.Core.Lib.Managers.LangMan.ResourceFile(System.String, System.String, MyApp.Core.Lib.Managers.ResourceMode) 05e5efd8 04674f0c MyApp.Core.Lib.Managers.LangMan.ResourceFile(System.String, System.String) 05e5efe0 04674ee5 MyApp.FPCore.Pages.Page.ResourceFile(System.String, System.String) 05e5efec 04674d69 FP_Login.formslogin.Page_Load(System.Object, System.EventArgs) 05e5f028 031bb2b5 [MulticastFrame: 05e5f028] System.EventHandler.Invoke(System.Object, System.EventArgs) [...]
!showstringw from SieExtPub on the first parameter passed to mscorwks!ExplicitBind we can get the name of the file we’re trying to load:
0:095> kb ChildEBP RetAddr Args to Child 05e5e860 7c822124 77e6baa8 00001348 00000000 ntdll!KiFastSystemCallRet 05e5e864 77e6baa8 00001348 00000000 00000000 ntdll!NtWaitForSingleObject+0xc 05e5e8d4 79e718fd 00001348 ffffffff 00000000 kernel32!WaitForSingleObjectEx+0xac 05e5e918 79e718c6 00001348 ffffffff 00000000 mscorwks!PEImage::LoadImage+0x199 05e5e968 79e7187c ffffffff 00000000 00000000 mscorwks!CLREvent::WaitEx+0x117 05e5e978 79f90f81 ffffffff 00000000 00000000 mscorwks!CLREvent::Wait+0x17 05e5e988 79f917ba 7a390970 03c55358 ffffffff mscorwks!CExecutionEngine::WaitForEvent+0x37 05e5e99c 7a15cfce 03c55358 ffffffff 00000000 mscorwks!ClrWaitEvent+0x17 05e5e9ac 7a153339 05e5ed7c 00000000 00000000 mscorwks!CSimpleFusionBindSink::Wait+0xe 05e5e9c8 7a1545d2 03cecd70 0016e3d0 00000200 mscorwks!BindHelper+0x7d 05e5ec64 7a2d0c7c 05e5ed7c 0016e3d0 00000000 mscorwks!ExplicitBind+0x1a6 05e5ef80 793f583f 00000000 16577088 1657703c mscorwks!AssemblyNative::LoadFile+0x1de [...] 0:095> !showstringw 05e5ed7c D:\MyApp\App\Members\bin\DescMembers.dll
Looking around in the stack it’s interesting to note that also thread 81 was loading the same file from from ASP.NET temporary folder:
0:081> kb ChildEBP RetAddr Args to Child 0573cf58 7c822124 77e6baa8 000004f4 00000000 ntdll!KiFastSystemCallRet 0573cf5c 77e6baa8 000004f4 00000000 00000000 ntdll!NtWaitForSingleObject+0xc 0573cfcc 79e718fd 000004f4 ffffffff 00000000 kernel32!WaitForSingleObjectEx+0xac 0573d010 79e718c6 000004f4 ffffffff 00000000 mscorwks!PEImage::LoadImage+0x199 0573d060 79e7187c ffffffff 00000000 00000000 mscorwks!CLREvent::WaitEx+0x117 0573d070 7a0851cb ffffffff 00000000 00000000 mscorwks!CLREvent::Wait+0x17 0573d080 79f40e96 00000000 5ed28252 79eaebea mscorwks!SVR::GCHeap::WaitUntilGCComplete+0x32 0573d0bc 79e797c6 5ed2822a 79eaebea 053ee59a mscorwks!Thread::RareDisablePreemptiveGC+0x1a1 0573d0e8 79ea91b2 7a3879e0 5ed283e6 79eaebea mscorwks!CrstBase::AcquirePreempLock+0x2e 0573d124 79eb4132 0573d154 00000000 0573d1b4 mscorwks!PEImage::OpenImage+0x92 0573d1c0 79eb3d78 053ee598 0573d1f8 0573d1f4 mscorwks!RuntimeOpenImageInternal+0xcd 0573d208 79eb3cee 053ee598 79eaebb0 00000000 mscorwks!GetAssemblyMDInternalImportEx+0x9d 0573d21c 79ecf4b9 053ee598 00000000 0573d238 mscorwks!CreateMetaDataImport+0x16 0573d23c 79ecf462 053ee598 5ed280a6 00000000 mscorwks!CAssemblyManifestImport::Init+0x35 0573d264 79f2e637 053ee598 0573d27c 000f2a20 mscorwks!CreateAssemblyManifestImport+0x53 0573d280 7a161a1d 053ee598 03d09c00 0573d9a8 mscorwks!CreateAssemblyFromManifestFile+0x48 0573d930 7a161e12 0573dcc4 03d09c00 0573d9a8 mscorwks!CAsmDownloadMgr::CreateAssembly+0x8e2 0573d96c 7a162550 0573dcc4 03d09c00 0573d9a8 mscorwks!CAsmDownloadMgr::DoSetupPushToCache+0x50 0573dbf4 79f90e65 000f2a20 03d09c00 0573dcc4 mscorwks!CAsmDownloadMgr::DoSetup+0x26a 0573dc40 79f8f2cc 0573dcc4 00000000 5ed28eb6 mscorwks!CAssemblyDownload::DoSetup+0x7b 0:081> !showstringw 053ee598 C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\_mem_bin\bf10f877\41cee659\assembly\dl3\9cf10f5d\0090cb26_8229c601\DescMembers.DLL 0:081> !showstringw 0573dcc4 D:\MyApp\App\Members\bin\DescMembers.dll
Another set of dumps helped to further clarify the situation: thread 94 holds the lock and is trying to load a previously compiled assembly from disk. It is trying to load this in relation to page D:\MyApp\App\Secure\Root\default.aspx and this this is leading to the attempt to load the assembly for global.asax: C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\App_global.asax.ylfot7zw.dll. As part of this assembly load, thread 94 is blocked waiting to acquire a critical section owned by thread 123:
0:094> !syncblk Index SyncBlock MonitorHeld Recursion Owning Thread Info SyncBlock Owner 140 03f05a84 13 1 03ebd208 714 94 1225a688 System.Web.Compilation.BuildManager ----------------------------- Total 219 CCW 11 RCW 0 ComClassFactory 0 Free 0 0:094> !dso OS Thread Id: 0x714 (94) ESP/REG Object Name 04f7e9e0 790d6654 System.String 04f7eb70 1c571e2c System.Byte[] 04f7ecb8 1c571df0 System.Reflection.AssemblyName 04f7eccc 1c571df0 System.Reflection.AssemblyName 04f7edac 1c571df0 System.Reflection.AssemblyName 04f7edc4 1c571df0 System.Reflection.AssemblyName 04f7edf8 1c571ad0 System.String C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\App_global.asax.ylfot7zw.dll.delete 04f7ee00 1c47a640 System.Object[] (System.Reflection.AssemblyName[]) 04f7ee04 1c5715e8 System.String C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\App_global.asax.ylfot7zw.dll 04f7ee24 1c4775e0 System.Reflection.Assembly 04f7ee28 1c468e58 System.String App_Web_gf9hosur 04f7ee2c 1c428b70 System.Web.Compilation.PreservationFileReader 04f7ee30 121a2004 System.Globalization.CultureInfo 04f7ee34 1c468dbc System.String 110000 04f7ee44 1c1f7e58 System.Web.VirtualPath 04f7ee48 1c428b70 System.Web.Compilation.PreservationFileReader 04f7ee4c 1c4693bc System.Web.Compilation.BuildResultCompiledTemplateType 04f7ee58 1c1f7e58 System.Web.VirtualPath 04f7ee5c 1c428b70 System.Web.Compilation.PreservationFileReader 04f7ee60 1c4693bc System.Web.Compilation.BuildResultCompiledTemplateType 04f7ee7c 1c468cbc System.String fffffff8697ef330 04f7ee84 1c428a68 System.String C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\default.aspx.8cd6fbf4.compiled 04f7ee88 1c428a68 System.String C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\default.aspx.8cd6fbf4.compiled 04f7ee8c 1c1f7e58 System.Web.VirtualPath 04f7ee9c 1c428a68 System.String C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\default.aspx.8cd6fbf4.compiled 04f7eea4 1c428b70 System.Web.Compilation.PreservationFileReader 04f7eea8 1a200b5c System.String .compiled 04f7eeac 1c428970 System.String C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\default.aspx.8cd6fbf4 04f7eeb8 1a200b5c System.String .compiled 04f7eebc 1c428a68 System.String C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\default.aspx.8cd6fbf4.compiled 04f7eec0 1c428b70 System.Web.Compilation.PreservationFileReader 04f7eec4 1225b41c System.Web.Compilation.StandardDiskBuildResultCache 04f7eec8 1c1f7e58 System.Web.VirtualPath 04f7eed8 1c428a68 System.String C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\default.aspx.8cd6fbf4.compiled 04f7eee8 1c1f7e58 System.Web.VirtualPath 04f7eef8 1c1f7e58 System.Web.VirtualPath 04f7ef00 1225a688 System.Web.Compilation.BuildManager 04f7ef08 1c427124 System.String default.aspx.8cd6fbf4 04f7ef0c 1225a7c4 System.Web.Util.SimpleRecyclingCache 04f7ef20 1c427100 System.String 8cd6fbf4 04f7ef2c 1c1f7e58 System.Web.VirtualPath 04f7ef30 1225a688 System.Web.Compilation.BuildManager 04f7ef44 1c1f7e58 System.Web.VirtualPath 04f7ef50 1225a688 System.Web.Compilation.BuildManager 04f7ef54 1c1f7e58 System.Web.VirtualPath 04f7ef70 1c418fb8 System.Web.ApplicationImpersonationContext 04f7ef74 1c418fb8 System.Web.ApplicationImpersonationContext 04f7ef78 1c418fb8 System.Web.ApplicationImpersonationContext 04f7ef88 1c1f7e58 System.Web.VirtualPath 04f7ef9c 1c418fb8 System.Web.ApplicationImpersonationContext 04f7efa8 1c418fb8 System.Web.ApplicationImpersonationContext 04f7efac 1c1f7e58 System.Web.VirtualPath 04f7efc4 1c418fb8 System.Web.ApplicationImpersonationContext 04f7efcc 1c418f88 System.Object[] (System.Type[]) 04f7efd0 1c418fac System.Web.UI.PageHandlerFactory 04f7efe4 1c1f7e58 System.Web.VirtualPath 04f7efe8 1c1f7c2c System.Web.HttpContext 04f7f000 1c1f7c2c System.Web.HttpContext 04f7f004 1444b800 System.RuntimeType 04f7f024 1239e718 System.String System.Web.UI.PageHandlerFactory 04f7f02c 1c393abc System.Collections.Hashtable 04f7f034 1c1f73a4 System.String D:\MyApp\App\Secure\Root\default.aspx 04f7f038 1c1f7c2c System.Web.HttpContext 04f7f03c 1c1f7e58 System.Web.VirtualPath 04f7f050 1c1f7c2c System.Web.HttpContext 04f7f054 1c418fac System.Web.UI.PageHandlerFactory 04f7f058 1c1f7c2c System.Web.HttpContext 04f7f060 1c1f73a4 System.String D:\MyApp\App\Secure\Root\default.aspx 04f7f064 1c1f7e58 System.Web.VirtualPath 04f7f068 1c1f7338 System.String POST 04f7f06c 1c418fac System.Web.UI.PageHandlerFactory 04f7f070 1c4187b0 System.Web.ApplicationImpersonationContext 04f7f074 1c1f87b8 ASP.global_asax 04f7f07c 1c1f73a4 System.String D:\MyApp\App\Secure\Root\default.aspx 04f7f080 1c1f7e58 System.Web.VirtualPath 04f7f084 1c1f7338 System.String POST 04f7f088 1c4187b0 System.Web.ApplicationImpersonationContext 04f7f090 1c1f7c2c System.Web.HttpContext 04f7f0a4 1c1f87b8 ASP.global_asax 04f7f0a8 1c1f87b8 ASP.global_asax 04f7f0ac 1c1f7cd4 System.Web.HttpRequest 04f7f0b0 1c1f7c2c System.Web.HttpContext 04f7f0c0 1c1f73a4 System.String D:\MyApp\App\Secure\Root\default.aspx 04f7f0c4 1c1f7e58 System.Web.VirtualPath 04f7f0c8 1c1f7338 System.String POST 04f7f0cc 1c1f7c2c System.Web.HttpContext 04f7f0d0 1c1f87b8 ASP.global_asax 04f7f0e8 1c411f58 System.Web.HttpApplication+MapHandlerExecutionStep 04f7f0ec 1c1f87b8 ASP.global_asax 04f7f0f0 1c1f7c2c System.Web.HttpContext 04f7f104 1c1f7c2c System.Web.HttpContext 04f7f108 1c1f87b8 ASP.global_asax 04f7f11c 1c1f87b8 ASP.global_asax 04f7f120 1c412258 System.Web.HttpApplication+ThreadContext 04f7f124 1c1f7c2c System.Web.HttpContext 04f7f128 1c1f87b8 ASP.global_asax 04f7f12c 1c412238 System.Web.AspNetSynchronizationContext 04f7f134 1c411cdc System.Web.HttpApplication+ApplicationStepManager 04f7f158 1c1f7c2c System.Web.HttpContext 04f7f15c 1c412218 System.Web.HttpAsyncResult 04f7f160 1c1f87b8 ASP.global_asax 04f7f164 121c489c System.AsyncCallback 04f7f170 1c1f4ff4 System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6 04f7f174 1c1f87b8 ASP.global_asax 04f7f178 1c1f7e6c System.Web.HttpWriter 04f7f180 1c1f7c2c System.Web.HttpContext 04f7f184 121c489c System.AsyncCallback 04f7f188 1c1f7c2c System.Web.HttpContext 04f7f18c 1c1f4ff4 System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6 04f7f190 121c4138 System.Web.HttpRuntime 04f7f1a8 1c1f72dc System.String D:\MyApp\App\Secure\ 04f7f1ac 1c1f4ff4 System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6 04f7f1bc 121a1038 System.String D:\MyApp\App\Secure\ 04f7f1c0 1c1f4ff4 System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6 04f7f1f8 122f2880 System.Web.Hosting.ISAPIRuntime 0:094> !do 1c1f4ff4 Name: System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6 MethodTable: 663a6f84 EEClass: 663a6f04 Size: 256(0x100) bytes (C:\WINDOWS\assembly\GAC_32\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll) Fields: MT Field Offset Type VT Attr Value Name 79105fac 400133a 4 System.DateTime 1 instance 1c1f4ff8 _startTime 7910ead0 400133b c System.Guid 1 instance 1c1f5000 _traceId 7912ab5c 400133c 194 System.String[][] 0 shared static s_HTTPStatusDescriptions >> Domain:Value 001171e8:NotInit 00159f68:1a227064 03ea3b78:122f6384 << 79124304 400133d 198 System.Object[] 0 shared static s_serverVarFromRequestHeaderNames >> Domain:Value 001171e8:NotInit 00159f68:1a2271ac 03ea3b78:122f64cc << 79124304 400133e 19c System.Object[] 0 shared static s_requestHeaderNames >> Domain:Value 001171e8:NotInit 00159f68:1a22725c 03ea3b78:122f657c << 79124304 400133f 1a0 System.Object[] 0 shared static s_responseHeaderNames >> Domain:Value 001171e8:NotInit 00159f68:1a22730c 03ea3b78:122f662c << 790feb48 4001340 1a4 ...ections.Hashtable 0 shared static s_requestHeadersLoookupTable >> Domain:Value 001171e8:NotInit 00159f68:1a227394 03ea3b78:122f66b4 << 790feb48 4001341 1a8 ...ections.Hashtable 0 shared static s_responseHeadersLoookupTable >> Domain:Value 001171e8:NotInit 00159f68:1a22745c 03ea3b78:122f677c << 790fe238 4001b3a 70 System.IntPtr 1 instance 3066598 _ecb 790fe238 4001b3b 74 System.IntPtr 1 instance 2d0 _token 7910ead0 4001b3c a8 System.Guid 1 instance 1c1f509c _traceId 790fa4b0 4001b3d 1c System.String 0 instance 1c1f7338 _method 790fa4b0 4001b3e 20 System.String 0 instance 1c1f7354 _path 790fa4b0 4001b3f 24 System.String 0 instance 1c1f7354 _filePath 790fa4b0 4001b40 28 System.String 0 instance 790d6654 _pathInfo 790fa4b0 4001b41 2c System.String 0 instance 1c1f73a4 _pathTranslated 790fa4b0 4001b42 30 System.String 0 instance 141a4b28 _appPath 790fa4b0 4001b43 34 System.String 0 instance 1c1f72dc _appPathTranslated [...] 0:094> !do 1c1f73a4 Name: System.String MethodTable: 790fa4b0 EEClass: 790fa410 Size: 150(0x96) bytes (C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll) String: D:\MyApp\App\Secure\Root\default.aspx Fields: MT Field Offset Type VT Attr Value Name 790fedf4 4000096 4 System.Int32 1 instance 67 m_arrayLength 790fedf4 4000097 8 System.Int32 1 instance 66 m_stringLength 790fbfcc 4000098 c System.Char 1 instance 44 m_firstChar 790fa4b0 4000099 10 System.String 0 shared static Empty >> Domain:Value 001171e8:790d6654 00159f68:790d6654 03ea3b78:790d6654 << 7912474c 400009a 14 System.Char[] 0 shared static WhitespaceChars >> Domain:Value 001171e8:141a0624 00159f68:141a77f8 03ea3b78:121a2374 << 0:094> !do 1c1f87b8 Name: ASP.global_asax MethodTable: 04c35adc EEClass: 04ff1dd8 Size: 140(0x8c) bytes (C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\App_global.asax.ylfot7zw.dll) Fields: MT Field Offset Type VT Attr Value Name 663a2450 4000dc8 4 ...pApplicationState 0 instance 122e14dc _state 663a2750 4000dc9 8 ...m.Web.HttpContext 0 instance 00000000 _initContext 663a359c 4000dca c ...b.HttpAsyncResult 0 instance 1c412218 _ar 663a3394 4000dcb 10 ...pModuleCollection 0 instance 1c3a648c _moduleCollection 79118214 4000dcc 14 ...ncipal.IPrincipal 0 instance 1c413d68 _savedPrincipal 79105020 4000dcd 78 System.Boolean 1 instance 1 _restorePrincipal 7a74da98 4000de8 18 ....EventHandlerList 0 instance 1c3f7408 _events 663a2aec 4000de9 1c ...ventHandlersTable 0 instance 1c401980 _asyncEvents [...] 0:094> !critlist CritSec at 7a38ee88. Owned by thread 123. Waiting Threads: 26 66 94
0:094> !syncblk Index SyncBlock MonitorHeld Recursion Owning Thread Info SyncBlock Owner 140 03f05a84 13 1 03ebd208 714 94 1225a688 System.Web.Compilation.BuildManager ----------------------------- Total 219 CCW 11 RCW 0 ComClassFactory 0 Free 0 0:094> !dso OS Thread Id: 0x714 (94) ESP/REG Object Name 04f7e9e0 790d6654 System.String 04f7eb70 1c571e2c System.Byte[] 04f7ecb8 1c571df0 System.Reflection.AssemblyName 04f7eccc 1c571df0 System.Reflection.AssemblyName 04f7edac 1c571df0 System.Reflection.AssemblyName 04f7edc4 1c571df0 System.Reflection.AssemblyName 04f7edf8 1c571ad0 System.String C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\App_global.asax.ylfot7zw.dll.delete 04f7ee00 1c47a640 System.Object[] (System.Reflection.AssemblyName[]) 04f7ee04 1c5715e8 System.String C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\App_global.asax.ylfot7zw.dll 04f7ee24 1c4775e0 System.Reflection.Assembly 04f7ee28 1c468e58 System.String App_Web_gf9hosur 04f7ee2c 1c428b70 System.Web.Compilation.PreservationFileReader 04f7ee30 121a2004 System.Globalization.CultureInfo 04f7ee34 1c468dbc System.String 110000 04f7ee44 1c1f7e58 System.Web.VirtualPath 04f7ee48 1c428b70 System.Web.Compilation.PreservationFileReader 04f7ee4c 1c4693bc System.Web.Compilation.BuildResultCompiledTemplateType 04f7ee58 1c1f7e58 System.Web.VirtualPath 04f7ee5c 1c428b70 System.Web.Compilation.PreservationFileReader 04f7ee60 1c4693bc System.Web.Compilation.BuildResultCompiledTemplateType 04f7ee7c 1c468cbc System.String fffffff8697ef330 04f7ee84 1c428a68 System.String C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\default.aspx.8cd6fbf4.compiled 04f7ee88 1c428a68 System.String C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\default.aspx.8cd6fbf4.compiled 04f7ee8c 1c1f7e58 System.Web.VirtualPath 04f7ee9c 1c428a68 System.String C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\default.aspx.8cd6fbf4.compiled 04f7eea4 1c428b70 System.Web.Compilation.PreservationFileReader 04f7eea8 1a200b5c System.String .compiled 04f7eeac 1c428970 System.String C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\default.aspx.8cd6fbf4 04f7eeb8 1a200b5c System.String .compiled 04f7eebc 1c428a68 System.String C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\default.aspx.8cd6fbf4.compiled 04f7eec0 1c428b70 System.Web.Compilation.PreservationFileReader 04f7eec4 1225b41c System.Web.Compilation.StandardDiskBuildResultCache 04f7eec8 1c1f7e58 System.Web.VirtualPath 04f7eed8 1c428a68 System.String C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\default.aspx.8cd6fbf4.compiled 04f7eee8 1c1f7e58 System.Web.VirtualPath 04f7eef8 1c1f7e58 System.Web.VirtualPath 04f7ef00 1225a688 System.Web.Compilation.BuildManager 04f7ef08 1c427124 System.String default.aspx.8cd6fbf4 04f7ef0c 1225a7c4 System.Web.Util.SimpleRecyclingCache 04f7ef20 1c427100 System.String 8cd6fbf4 04f7ef2c 1c1f7e58 System.Web.VirtualPath 04f7ef30 1225a688 System.Web.Compilation.BuildManager 04f7ef44 1c1f7e58 System.Web.VirtualPath 04f7ef50 1225a688 System.Web.Compilation.BuildManager 04f7ef54 1c1f7e58 System.Web.VirtualPath 04f7ef70 1c418fb8 System.Web.ApplicationImpersonationContext 04f7ef74 1c418fb8 System.Web.ApplicationImpersonationContext 04f7ef78 1c418fb8 System.Web.ApplicationImpersonationContext 04f7ef88 1c1f7e58 System.Web.VirtualPath 04f7ef9c 1c418fb8 System.Web.ApplicationImpersonationContext 04f7efa8 1c418fb8 System.Web.ApplicationImpersonationContext 04f7efac 1c1f7e58 System.Web.VirtualPath 04f7efc4 1c418fb8 System.Web.ApplicationImpersonationContext 04f7efcc 1c418f88 System.Object[] (System.Type[]) 04f7efd0 1c418fac System.Web.UI.PageHandlerFactory 04f7efe4 1c1f7e58 System.Web.VirtualPath 04f7efe8 1c1f7c2c System.Web.HttpContext 04f7f000 1c1f7c2c System.Web.HttpContext 04f7f004 1444b800 System.RuntimeType 04f7f024 1239e718 System.String System.Web.UI.PageHandlerFactory 04f7f02c 1c393abc System.Collections.Hashtable 04f7f034 1c1f73a4 System.String D:\MyApp\App\Secure\Root\default.aspx 04f7f038 1c1f7c2c System.Web.HttpContext 04f7f03c 1c1f7e58 System.Web.VirtualPath 04f7f050 1c1f7c2c System.Web.HttpContext 04f7f054 1c418fac System.Web.UI.PageHandlerFactory 04f7f058 1c1f7c2c System.Web.HttpContext 04f7f060 1c1f73a4 System.String D:\MyApp\App\Secure\Root\default.aspx 04f7f064 1c1f7e58 System.Web.VirtualPath 04f7f068 1c1f7338 System.String POST 04f7f06c 1c418fac System.Web.UI.PageHandlerFactory 04f7f070 1c4187b0 System.Web.ApplicationImpersonationContext 04f7f074 1c1f87b8 ASP.global_asax 04f7f07c 1c1f73a4 System.String D:\MyApp\App\Secure\Root\default.aspx 04f7f080 1c1f7e58 System.Web.VirtualPath 04f7f084 1c1f7338 System.String POST 04f7f088 1c4187b0 System.Web.ApplicationImpersonationContext 04f7f090 1c1f7c2c System.Web.HttpContext 04f7f0a4 1c1f87b8 ASP.global_asax 04f7f0a8 1c1f87b8 ASP.global_asax 04f7f0ac 1c1f7cd4 System.Web.HttpRequest 04f7f0b0 1c1f7c2c System.Web.HttpContext 04f7f0c0 1c1f73a4 System.String D:\MyApp\App\Secure\Root\default.aspx 04f7f0c4 1c1f7e58 System.Web.VirtualPath 04f7f0c8 1c1f7338 System.String POST 04f7f0cc 1c1f7c2c System.Web.HttpContext 04f7f0d0 1c1f87b8 ASP.global_asax 04f7f0e8 1c411f58 System.Web.HttpApplication+MapHandlerExecutionStep 04f7f0ec 1c1f87b8 ASP.global_asax 04f7f0f0 1c1f7c2c System.Web.HttpContext 04f7f104 1c1f7c2c System.Web.HttpContext 04f7f108 1c1f87b8 ASP.global_asax 04f7f11c 1c1f87b8 ASP.global_asax 04f7f120 1c412258 System.Web.HttpApplication+ThreadContext 04f7f124 1c1f7c2c System.Web.HttpContext 04f7f128 1c1f87b8 ASP.global_asax 04f7f12c 1c412238 System.Web.AspNetSynchronizationContext 04f7f134 1c411cdc System.Web.HttpApplication+ApplicationStepManager 04f7f158 1c1f7c2c System.Web.HttpContext 04f7f15c 1c412218 System.Web.HttpAsyncResult 04f7f160 1c1f87b8 ASP.global_asax 04f7f164 121c489c System.AsyncCallback 04f7f170 1c1f4ff4 System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6 04f7f174 1c1f87b8 ASP.global_asax 04f7f178 1c1f7e6c System.Web.HttpWriter 04f7f180 1c1f7c2c System.Web.HttpContext 04f7f184 121c489c System.AsyncCallback 04f7f188 1c1f7c2c System.Web.HttpContext 04f7f18c 1c1f4ff4 System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6 04f7f190 121c4138 System.Web.HttpRuntime 04f7f1a8 1c1f72dc System.String D:\MyApp\App\Secure\ 04f7f1ac 1c1f4ff4 System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6 04f7f1bc 121a1038 System.String D:\MyApp\App\Secure\ 04f7f1c0 1c1f4ff4 System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6 04f7f1f8 122f2880 System.Web.Hosting.ISAPIRuntime 0:094> !do 1c1f4ff4 Name: System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6 MethodTable: 663a6f84 EEClass: 663a6f04 Size: 256(0x100) bytes (C:\WINDOWS\assembly\GAC_32\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll) Fields: MT Field Offset Type VT Attr Value Name 79105fac 400133a 4 System.DateTime 1 instance 1c1f4ff8 _startTime 7910ead0 400133b c System.Guid 1 instance 1c1f5000 _traceId 7912ab5c 400133c 194 System.String[][] 0 shared static s_HTTPStatusDescriptions >> Domain:Value 001171e8:NotInit 00159f68:1a227064 03ea3b78:122f6384 << 79124304 400133d 198 System.Object[] 0 shared static s_serverVarFromRequestHeaderNames >> Domain:Value 001171e8:NotInit 00159f68:1a2271ac 03ea3b78:122f64cc << 79124304 400133e 19c System.Object[] 0 shared static s_requestHeaderNames >> Domain:Value 001171e8:NotInit 00159f68:1a22725c 03ea3b78:122f657c << 79124304 400133f 1a0 System.Object[] 0 shared static s_responseHeaderNames >> Domain:Value 001171e8:NotInit 00159f68:1a22730c 03ea3b78:122f662c << 790feb48 4001340 1a4 ...ections.Hashtable 0 shared static s_requestHeadersLoookupTable >> Domain:Value 001171e8:NotInit 00159f68:1a227394 03ea3b78:122f66b4 << 790feb48 4001341 1a8 ...ections.Hashtable 0 shared static s_responseHeadersLoookupTable >> Domain:Value 001171e8:NotInit 00159f68:1a22745c 03ea3b78:122f677c << 790fe238 4001b3a 70 System.IntPtr 1 instance 3066598 _ecb 790fe238 4001b3b 74 System.IntPtr 1 instance 2d0 _token 7910ead0 4001b3c a8 System.Guid 1 instance 1c1f509c _traceId 790fa4b0 4001b3d 1c System.String 0 instance 1c1f7338 _method 790fa4b0 4001b3e 20 System.String 0 instance 1c1f7354 _path 790fa4b0 4001b3f 24 System.String 0 instance 1c1f7354 _filePath 790fa4b0 4001b40 28 System.String 0 instance 790d6654 _pathInfo 790fa4b0 4001b41 2c System.String 0 instance 1c1f73a4 _pathTranslated 790fa4b0 4001b42 30 System.String 0 instance 141a4b28 _appPath 790fa4b0 4001b43 34 System.String 0 instance 1c1f72dc _appPathTranslated [...] 0:094> !do 1c1f73a4 Name: System.String MethodTable: 790fa4b0 EEClass: 790fa410 Size: 150(0x96) bytes (C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll) String: D:\MyApp\App\Secure\Root\default.aspx Fields: MT Field Offset Type VT Attr Value Name 790fedf4 4000096 4 System.Int32 1 instance 67 m_arrayLength 790fedf4 4000097 8 System.Int32 1 instance 66 m_stringLength 790fbfcc 4000098 c System.Char 1 instance 44 m_firstChar 790fa4b0 4000099 10 System.String 0 shared static Empty >> Domain:Value 001171e8:790d6654 00159f68:790d6654 03ea3b78:790d6654 << 7912474c 400009a 14 System.Char[] 0 shared static WhitespaceChars >> Domain:Value 001171e8:141a0624 00159f68:141a77f8 03ea3b78:121a2374 << 0:094> !do 1c1f87b8 Name: ASP.global_asax MethodTable: 04c35adc EEClass: 04ff1dd8 Size: 140(0x8c) bytes (C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\App_global.asax.ylfot7zw.dll) Fields: MT Field Offset Type VT Attr Value Name 663a2450 4000dc8 4 ...pApplicationState 0 instance 122e14dc _state 663a2750 4000dc9 8 ...m.Web.HttpContext 0 instance 00000000 _initContext 663a359c 4000dca c ...b.HttpAsyncResult 0 instance 1c412218 _ar 663a3394 4000dcb 10 ...pModuleCollection 0 instance 1c3a648c _moduleCollection 79118214 4000dcc 14 ...ncipal.IPrincipal 0 instance 1c413d68 _savedPrincipal 79105020 4000dcd 78 System.Boolean 1 instance 1 _restorePrincipal 7a74da98 4000de8 18 ....EventHandlerList 0 instance 1c3f7408 _events 663a2aec 4000de9 1c ...ventHandlersTable 0 instance 1c401980 _asyncEvents [...]
0:094> !critlist CritSec at 7a38ee88. Owned by thread 123. Waiting Threads: 26 66 94
!dso output already shows objects and strings in the same order as they are stored on the stack and executed, so it’s quite easy to spot interesting file paths, URLs etc… and often that is enough to suggest which files are involved, which page or web service you’re accessing and so on; of course you can always use !do <address> to dig into the stack to be 100% sure you’re on the right path.
Using the same commands above we found that thread 123 is trying to explicitly load "D:\MyApp\App\Secure\bin\secure.dll" whilst executing "ASP.logout_aspx"; after transitioning from managed to unmanaged code, this thread then gets blocked waiting for garbage collection to complete (because it needs to disable pre-emptive mode in order to acquire a particular lock).
In summary, the hang was due to a complicated deadlock situation involving “D:\MyApp\App\Secure\Root\default.aspx” being compiled by ASP.NET 2.0 compilation engine. This leads to an attempt to load “C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\App_global.asax.ylfot7zw.dll” from disk (the default.aspx page has a dependency on the global.asax) which in turn leads to an attempt to acquire a critical section owned by thread 123. Thread 123 is processing the request for “logout.aspx” which involves the execution of “MyApp.Core.Library.Managers.LanguageManager.ResourceFile” which leads to an attempt to load “D:\MyApp\App\Secure\bin\secure.dll” from disk. However this load operation gets blocked waiting for garbage collection to complete as it must acquire a particular lock when GC is not running. The GC was triggered (incidentally) by thread 56, but the GC has not actually commenced yet. The primary GC thread (29) has asked all thread executing managed code to get to a 'safe' point where GC can begin, but that has not happened (and will never happen) because some of those threads are blocked doing assembly load operations and are waiting on critical section.
Finally, the resolution: the customer confirmed the involved assemblies are “resource only” files which they access from everywhere in their quite complex application (basically those were localization assemblies they were explicitly loading when needed to localize their interface, error messages etc…); we strong-named those “sensitive” assemblies and installed them in the GAC: easy solution, when you know which is the problem…
By the way, there are a couple hotfixes available for this situation (see http://support.microsoft.com/kb/936757/en-us and 946644, KB not yet available but you can ask for the fix to CSS), but did not help in this case.
Carlo
PingBack from http://wordnew.acne-reveiw.info/?p=2029
Hello,
I found this article very interesting, I think I'm going through a similar problem but my worker process crashes rather than hang.
I'm relatively new to dump analysis and I'm stuck, may I ask you for your opinion?
Here is the stack:
0f21f36c 02f9b090 System.ArgumentOutOfRangeException
0f21f3b8 02f9b090 System.ArgumentOutOfRangeException
0f21f3c8 06bef6e4 System.String currentUser
0f21f3cc 02f9a2c8 System.Collections.ArrayList
0f21f3d0 02f99ac0 System.Web.SessionState.HttpSessionState
0f21f3e0 02f99ac0 System.Web.SessionState.HttpSessionState
0f21f3ec 06bef6e4 System.String currentUser
0f21f3fc 02f9b090 System.ArgumentOutOfRangeException
0f21f404 02f9a2c8 System.Collections.ArrayList
0f21f40c 02f99ac0 System.Web.SessionState.HttpSessionState
0f21f460 02f9b090 System.ArgumentOutOfRangeException
0f21f47c 02f9b090 System.ArgumentOutOfRangeException
0f21f480 02c76fa4 System.String index
0f21f48c 06c005a4 ASP.global_asax
0f21f494 06c03864 System.Web.SessionState.SessionStateModule
0f21f4a4 06c03864 System.Web.SessionState.SessionStateModule
0f21f4b0 06c03864 System.Web.SessionState.SessionStateModule
0f21f4b4 02c12e74 System.EventArgs
0f21f4c0 02c12e74 System.EventArgs
0f21f4c8 06c03864 System.Web.SessionState.SessionStateModule
0f21f4d4 06c03864 System.Web.SessionState.SessionStateModule
0f21f4dc 02f73d94 System.Web.HttpContext
0f21f500 06c03864 System.Web.SessionState.SessionStateModule
0f21f508 06c03864 System.Web.SessionState.SessionStateModule
0f21f524 02f73d94 System.Web.HttpContext
0f21f528 06c057e4 System.Web.BeginEventHandler
0f21f52c 06c09f90 System.Web.HttpApplication+AsyncEventExecutionStep
0f21f530 06c005a4 ASP.global_asax
0f21f534 06c005a4 ASP.global_asax
0f21f540 06c09fb4 System.AsyncCallback
0f21f544 02c12e74 System.EventArgs
0f21f54c 06bd8be0 System.AsyncCallback
0f21f550 06c005a4 ASP.global_asax
0f21f568 06c09f90 System.Web.HttpApplication+AsyncEventExecutionStep
0f21f56c 06c005a4 ASP.global_asax
0f21f570 06bd8988 System.Web.RequestTimeoutManager
0f21f580 02f740c0 System.Web.HttpApplication+ThreadContext
0f21f584 06bd8be0 System.AsyncCallback
0f21f588 06c005a4 ASP.global_asax
0f21f59c 06c005a4 ASP.global_asax
0f21f5a0 02f740a0 System.Web.AspNetSynchronizationContext
0f21f5a4 02f740c0 System.Web.HttpApplication+ThreadContext
0f21f5a8 02f73d94 System.Web.HttpContext
0f21f5ac 06c005a4 ASP.global_asax
0f21f5b4 06c09568 System.Web.HttpApplication+ApplicationStepManager
0f21f5d8 06bd8be0 System.AsyncCallback
0f21f5dc 02f74080 System.Web.HttpAsyncResult
0f21f5e0 06c005a4 ASP.global_asax
0f21f5e4 02f73d94 System.Web.HttpContext
0f21f5f0 02f738a0 System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6
0f21f5f4 06c005a4 ASP.global_asax
0f21f5f8 02f73fe8 System.Web.HttpWriter
0f21f600 02f73d94 System.Web.HttpContext
0f21f604 06bd8be0 System.AsyncCallback
0f21f608 02f73d94 System.Web.HttpContext
0f21f60c 02f738a0 System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6
0f21f610 06bd88d4 System.Web.HttpRuntime
0f21f62c 02f738a0 System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6
0f21f63c 06bc1a70 System.String D:\MyWebsite\www.mywebsite.com\
0f21f640 02f73a88 System.String D:\MyWebsite\www.mywebsite.com\
0f21f67c 02f738a0 System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6
0f21f680 02bfc304 System.Web.Hosting.ISAPIRuntime
0f21f6c0 02bfc304 System.Web.Hosting.ISAPIRuntime
0f21f814 02f6bf0c System.Threading.ExecutionContext
0f21f81c 02c1d27c System.Threading._TimerCallback
0f21f830 02c1d27c System.Threading._TimerCallback
0f21f834 02f6bf0c System.Threading.ExecutionContext
I think the problem is with the arguments sent to ASP.global_asax being "out of range" (I checked the values m_paramName and m_actualValue for System.ArgumentOutOfRangeException and they are "index" and 0).
I am trying to understand what should be my next step, how can I understand who's making that call with wrong arguments?
Thanks a lot,
Roberto.
Roberto, that seems to be the output of a !dumpstackobjects command, correct? If you use !clrstack you'll get more detailed information about the managed stack.
Apart from that, it's really hard to tell what's going wrong without having the change to look at a read dump... Have you considered opening a call with Microsoft Support? From your name I guess you're Italian (or maybe Spanish), are you? Either me or my colleagues will be happy to assist you on this matter.
HTH
Hey Carlo,
I hadn't noticed your name, I'm italian and I used to work for the MSPSS in Dublin.
I was thinking of opening a case with microsoft but I would have liked to solve this one on my own, you guys make it too easy.
I'll keep trying if I don't manage I'll give you a call.
Bye,
Roberto Santoro.
P.S.: do you have any interesting reading about IIS dump analysis (for dummies)?
Hi Roberto,
For dump analysis I can suggest to have a look at my blogroll where you can find blogs on debugging from some of my colleagues (Tess, Doug, Johan, Tom and the others) and some interesting sites as well.
Speaking of books, I read "Advanced Windows Debugging" and "Debugging Applications for Microsoft .NET and Microsoft Windows" which are good and useful, plus I've heard of "Memory Dump Analysis Anthology" (two volumes) but I have not read them myself.