This has been an interesting case where we had an ASP.NET 2.0 application which under load was completely blocked after a few minutes. Since we were talking about a hang/deadlock (as reported from the customer), the !critlist command (you can find the command within the SieExtPub.dll extension) is a good start:
0:021> !critlist CritSec at 7a393800. Owned by thread 21. Waiting Threads: 6 7 8 9 10 11 12 13 42 72 104 107 108 109 110 111 112 113 114 115 116 CritSec at e194c. Owned by thread 32. Waiting Threads: 21
Thread 32 holds a critical section and thread 21 is waiting on it, but thread 21 is also owning another critical section, and about 20 other threads are waiting there… so apparently this is not a real deadlock (21 is waiting on 32 but 32 is not waiting on 21); but what is thread 32 doing?
0:032> kpL2000 ChildEBP RetAddr 0327fd30 7c822124 ntdll!KiFastSystemCallRet 0327fd34 77e6baa8 ntdll!NtWaitForSingleObject+0xc 0327fda4 79e718fd kernel32!WaitForSingleObjectEx+0xac 0327fde8 79e718c6 mscorwks!PEImage::LoadImage+0x199 0327fe38 79e7187c mscorwks!CLREvent::WaitEx+0x117 0327fe48 7a0e288e mscorwks!CLREvent::Wait+0x17 0327fe9c 7a086e76 mscorwks!Thread::SysSuspendForGC+0x52a 0327ff88 7a0d867b mscorwks!SVR::GCHeap::SuspendEE+0x16c 0327ffa8 7a0d8987 mscorwks!SVR::gc_heap::gc_thread_function+0x3b 0327ffb8 77e66063 mscorwks!SVR::gc_heap::gc_thread_stub+0x9b 0327ffec 00000000 kernel32!BaseThreadStart+0x34
It is waiting to start GC but it cannot, because thread 95 has PreEmptive disabled:
0:032> !threads ThreadCount: 69 UnstartedThread: 0 BackgroundThread: 69 PendingThread: 0 DeadThread: 0 Hosted Runtime: yes PreEmptive GC Alloc Lock ID OSID ThreadOBJ State GC Context Domain Count APT Exception 23 1 1288 0012efb8 1808220 Enabled 1048ebfc:1048efe8 0011f578 0 MTA (Threadpool Worker) [...] 95 35 1114 05424c30 180b222 Disabled 16577334:16579008 00162f38 1 MTA (Threadpool Worker) 96 36 1b54 05407bf8 180b220 Enabled 145124ec:14512fe8 0011f578 0 MTA (Threadpool Worker) [...]
As you might know, if a thread has PreEmptive GC disabled is because it does not want to be interrupted by the GC; loading a file is a good reason to disable PreEmptive and that’s exactly the case:
0:095> !clrstack OS Thread Id: 0x1114 (95) ESP EIP 05e5ecd0 7c82ed54 [HelperMethodFrame_PROTECTOBJ: 05e5ecd0] System.Reflection.Assembly.nLoadFile(System.String, System.Security.Policy.Evidence) 05e5ef88 793f583f System.Reflection.Assembly.LoadFile(System.String) 05e5ef9c 04674f82 MyApp.Core.Lib.Managers.LangMan.ResourceFile(System.String, System.String, MyApp.Core.Lib.Managers.ResourceMode) 05e5efd8 04674f0c MyApp.Core.Lib.Managers.LangMan.ResourceFile(System.String, System.String) 05e5efe0 04674ee5 MyApp.FPCore.Pages.Page.ResourceFile(System.String, System.String) 05e5efec 04674d69 FP_Login.formslogin.Page_Load(System.Object, System.EventArgs) 05e5f028 031bb2b5 [MulticastFrame: 05e5f028] System.EventHandler.Invoke(System.Object, System.EventArgs) [...]
!showstringw from SieExtPub on the first parameter passed to mscorwks!ExplicitBind we can get the name of the file we’re trying to load:
0:095> kb ChildEBP RetAddr Args to Child 05e5e860 7c822124 77e6baa8 00001348 00000000 ntdll!KiFastSystemCallRet 05e5e864 77e6baa8 00001348 00000000 00000000 ntdll!NtWaitForSingleObject+0xc 05e5e8d4 79e718fd 00001348 ffffffff 00000000 kernel32!WaitForSingleObjectEx+0xac 05e5e918 79e718c6 00001348 ffffffff 00000000 mscorwks!PEImage::LoadImage+0x199 05e5e968 79e7187c ffffffff 00000000 00000000 mscorwks!CLREvent::WaitEx+0x117 05e5e978 79f90f81 ffffffff 00000000 00000000 mscorwks!CLREvent::Wait+0x17 05e5e988 79f917ba 7a390970 03c55358 ffffffff mscorwks!CExecutionEngine::WaitForEvent+0x37 05e5e99c 7a15cfce 03c55358 ffffffff 00000000 mscorwks!ClrWaitEvent+0x17 05e5e9ac 7a153339 05e5ed7c 00000000 00000000 mscorwks!CSimpleFusionBindSink::Wait+0xe 05e5e9c8 7a1545d2 03cecd70 0016e3d0 00000200 mscorwks!BindHelper+0x7d 05e5ec64 7a2d0c7c 05e5ed7c 0016e3d0 00000000 mscorwks!ExplicitBind+0x1a6 05e5ef80 793f583f 00000000 16577088 1657703c mscorwks!AssemblyNative::LoadFile+0x1de [...] 0:095> !showstringw 05e5ed7c D:\MyApp\App\Members\bin\DescMembers.dll
Looking around in the stack it’s interesting to note that also thread 81 was loading the same file from from ASP.NET temporary folder:
0:081> kb ChildEBP RetAddr Args to Child 0573cf58 7c822124 77e6baa8 000004f4 00000000 ntdll!KiFastSystemCallRet 0573cf5c 77e6baa8 000004f4 00000000 00000000 ntdll!NtWaitForSingleObject+0xc 0573cfcc 79e718fd 000004f4 ffffffff 00000000 kernel32!WaitForSingleObjectEx+0xac 0573d010 79e718c6 000004f4 ffffffff 00000000 mscorwks!PEImage::LoadImage+0x199 0573d060 79e7187c ffffffff 00000000 00000000 mscorwks!CLREvent::WaitEx+0x117 0573d070 7a0851cb ffffffff 00000000 00000000 mscorwks!CLREvent::Wait+0x17 0573d080 79f40e96 00000000 5ed28252 79eaebea mscorwks!SVR::GCHeap::WaitUntilGCComplete+0x32 0573d0bc 79e797c6 5ed2822a 79eaebea 053ee59a mscorwks!Thread::RareDisablePreemptiveGC+0x1a1 0573d0e8 79ea91b2 7a3879e0 5ed283e6 79eaebea mscorwks!CrstBase::AcquirePreempLock+0x2e 0573d124 79eb4132 0573d154 00000000 0573d1b4 mscorwks!PEImage::OpenImage+0x92 0573d1c0 79eb3d78 053ee598 0573d1f8 0573d1f4 mscorwks!RuntimeOpenImageInternal+0xcd 0573d208 79eb3cee 053ee598 79eaebb0 00000000 mscorwks!GetAssemblyMDInternalImportEx+0x9d 0573d21c 79ecf4b9 053ee598 00000000 0573d238 mscorwks!CreateMetaDataImport+0x16 0573d23c 79ecf462 053ee598 5ed280a6 00000000 mscorwks!CAssemblyManifestImport::Init+0x35 0573d264 79f2e637 053ee598 0573d27c 000f2a20 mscorwks!CreateAssemblyManifestImport+0x53 0573d280 7a161a1d 053ee598 03d09c00 0573d9a8 mscorwks!CreateAssemblyFromManifestFile+0x48 0573d930 7a161e12 0573dcc4 03d09c00 0573d9a8 mscorwks!CAsmDownloadMgr::CreateAssembly+0x8e2 0573d96c 7a162550 0573dcc4 03d09c00 0573d9a8 mscorwks!CAsmDownloadMgr::DoSetupPushToCache+0x50 0573dbf4 79f90e65 000f2a20 03d09c00 0573dcc4 mscorwks!CAsmDownloadMgr::DoSetup+0x26a 0573dc40 79f8f2cc 0573dcc4 00000000 5ed28eb6 mscorwks!CAssemblyDownload::DoSetup+0x7b 0:081> !showstringw 053ee598 C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\_mem_bin\bf10f877\41cee659\assembly\dl3\9cf10f5d\0090cb26_8229c601\DescMembers.DLL 0:081> !showstringw 0573dcc4 D:\MyApp\App\Members\bin\DescMembers.dll
Another set of dumps helped to further clarify the situation: thread 94 holds the lock and is trying to load a previously compiled assembly from disk. It is trying to load this in relation to page D:\MyApp\App\Secure\Root\default.aspx and this this is leading to the attempt to load the assembly for global.asax: C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\App_global.asax.ylfot7zw.dll. As part of this assembly load, thread 94 is blocked waiting to acquire a critical section owned by thread 123:
0:094> !syncblk Index SyncBlock MonitorHeld Recursion Owning Thread Info SyncBlock Owner 140 03f05a84 13 1 03ebd208 714 94 1225a688 System.Web.Compilation.BuildManager ----------------------------- Total 219 CCW 11 RCW 0 ComClassFactory 0 Free 0 0:094> !dso OS Thread Id: 0x714 (94) ESP/REG Object Name 04f7e9e0 790d6654 System.String 04f7eb70 1c571e2c System.Byte[] 04f7ecb8 1c571df0 System.Reflection.AssemblyName 04f7eccc 1c571df0 System.Reflection.AssemblyName 04f7edac 1c571df0 System.Reflection.AssemblyName 04f7edc4 1c571df0 System.Reflection.AssemblyName 04f7edf8 1c571ad0 System.String C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\App_global.asax.ylfot7zw.dll.delete 04f7ee00 1c47a640 System.Object[] (System.Reflection.AssemblyName[]) 04f7ee04 1c5715e8 System.String C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\App_global.asax.ylfot7zw.dll 04f7ee24 1c4775e0 System.Reflection.Assembly 04f7ee28 1c468e58 System.String App_Web_gf9hosur 04f7ee2c 1c428b70 System.Web.Compilation.PreservationFileReader 04f7ee30 121a2004 System.Globalization.CultureInfo 04f7ee34 1c468dbc System.String 110000 04f7ee44 1c1f7e58 System.Web.VirtualPath 04f7ee48 1c428b70 System.Web.Compilation.PreservationFileReader 04f7ee4c 1c4693bc System.Web.Compilation.BuildResultCompiledTemplateType 04f7ee58 1c1f7e58 System.Web.VirtualPath 04f7ee5c 1c428b70 System.Web.Compilation.PreservationFileReader 04f7ee60 1c4693bc System.Web.Compilation.BuildResultCompiledTemplateType 04f7ee7c 1c468cbc System.String fffffff8697ef330 04f7ee84 1c428a68 System.String C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\default.aspx.8cd6fbf4.compiled 04f7ee88 1c428a68 System.String C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\default.aspx.8cd6fbf4.compiled 04f7ee8c 1c1f7e58 System.Web.VirtualPath 04f7ee9c 1c428a68 System.String C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\default.aspx.8cd6fbf4.compiled 04f7eea4 1c428b70 System.Web.Compilation.PreservationFileReader 04f7eea8 1a200b5c System.String .compiled 04f7eeac 1c428970 System.String C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\default.aspx.8cd6fbf4 04f7eeb8 1a200b5c System.String .compiled 04f7eebc 1c428a68 System.String C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\default.aspx.8cd6fbf4.compiled 04f7eec0 1c428b70 System.Web.Compilation.PreservationFileReader 04f7eec4 1225b41c System.Web.Compilation.StandardDiskBuildResultCache 04f7eec8 1c1f7e58 System.Web.VirtualPath 04f7eed8 1c428a68 System.String C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\default.aspx.8cd6fbf4.compiled 04f7eee8 1c1f7e58 System.Web.VirtualPath 04f7eef8 1c1f7e58 System.Web.VirtualPath 04f7ef00 1225a688 System.Web.Compilation.BuildManager 04f7ef08 1c427124 System.String default.aspx.8cd6fbf4 04f7ef0c 1225a7c4 System.Web.Util.SimpleRecyclingCache 04f7ef20 1c427100 System.String 8cd6fbf4 04f7ef2c 1c1f7e58 System.Web.VirtualPath 04f7ef30 1225a688 System.Web.Compilation.BuildManager 04f7ef44 1c1f7e58 System.Web.VirtualPath 04f7ef50 1225a688 System.Web.Compilation.BuildManager 04f7ef54 1c1f7e58 System.Web.VirtualPath 04f7ef70 1c418fb8 System.Web.ApplicationImpersonationContext 04f7ef74 1c418fb8 System.Web.ApplicationImpersonationContext 04f7ef78 1c418fb8 System.Web.ApplicationImpersonationContext 04f7ef88 1c1f7e58 System.Web.VirtualPath 04f7ef9c 1c418fb8 System.Web.ApplicationImpersonationContext 04f7efa8 1c418fb8 System.Web.ApplicationImpersonationContext 04f7efac 1c1f7e58 System.Web.VirtualPath 04f7efc4 1c418fb8 System.Web.ApplicationImpersonationContext 04f7efcc 1c418f88 System.Object[] (System.Type[]) 04f7efd0 1c418fac System.Web.UI.PageHandlerFactory 04f7efe4 1c1f7e58 System.Web.VirtualPath 04f7efe8 1c1f7c2c System.Web.HttpContext 04f7f000 1c1f7c2c System.Web.HttpContext 04f7f004 1444b800 System.RuntimeType 04f7f024 1239e718 System.String System.Web.UI.PageHandlerFactory 04f7f02c 1c393abc System.Collections.Hashtable 04f7f034 1c1f73a4 System.String D:\MyApp\App\Secure\Root\default.aspx 04f7f038 1c1f7c2c System.Web.HttpContext 04f7f03c 1c1f7e58 System.Web.VirtualPath 04f7f050 1c1f7c2c System.Web.HttpContext 04f7f054 1c418fac System.Web.UI.PageHandlerFactory 04f7f058 1c1f7c2c System.Web.HttpContext 04f7f060 1c1f73a4 System.String D:\MyApp\App\Secure\Root\default.aspx 04f7f064 1c1f7e58 System.Web.VirtualPath 04f7f068 1c1f7338 System.String POST 04f7f06c 1c418fac System.Web.UI.PageHandlerFactory 04f7f070 1c4187b0 System.Web.ApplicationImpersonationContext 04f7f074 1c1f87b8 ASP.global_asax 04f7f07c 1c1f73a4 System.String D:\MyApp\App\Secure\Root\default.aspx 04f7f080 1c1f7e58 System.Web.VirtualPath 04f7f084 1c1f7338 System.String POST 04f7f088 1c4187b0 System.Web.ApplicationImpersonationContext 04f7f090 1c1f7c2c System.Web.HttpContext 04f7f0a4 1c1f87b8 ASP.global_asax 04f7f0a8 1c1f87b8 ASP.global_asax 04f7f0ac 1c1f7cd4 System.Web.HttpRequest 04f7f0b0 1c1f7c2c System.Web.HttpContext 04f7f0c0 1c1f73a4 System.String D:\MyApp\App\Secure\Root\default.aspx 04f7f0c4 1c1f7e58 System.Web.VirtualPath 04f7f0c8 1c1f7338 System.String POST 04f7f0cc 1c1f7c2c System.Web.HttpContext 04f7f0d0 1c1f87b8 ASP.global_asax 04f7f0e8 1c411f58 System.Web.HttpApplication+MapHandlerExecutionStep 04f7f0ec 1c1f87b8 ASP.global_asax 04f7f0f0 1c1f7c2c System.Web.HttpContext 04f7f104 1c1f7c2c System.Web.HttpContext 04f7f108 1c1f87b8 ASP.global_asax 04f7f11c 1c1f87b8 ASP.global_asax 04f7f120 1c412258 System.Web.HttpApplication+ThreadContext 04f7f124 1c1f7c2c System.Web.HttpContext 04f7f128 1c1f87b8 ASP.global_asax 04f7f12c 1c412238 System.Web.AspNetSynchronizationContext 04f7f134 1c411cdc System.Web.HttpApplication+ApplicationStepManager 04f7f158 1c1f7c2c System.Web.HttpContext 04f7f15c 1c412218 System.Web.HttpAsyncResult 04f7f160 1c1f87b8 ASP.global_asax 04f7f164 121c489c System.AsyncCallback 04f7f170 1c1f4ff4 System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6 04f7f174 1c1f87b8 ASP.global_asax 04f7f178 1c1f7e6c System.Web.HttpWriter 04f7f180 1c1f7c2c System.Web.HttpContext 04f7f184 121c489c System.AsyncCallback 04f7f188 1c1f7c2c System.Web.HttpContext 04f7f18c 1c1f4ff4 System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6 04f7f190 121c4138 System.Web.HttpRuntime 04f7f1a8 1c1f72dc System.String D:\MyApp\App\Secure\ 04f7f1ac 1c1f4ff4 System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6 04f7f1bc 121a1038 System.String D:\MyApp\App\Secure\ 04f7f1c0 1c1f4ff4 System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6 04f7f1f8 122f2880 System.Web.Hosting.ISAPIRuntime 0:094> !do 1c1f4ff4 Name: System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6 MethodTable: 663a6f84 EEClass: 663a6f04 Size: 256(0x100) bytes (C:\WINDOWS\assembly\GAC_32\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll) Fields: MT Field Offset Type VT Attr Value Name 79105fac 400133a 4 System.DateTime 1 instance 1c1f4ff8 _startTime 7910ead0 400133b c System.Guid 1 instance 1c1f5000 _traceId 7912ab5c 400133c 194 System.String[][] 0 shared static s_HTTPStatusDescriptions >> Domain:Value 001171e8:NotInit 00159f68:1a227064 03ea3b78:122f6384 << 79124304 400133d 198 System.Object[] 0 shared static s_serverVarFromRequestHeaderNames >> Domain:Value 001171e8:NotInit 00159f68:1a2271ac 03ea3b78:122f64cc << 79124304 400133e 19c System.Object[] 0 shared static s_requestHeaderNames >> Domain:Value 001171e8:NotInit 00159f68:1a22725c 03ea3b78:122f657c << 79124304 400133f 1a0 System.Object[] 0 shared static s_responseHeaderNames >> Domain:Value 001171e8:NotInit 00159f68:1a22730c 03ea3b78:122f662c << 790feb48 4001340 1a4 ...ections.Hashtable 0 shared static s_requestHeadersLoookupTable >> Domain:Value 001171e8:NotInit 00159f68:1a227394 03ea3b78:122f66b4 << 790feb48 4001341 1a8 ...ections.Hashtable 0 shared static s_responseHeadersLoookupTable >> Domain:Value 001171e8:NotInit 00159f68:1a22745c 03ea3b78:122f677c << 790fe238 4001b3a 70 System.IntPtr 1 instance 3066598 _ecb 790fe238 4001b3b 74 System.IntPtr 1 instance 2d0 _token 7910ead0 4001b3c a8 System.Guid 1 instance 1c1f509c _traceId 790fa4b0 4001b3d 1c System.String 0 instance 1c1f7338 _method 790fa4b0 4001b3e 20 System.String 0 instance 1c1f7354 _path 790fa4b0 4001b3f 24 System.String 0 instance 1c1f7354 _filePath 790fa4b0 4001b40 28 System.String 0 instance 790d6654 _pathInfo 790fa4b0 4001b41 2c System.String 0 instance 1c1f73a4 _pathTranslated 790fa4b0 4001b42 30 System.String 0 instance 141a4b28 _appPath 790fa4b0 4001b43 34 System.String 0 instance 1c1f72dc _appPathTranslated [...] 0:094> !do 1c1f73a4 Name: System.String MethodTable: 790fa4b0 EEClass: 790fa410 Size: 150(0x96) bytes (C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll) String: D:\MyApp\App\Secure\Root\default.aspx Fields: MT Field Offset Type VT Attr Value Name 790fedf4 4000096 4 System.Int32 1 instance 67 m_arrayLength 790fedf4 4000097 8 System.Int32 1 instance 66 m_stringLength 790fbfcc 4000098 c System.Char 1 instance 44 m_firstChar 790fa4b0 4000099 10 System.String 0 shared static Empty >> Domain:Value 001171e8:790d6654 00159f68:790d6654 03ea3b78:790d6654 << 7912474c 400009a 14 System.Char[] 0 shared static WhitespaceChars >> Domain:Value 001171e8:141a0624 00159f68:141a77f8 03ea3b78:121a2374 << 0:094> !do 1c1f87b8 Name: ASP.global_asax MethodTable: 04c35adc EEClass: 04ff1dd8 Size: 140(0x8c) bytes (C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\App_global.asax.ylfot7zw.dll) Fields: MT Field Offset Type VT Attr Value Name 663a2450 4000dc8 4 ...pApplicationState 0 instance 122e14dc _state 663a2750 4000dc9 8 ...m.Web.HttpContext 0 instance 00000000 _initContext 663a359c 4000dca c ...b.HttpAsyncResult 0 instance 1c412218 _ar 663a3394 4000dcb 10 ...pModuleCollection 0 instance 1c3a648c _moduleCollection 79118214 4000dcc 14 ...ncipal.IPrincipal 0 instance 1c413d68 _savedPrincipal 79105020 4000dcd 78 System.Boolean 1 instance 1 _restorePrincipal 7a74da98 4000de8 18 ....EventHandlerList 0 instance 1c3f7408 _events 663a2aec 4000de9 1c ...ventHandlersTable 0 instance 1c401980 _asyncEvents [...] 0:094> !critlist CritSec at 7a38ee88. Owned by thread 123. Waiting Threads: 26 66 94
0:094> !syncblk Index SyncBlock MonitorHeld Recursion Owning Thread Info SyncBlock Owner 140 03f05a84 13 1 03ebd208 714 94 1225a688 System.Web.Compilation.BuildManager ----------------------------- Total 219 CCW 11 RCW 0 ComClassFactory 0 Free 0 0:094> !dso OS Thread Id: 0x714 (94) ESP/REG Object Name 04f7e9e0 790d6654 System.String 04f7eb70 1c571e2c System.Byte[] 04f7ecb8 1c571df0 System.Reflection.AssemblyName 04f7eccc 1c571df0 System.Reflection.AssemblyName 04f7edac 1c571df0 System.Reflection.AssemblyName 04f7edc4 1c571df0 System.Reflection.AssemblyName 04f7edf8 1c571ad0 System.String C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\App_global.asax.ylfot7zw.dll.delete 04f7ee00 1c47a640 System.Object[] (System.Reflection.AssemblyName[]) 04f7ee04 1c5715e8 System.String C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\App_global.asax.ylfot7zw.dll 04f7ee24 1c4775e0 System.Reflection.Assembly 04f7ee28 1c468e58 System.String App_Web_gf9hosur 04f7ee2c 1c428b70 System.Web.Compilation.PreservationFileReader 04f7ee30 121a2004 System.Globalization.CultureInfo 04f7ee34 1c468dbc System.String 110000 04f7ee44 1c1f7e58 System.Web.VirtualPath 04f7ee48 1c428b70 System.Web.Compilation.PreservationFileReader 04f7ee4c 1c4693bc System.Web.Compilation.BuildResultCompiledTemplateType 04f7ee58 1c1f7e58 System.Web.VirtualPath 04f7ee5c 1c428b70 System.Web.Compilation.PreservationFileReader 04f7ee60 1c4693bc System.Web.Compilation.BuildResultCompiledTemplateType 04f7ee7c 1c468cbc System.String fffffff8697ef330 04f7ee84 1c428a68 System.String C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\default.aspx.8cd6fbf4.compiled 04f7ee88 1c428a68 System.String C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\default.aspx.8cd6fbf4.compiled 04f7ee8c 1c1f7e58 System.Web.VirtualPath 04f7ee9c 1c428a68 System.String C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\default.aspx.8cd6fbf4.compiled 04f7eea4 1c428b70 System.Web.Compilation.PreservationFileReader 04f7eea8 1a200b5c System.String .compiled 04f7eeac 1c428970 System.String C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\default.aspx.8cd6fbf4 04f7eeb8 1a200b5c System.String .compiled 04f7eebc 1c428a68 System.String C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\default.aspx.8cd6fbf4.compiled 04f7eec0 1c428b70 System.Web.Compilation.PreservationFileReader 04f7eec4 1225b41c System.Web.Compilation.StandardDiskBuildResultCache 04f7eec8 1c1f7e58 System.Web.VirtualPath 04f7eed8 1c428a68 System.String C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\default.aspx.8cd6fbf4.compiled 04f7eee8 1c1f7e58 System.Web.VirtualPath 04f7eef8 1c1f7e58 System.Web.VirtualPath 04f7ef00 1225a688 System.Web.Compilation.BuildManager 04f7ef08 1c427124 System.String default.aspx.8cd6fbf4 04f7ef0c 1225a7c4 System.Web.Util.SimpleRecyclingCache 04f7ef20 1c427100 System.String 8cd6fbf4 04f7ef2c 1c1f7e58 System.Web.VirtualPath 04f7ef30 1225a688 System.Web.Compilation.BuildManager 04f7ef44 1c1f7e58 System.Web.VirtualPath 04f7ef50 1225a688 System.Web.Compilation.BuildManager 04f7ef54 1c1f7e58 System.Web.VirtualPath 04f7ef70 1c418fb8 System.Web.ApplicationImpersonationContext 04f7ef74 1c418fb8 System.Web.ApplicationImpersonationContext 04f7ef78 1c418fb8 System.Web.ApplicationImpersonationContext 04f7ef88 1c1f7e58 System.Web.VirtualPath 04f7ef9c 1c418fb8 System.Web.ApplicationImpersonationContext 04f7efa8 1c418fb8 System.Web.ApplicationImpersonationContext 04f7efac 1c1f7e58 System.Web.VirtualPath 04f7efc4 1c418fb8 System.Web.ApplicationImpersonationContext 04f7efcc 1c418f88 System.Object[] (System.Type[]) 04f7efd0 1c418fac System.Web.UI.PageHandlerFactory 04f7efe4 1c1f7e58 System.Web.VirtualPath 04f7efe8 1c1f7c2c System.Web.HttpContext 04f7f000 1c1f7c2c System.Web.HttpContext 04f7f004 1444b800 System.RuntimeType 04f7f024 1239e718 System.String System.Web.UI.PageHandlerFactory 04f7f02c 1c393abc System.Collections.Hashtable 04f7f034 1c1f73a4 System.String D:\MyApp\App\Secure\Root\default.aspx 04f7f038 1c1f7c2c System.Web.HttpContext 04f7f03c 1c1f7e58 System.Web.VirtualPath 04f7f050 1c1f7c2c System.Web.HttpContext 04f7f054 1c418fac System.Web.UI.PageHandlerFactory 04f7f058 1c1f7c2c System.Web.HttpContext 04f7f060 1c1f73a4 System.String D:\MyApp\App\Secure\Root\default.aspx 04f7f064 1c1f7e58 System.Web.VirtualPath 04f7f068 1c1f7338 System.String POST 04f7f06c 1c418fac System.Web.UI.PageHandlerFactory 04f7f070 1c4187b0 System.Web.ApplicationImpersonationContext 04f7f074 1c1f87b8 ASP.global_asax 04f7f07c 1c1f73a4 System.String D:\MyApp\App\Secure\Root\default.aspx 04f7f080 1c1f7e58 System.Web.VirtualPath 04f7f084 1c1f7338 System.String POST 04f7f088 1c4187b0 System.Web.ApplicationImpersonationContext 04f7f090 1c1f7c2c System.Web.HttpContext 04f7f0a4 1c1f87b8 ASP.global_asax 04f7f0a8 1c1f87b8 ASP.global_asax 04f7f0ac 1c1f7cd4 System.Web.HttpRequest 04f7f0b0 1c1f7c2c System.Web.HttpContext 04f7f0c0 1c1f73a4 System.String D:\MyApp\App\Secure\Root\default.aspx 04f7f0c4 1c1f7e58 System.Web.VirtualPath 04f7f0c8 1c1f7338 System.String POST 04f7f0cc 1c1f7c2c System.Web.HttpContext 04f7f0d0 1c1f87b8 ASP.global_asax 04f7f0e8 1c411f58 System.Web.HttpApplication+MapHandlerExecutionStep 04f7f0ec 1c1f87b8 ASP.global_asax 04f7f0f0 1c1f7c2c System.Web.HttpContext 04f7f104 1c1f7c2c System.Web.HttpContext 04f7f108 1c1f87b8 ASP.global_asax 04f7f11c 1c1f87b8 ASP.global_asax 04f7f120 1c412258 System.Web.HttpApplication+ThreadContext 04f7f124 1c1f7c2c System.Web.HttpContext 04f7f128 1c1f87b8 ASP.global_asax 04f7f12c 1c412238 System.Web.AspNetSynchronizationContext 04f7f134 1c411cdc System.Web.HttpApplication+ApplicationStepManager 04f7f158 1c1f7c2c System.Web.HttpContext 04f7f15c 1c412218 System.Web.HttpAsyncResult 04f7f160 1c1f87b8 ASP.global_asax 04f7f164 121c489c System.AsyncCallback 04f7f170 1c1f4ff4 System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6 04f7f174 1c1f87b8 ASP.global_asax 04f7f178 1c1f7e6c System.Web.HttpWriter 04f7f180 1c1f7c2c System.Web.HttpContext 04f7f184 121c489c System.AsyncCallback 04f7f188 1c1f7c2c System.Web.HttpContext 04f7f18c 1c1f4ff4 System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6 04f7f190 121c4138 System.Web.HttpRuntime 04f7f1a8 1c1f72dc System.String D:\MyApp\App\Secure\ 04f7f1ac 1c1f4ff4 System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6 04f7f1bc 121a1038 System.String D:\MyApp\App\Secure\ 04f7f1c0 1c1f4ff4 System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6 04f7f1f8 122f2880 System.Web.Hosting.ISAPIRuntime 0:094> !do 1c1f4ff4 Name: System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6 MethodTable: 663a6f84 EEClass: 663a6f04 Size: 256(0x100) bytes (C:\WINDOWS\assembly\GAC_32\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll) Fields: MT Field Offset Type VT Attr Value Name 79105fac 400133a 4 System.DateTime 1 instance 1c1f4ff8 _startTime 7910ead0 400133b c System.Guid 1 instance 1c1f5000 _traceId 7912ab5c 400133c 194 System.String[][] 0 shared static s_HTTPStatusDescriptions >> Domain:Value 001171e8:NotInit 00159f68:1a227064 03ea3b78:122f6384 << 79124304 400133d 198 System.Object[] 0 shared static s_serverVarFromRequestHeaderNames >> Domain:Value 001171e8:NotInit 00159f68:1a2271ac 03ea3b78:122f64cc << 79124304 400133e 19c System.Object[] 0 shared static s_requestHeaderNames >> Domain:Value 001171e8:NotInit 00159f68:1a22725c 03ea3b78:122f657c << 79124304 400133f 1a0 System.Object[] 0 shared static s_responseHeaderNames >> Domain:Value 001171e8:NotInit 00159f68:1a22730c 03ea3b78:122f662c << 790feb48 4001340 1a4 ...ections.Hashtable 0 shared static s_requestHeadersLoookupTable >> Domain:Value 001171e8:NotInit 00159f68:1a227394 03ea3b78:122f66b4 << 790feb48 4001341 1a8 ...ections.Hashtable 0 shared static s_responseHeadersLoookupTable >> Domain:Value 001171e8:NotInit 00159f68:1a22745c 03ea3b78:122f677c << 790fe238 4001b3a 70 System.IntPtr 1 instance 3066598 _ecb 790fe238 4001b3b 74 System.IntPtr 1 instance 2d0 _token 7910ead0 4001b3c a8 System.Guid 1 instance 1c1f509c _traceId 790fa4b0 4001b3d 1c System.String 0 instance 1c1f7338 _method 790fa4b0 4001b3e 20 System.String 0 instance 1c1f7354 _path 790fa4b0 4001b3f 24 System.String 0 instance 1c1f7354 _filePath 790fa4b0 4001b40 28 System.String 0 instance 790d6654 _pathInfo 790fa4b0 4001b41 2c System.String 0 instance 1c1f73a4 _pathTranslated 790fa4b0 4001b42 30 System.String 0 instance 141a4b28 _appPath 790fa4b0 4001b43 34 System.String 0 instance 1c1f72dc _appPathTranslated [...] 0:094> !do 1c1f73a4 Name: System.String MethodTable: 790fa4b0 EEClass: 790fa410 Size: 150(0x96) bytes (C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll) String: D:\MyApp\App\Secure\Root\default.aspx Fields: MT Field Offset Type VT Attr Value Name 790fedf4 4000096 4 System.Int32 1 instance 67 m_arrayLength 790fedf4 4000097 8 System.Int32 1 instance 66 m_stringLength 790fbfcc 4000098 c System.Char 1 instance 44 m_firstChar 790fa4b0 4000099 10 System.String 0 shared static Empty >> Domain:Value 001171e8:790d6654 00159f68:790d6654 03ea3b78:790d6654 << 7912474c 400009a 14 System.Char[] 0 shared static WhitespaceChars >> Domain:Value 001171e8:141a0624 00159f68:141a77f8 03ea3b78:121a2374 << 0:094> !do 1c1f87b8 Name: ASP.global_asax MethodTable: 04c35adc EEClass: 04ff1dd8 Size: 140(0x8c) bytes (C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\App_global.asax.ylfot7zw.dll) Fields: MT Field Offset Type VT Attr Value Name 663a2450 4000dc8 4 ...pApplicationState 0 instance 122e14dc _state 663a2750 4000dc9 8 ...m.Web.HttpContext 0 instance 00000000 _initContext 663a359c 4000dca c ...b.HttpAsyncResult 0 instance 1c412218 _ar 663a3394 4000dcb 10 ...pModuleCollection 0 instance 1c3a648c _moduleCollection 79118214 4000dcc 14 ...ncipal.IPrincipal 0 instance 1c413d68 _savedPrincipal 79105020 4000dcd 78 System.Boolean 1 instance 1 _restorePrincipal 7a74da98 4000de8 18 ....EventHandlerList 0 instance 1c3f7408 _events 663a2aec 4000de9 1c ...ventHandlersTable 0 instance 1c401980 _asyncEvents [...]
0:094> !critlist CritSec at 7a38ee88. Owned by thread 123. Waiting Threads: 26 66 94
!dso output already shows objects and strings in the same order as they are stored on the stack and executed, so it’s quite easy to spot interesting file paths, URLs etc… and often that is enough to suggest which files are involved, which page or web service you’re accessing and so on; of course you can always use !do <address> to dig into the stack to be 100% sure you’re on the right path.
Using the same commands above we found that thread 123 is trying to explicitly load "D:\MyApp\App\Secure\bin\secure.dll" whilst executing "ASP.logout_aspx"; after transitioning from managed to unmanaged code, this thread then gets blocked waiting for garbage collection to complete (because it needs to disable pre-emptive mode in order to acquire a particular lock).
In summary, the hang was due to a complicated deadlock situation involving “D:\MyApp\App\Secure\Root\default.aspx” being compiled by ASP.NET 2.0 compilation engine. This leads to an attempt to load “C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\App_global.asax.ylfot7zw.dll” from disk (the default.aspx page has a dependency on the global.asax) which in turn leads to an attempt to acquire a critical section owned by thread 123. Thread 123 is processing the request for “logout.aspx” which involves the execution of “MyApp.Core.Library.Managers.LanguageManager.ResourceFile” which leads to an attempt to load “D:\MyApp\App\Secure\bin\secure.dll” from disk. However this load operation gets blocked waiting for garbage collection to complete as it must acquire a particular lock when GC is not running. The GC was triggered (incidentally) by thread 56, but the GC has not actually commenced yet. The primary GC thread (29) has asked all thread executing managed code to get to a 'safe' point where GC can begin, but that has not happened (and will never happen) because some of those threads are blocked doing assembly load operations and are waiting on critical section.
Finally, the resolution: the customer confirmed the involved assemblies are “resource only” files which they access from everywhere in their quite complex application (basically those were localization assemblies they were explicitly loading when needed to localize their interface, error messages etc…); we strong-named those “sensitive” assemblies and installed them in the GAC: easy solution, when you know which is the problem…
By the way, there are a couple hotfixes available for this situation (see http://support.microsoft.com/kb/936757/en-us and 946644, KB not yet available but you can ask for the fix to CSS), but did not help in this case.
Carlo
My post on autoConfig triggered and interesting discussion with my colleague Cenk about which is the real effect of this setting; one of his customers asked what to do if they needed to adjust only one or two values and not all the ones discussed in Contention, poor performance, and deadlocks when you make Web service requests from ASP.NET applications, they wanted to be sure they could change only what they needed to fine tune and forget about the rest. The short answer is: you can change only what you need to and leave he rest as is, since we have default values hardcoded to improve performance so you’re safe (provided the values you’re changing make sense in the context, of course).
If we have a look at System.Web.Configuration.ProcessModelSection with Windbg, we can see all the properties we have in <processModel> and _defaultValue corresponds to the defaults we have in machine.config.comments:
0:000> !do 06904868 Name: System.Web.Configuration.ProcessModelSection MethodTable: 6638dcb0 EEClass: 663a0604 Size: 64(0x40) bytes (C:\Windows\assembly\GAC_32\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll) Fields: MT Field Offset Type VT Attr Value Name 7910be50 400014d 30 System.Boolean 1 instance 0 _bDataToWrite 7910be50 400014e 31 System.Boolean 1 instance 0 _bModified 7910be50 400014f 32 System.Boolean 1 instance 1 _bReadOnly 7910be50 4000150 33 System.Boolean 1 instance 0 _bElementPresent 7910be50 4000151 34 System.Boolean 1 instance 1 _bInited 648eff64 4000152 4 ...ionLockCollection 0 instance 00000000 _lockedAttributesList 648eff64 4000153 8 ...ionLockCollection 0 instance 00000000 _lockedAllExceptAttributesList 648eff64 4000154 c ...ionLockCollection 0 instance 00000000 _lockedElementsList 648eff64 4000155 10 ...ionLockCollection 0 instance 00000000 _lockedAllExceptElementsList 648ea704 4000156 14 ...nfigurationValues 0 instance 069048a8 _values 790fd8c4 4000157 18 System.String 0 instance 00000000 _elementTagName 648f032c 4000158 1c ...lementInformation 0 instance 00000000 _evaluationElement 648eee30 4000159 20 ...onElementProperty 0 instance 068fce20 _elementProperty 648f53d0 400015a 2c System.Int32 1 instance 0 _fItemLocked 648f0a48 400015b 24 ...ontextInformation 0 instance 00000000 _evalContext 648eda8c 400015c 28 ...nfigurationRecord 0 instance 028bea78 _configRecord 7912d8f8 4000148 8 System.Object[] 0 shared static s_lockAttributeNames >> Domain:Value 023d04b0:NotInit 024169d0:068fcd1c << 79101fe4 4000149 c ...ections.Hashtable 0 shared static s_propertyBags >> Domain:Value 023d04b0:NotInit 024169d0:068fcd40 << 00000000 400014a 10 0 shared static s_perTypeValidators >> Domain:Value 023d04b0:NotInit 024169d0:00000000 << 790fd0f0 400014b 14 System.Object 0 shared static s_nullPropertyValue >> Domain:Value 023d04b0:NotInit 024169d0:068fce08 << 648eee30 400014c 18 ...onElementProperty 0 shared static s_ElementProperty >> Domain:Value 023d04b0:NotInit 024169d0:068fce20 << 648eee84 400015d 38 ...ectionInformation 0 instance 069049b8 _section 648eee30 4001928 74c ...onElementProperty 0 shared static s_elemProperty >> Domain:Value 023d04b0:NotInit 024169d0:06903da8 << 7911228c 4001929 748 System.TimeSpan 1 shared static DefaultClientConnectedCheck >> Domain:Value 023d04b0:NotInit 024169d0:06903850 << 648ef118 400192a 750 ...ropertyCollection 0 shared static _properties >> Domain:Value 023d04b0:NotInit 024169d0:06904714 << 648ef05c 400192b 754 ...igurationProperty 0 shared static _propEnable >> Domain:Value 023d04b0:NotInit 024169d0:06903db4 << 648ef05c 400192c 758 ...igurationProperty 0 shared static _propTimeout >> Domain:Value 023d04b0:NotInit 024169d0:06903df4 << 648ef05c 400192d 75c ...igurationProperty 0 shared static _propIdleTimeout >> Domain:Value 023d04b0:NotInit 024169d0:06903e38 << 648ef05c 400192e 760 ...igurationProperty 0 shared static _propShutdownTimeout >> Domain:Value 023d04b0:NotInit 024169d0:06903e7c << 648ef05c 400192f 764 ...igurationProperty 0 shared static _propRequestLimit >> Domain:Value 023d04b0:NotInit 024169d0:06903ed8 << 648ef05c 4001930 768 ...igurationProperty 0 shared static _propRequestQueueLimit >> Domain:Value 023d04b0:NotInit 024169d0:06903f74 << 648ef05c 4001931 76c ...igurationProperty 0 shared static _propRestartQueueLimit >> Domain:Value 023d04b0:NotInit 024169d0:06903ff8 << 648ef05c 4001932 770 ...igurationProperty 0 shared static _propMemoryLimit >> Domain:Value 023d04b0:NotInit 024169d0:06904064 << 648ef05c 4001933 774 ...igurationProperty 0 shared static _propWebGarden >> Domain:Value 023d04b0:NotInit 024169d0:069040a4 << 648ef05c 4001934 778 ...igurationProperty 0 shared static _propCpuMask >> Domain:Value 023d04b0:NotInit 024169d0:069040e4 << 648ef05c 4001935 77c ...igurationProperty 0 shared static _propUserName >> Domain:Value 023d04b0:NotInit 024169d0:06904118 << 648ef05c 4001936 780 ...igurationProperty 0 shared static _propPassword >> Domain:Value 023d04b0:NotInit 024169d0:0690414c << 648ef05c 4001937 784 ...igurationProperty 0 shared static _propLogLevel >> Domain:Value 023d04b0:NotInit 024169d0:06904180 << 648ef05c 4001938 788 ...igurationProperty 0 shared static _propClientConnectedCheck >> Domain:Value 023d04b0:NotInit 024169d0:069041d4 << 648ef05c 4001939 78c ...igurationProperty 0 shared static _propComAuthenticationLevel >> Domain:Value 023d04b0:NotInit 024169d0:06904218 << 648ef05c 400193a 790 ...igurationProperty 0 shared static _propComImpersonationLevel >> Domain:Value 023d04b0:NotInit 024169d0:0690426c << 648ef05c 400193b 794 ...igurationProperty 0 shared static _propResponseDeadlockInterval >> Domain:Value 023d04b0:NotInit 024169d0:069042c0 << 648ef05c 400193c 798 ...igurationProperty 0 shared static _propResponseRestartDeadlockInterval >> Domain:Value 023d04b0:NotInit 024169d0:06904304 << 648ef05c 400193d 79c ...igurationProperty 0 shared static _propAutoConfig >> Domain:Value 023d04b0:NotInit 024169d0:06904348 << 648ef05c 400193e 7a0 ...igurationProperty 0 shared static _propMaxWorkerThreads >> Domain:Value 023d04b0:NotInit 024169d0:069043ac << 648ef05c 400193f 7a4 ...igurationProperty 0 shared static _propMaxIOThreads >> Domain:Value 023d04b0:NotInit 024169d0:0690443c << 648ef05c 4001940 7a8 ...igurationProperty 0 shared static _propMinWorkerThreads >> Domain:Value 023d04b0:NotInit 024169d0:069044cc << 648ef05c 4001941 7ac ...igurationProperty 0 shared static _propMinIOThreads >> Domain:Value 023d04b0:NotInit 024169d0:0690455c << 648ef05c 4001942 7b0 ...igurationProperty 0 shared static _propServerErrorMessageFile >> Domain:Value 023d04b0:NotInit 024169d0:069045c8 << 648ef05c 4001943 7b4 ...igurationProperty 0 shared static _propPingFrequency >> Domain:Value 023d04b0:NotInit 024169d0:069045fc << 648ef05c 4001944 7b8 ...igurationProperty 0 shared static _propPingTimeout >> Domain:Value 023d04b0:NotInit 024169d0:06904640 << 648ef05c 4001945 7bc ...igurationProperty 0 shared static _propMaxAppDomains >> Domain:Value 023d04b0:NotInit 024169d0:069046a8 << 79102290 4001946 938 System.Int32 1 shared static cpuCount >> Domain:Value 023d04b0:NotInit 024169d0:2 << 0:000> !do 0690443c Name: System.Configuration.ConfigurationProperty MethodTable: 648ef05c EEClass: 648eefec Size: 52(0x34) bytes (C:\Windows\assembly\GAC_MSIL\System.Configuration\2.0.0.0__b03f5f7f11d50a3a\System.Configuration.dll) Fields: MT Field Offset Type VT Attr Value Name 790fd8c4 4000277 4 System.String 0 instance 06903c08 _name 790fd8c4 4000278 8 System.String 0 instance 06903c08 _providedName 790fd8c4 4000279 c System.String 0 instance 00000000 _description 79106894 400027a 10 System.Type 0 instance 068b7554 _type 790fd0f0 400027b 14 System.Object 0 instance 06904430 _defaultValue 7a7588dc 400027c 18 ...del.TypeConverter 0 instance 00000000 _converter 648ef1a0 400027d 2c System.Int32 1 instance 0 _options 648f022c 400027e 1c ...tionValidatorBase 0 instance 06904418 _validator 790fd8c4 400027f 20 System.String 0 instance 00000000 _addElementName 790fd8c4 4000280 24 System.String 0 instance 00000000 _removeElementName 790fd8c4 4000281 28 System.String 0 instance 00000000 _clearElementName 648f022c 4000274 68 ...tionValidatorBase 0 shared static NonEmptyStringValidator >> Domain:Value 023d04b0:NotInit 024169d0:068fcc00 << 648f022c 4000275 6c ...tionValidatorBase 0 shared static DefaultValidatorInstance >> Domain:Value 023d04b0:NotInit 024169d0:068fcc14 << 790fd8c4 4000276 70 System.String 0 shared static DefaultCollectionPropertyName >> Domain:Value 023d04b0:NotInit 024169d0:068fcbec << 0:000> !do 06904430 Name: System.Int32 MethodTable: 79102290 EEClass: 79102218 Size: 12(0xc) bytes (C:\Windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll) Fields: MT Field Offset Type VT Attr Value Name 79102290 40003e9 4 System.Int32 1 instance 20 m_value
<processModel enable = "true" [true|false] timeout = "10675199.02:48:05.4775807" [HH:MM:SS] idleTimeout = "10675199.02:48:05.4775807" [HH:MM:SS] shutdownTimeout = "00:00:05" [HH:MM:SS] requestLimit = "2147483647" [number] requestQueueLimit = "5000" [number] restartQueueLimit = "10" [number] memoryLimit = "60" [number] webGarden = "false" [true|false] cpuMask = "0xffffffff" [number] userName = "machine" [String] password = "AutoGenerate" [String] logLevel = "Errors" [None | All | Errors] clientConnectedCheck = "00:00:05" [HH:MM:SS] comAuthenticationLevel = "Connect" [None | Call | Connect | Default | Pkt | PktIntegrity | PktPrivacy] comImpersonationLevel = "Impersonate" [Default | Anonymous | Delegate | Identify | Impersonate] responseDeadlockInterval = "00:03:00" [HH:MM:SS] responseRestartDeadlockInterval = "00:03:00" [HH:MM:SS] autoConfig = "false" [true|false] maxWorkerThreads = "20" [number] maxIoThreads = "20" [number] minWorkerThreads = "1" [number] minIoThreads = "1" [number] serverErrorMessageFile = "" [String] pingFrequency = "10675199.02:48:05.4775807" [HH:MM:SS] pingTimeout = "10675199.02:48:05.4775807" [HH:MM:SS] maxAppDomains = "2000" [number] />
Internally to debug managed code we use a Windbg extension which works well with private symbols and has some additional commands the public sos.dll does not have and one of those shows the ThreadPool configuration, very useful to have a quick report of those configuration values.
After the discussion I found myself thinking how to extract those values from a dump using only public symbols and the public sos extension everyone has access to; two of them are quite easy to find out (you can get a full list of members of ThreadpoolMgr class running mscorwks!ThreadpoolMgr*):
0:000> ?poi(mscorwks!ThreadpoolMgr::NumberOfProcessors) Evaluate expression: 2 = 00000002 0:000> ?poi(mscorwks!ThreadpoolMgr::MaxLimitTotalWorkerThreads) Evaluate expression: 200 = 000000c8
From System.Web.RequestQueue we can get _minLocalFreeThreads, _minExternFreeThreads and _queueLimit (RequestQueueLimit):
Name: System.Web.RequestQueue MethodTable: 663aebf0 EEClass: 663aeb80 Size: 64(0x40) bytes (C:\Windows\assembly\GAC_32\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll) Fields: MT Field Offset Type VT Attr Value Name 79102290 4001203 14 System.Int32 1 instance 176 _minExternFreeThreads 79102290 4001204 18 System.Int32 1 instance 152 _minLocalFreeThreads 79102290 4001205 1c System.Int32 1 instance 5000 _queueLimit 7911228c 4001206 2c System.TimeSpan 1 instance 0691da2c _clientConnectedTime 7910be50 4001207 28 System.Boolean 1 instance 1 _iis6 791087cc 4001208 4 ...Collections.Queue 0 instance 0691da40 _localQueue 791087cc 4001209 8 ...Collections.Queue 0 instance 0691daf4 _externQueue 79102290 400120a 20 System.Int32 1 instance 0 _count 791186fc 400120b c ...ding.WaitCallback 0 instance 0691dba8 _workItemCallback 79102290 400120c 24 System.Int32 1 instance 0 _workItemCount 7910be50 400120d 29 System.Boolean 1 instance 0 _draining 7911228c 400120e 34 System.TimeSpan 1 instance 0691da34 _timerPeriod 791127fc 400120f 10 ...m.Threading.Timer 0 instance 0691dbe8 _timer
I was trying to find a convenient way to get the remaining three values (autoConfig, maxConnection and Number of timers) when I found this post from Tom, and now my quest is over: !DumpThreadConfig is the same command we have in the internal Windbg extension I mentioned and also the output is the same… so apparently you just have to be patient and wait for the next Debugging Tools release
Perfmon is likely the first tool you think to when it comes to monitor some internals and performance of your application and it’s relatively easy to find out information and resources on the Internet; unfortunately this seems to also be a fragile component and every now and then we receive new calls about it: event log spammed with warnings about missing or corrupted counters, values returned are inconsistent or clearly wrong etc…
Messages like the following can appear after for example installing the .NET Framework 2.0 on your machine:
The Open Procedure for service "ASP.NET" in DLL "C:\Windows\Microsoft.NET\Framework\v2.0.50727\aspnet_perf.dll" failed. Performance data for this service will not be available. Status code returned is data DWORD 0
To note that such warnings are logged even if your application is not running.
First of all you can try to disable any third party services and processes (for example using msconfig.exe or Autoruns) and see if the problem still reproduces. If it does, you can use How to manually rebuild Performance Counter Library values to try to fix it manually, or try the following:
================ sample Perfini.txt :================== C:\WINDOWS\system32\axperf.ini:drivername=ASP C:\WINDOWS\system32\esentprf.ini:drivername=ESENT C:\WINDOWS\system32\ftpctrs.ini:drivername=MSFTPSVC C:\WINDOWS\system32\iasperf.ini:drivername=IAS C:\WINDOWS\system32\infoctrs.ini:drivername=inetinfo C:\WINDOWS\system32\mqperf.ini:drivername=MSMQ C:\WINDOWS\system32\msdtcprf.ini:drivername=MSDTC C:\WINDOWS\system32\ntdsctrs.ini:drivername=NTDS C:\WINDOWS\system32\ntfrscon.ini:drivername=FileReplicaConn C:\WINDOWS\system32\ntfrsrep.ini:drivername=FileReplicaSet C:\WINDOWS\system32\perfci.ini:drivername=ContentIndex C:\WINDOWS\system32\perffilt.ini:drivername=ContentFilter C:\WINDOWS\system32\perfwci.ini:drivername=ISAPISearch C:\WINDOWS\system32\pschdprf.ini:drivername=PSched C:\WINDOWS\system32\rasctrs.ini:drivername=RemoteAccess C:\WINDOWS\system32\tslabels.ini:drivername=TermService C:\WINDOWS\system32\w3ctrs.ini:drivername=W3SVC ================ Sample Perfini.txt :==================
================ sample Perfini.txt :==================
C:\WINDOWS\system32\axperf.ini:drivername=ASP C:\WINDOWS\system32\esentprf.ini:drivername=ESENT C:\WINDOWS\system32\ftpctrs.ini:drivername=MSFTPSVC C:\WINDOWS\system32\iasperf.ini:drivername=IAS C:\WINDOWS\system32\infoctrs.ini:drivername=inetinfo C:\WINDOWS\system32\mqperf.ini:drivername=MSMQ C:\WINDOWS\system32\msdtcprf.ini:drivername=MSDTC C:\WINDOWS\system32\ntdsctrs.ini:drivername=NTDS C:\WINDOWS\system32\ntfrscon.ini:drivername=FileReplicaConn C:\WINDOWS\system32\ntfrsrep.ini:drivername=FileReplicaSet C:\WINDOWS\system32\perfci.ini:drivername=ContentIndex C:\WINDOWS\system32\perffilt.ini:drivername=ContentFilter C:\WINDOWS\system32\perfwci.ini:drivername=ISAPISearch C:\WINDOWS\system32\pschdprf.ini:drivername=PSched C:\WINDOWS\system32\rasctrs.ini:drivername=RemoteAccess C:\WINDOWS\system32\tslabels.ini:drivername=TermService C:\WINDOWS\system32\w3ctrs.ini:drivername=W3SVC
================ Sample Perfini.txt :==================
Using this information unload und load the counter in System32:
unlodctr <drivername> lodctr <ini file> <drivername> driver's name from Perfini.txt <ini file> listed “.ini” files in Perfini.txt
Example: ASP Drivername: axperf.ini:drivername=ASP
unlodctr ASP lodctr axperf.ini
Or using the following command you can transfer settings from a working machine to a not working one:
lodctr /s:Perf_all.ini lodctr Perf_all.ini Reboot Check Perfdiag
If nothing else works as you wish or you do not want to waste time troubleshooting and just need the machine up and running without much annoyances you can disable the counters for the problematic service(s), of course at the cost of not having those data if you’ll ever need them.
For this task you can use Exctrlst.exe utility from the OS Resource Kit (see Exctrlst Overview and Exctrlst UI); it disables the specific counter by adding a registry key similar to the example below. You can add these registry entries under the specific counter entry manually if Exctrlst.exe is not available or desirable
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Spooler\Performance Value name: Disable Performance Counters Value type: REG_DWORD Value data: 1
I was discussing this morning with a customer, here’s a useful reminder to this post if you need to deal with “legacy” .evt logs on Vista/Win2008.
wevtutil epl application.evt application.evtx /lf:true
Over the past couple of weeks I got to almost identical cases where the customer had the event log on their servers “spammed” by the following messages:
Event Type: Warning Event Source: WinMgmt Event Category: None Event ID: 40 Date: <date> Time: <time> User: N/A Computer: <computername> Description: WMI ADAP was unable to create the object Win32_PerfRawData_ASPNET_2050727_ASPNETAppsv2050727 for Performance Library ASP.NET_2.0.50727 because error 0x80041001 was returned
Event Type: Warning Event Source: WinMgmt Event Category: None Event ID: 35 Date: <date> Time: <time> User: N/A Computer: <computername> Description: WMI ADAP was unable to load the ASP.NET_2.0.50727 performance library because it returned invalid data: 0x0
Article How to troubleshoot WinMgmt-based performance counter errors does not help in this case (we do not have Event ID 37, 41 or 61).
The fix for this problem was scheduled to be added in .NET 3.5 SP1 (as also explained in this post) then we decided to ship a standalone hotfix too: the KB article is not yet available but you can request the fix (which by the way resolved both my cases) to CSS as usual, asking for 951683.