June, 2008

  • Never doubt thy debugger

    An interesting fusion cache lock: that’s what the GAC is meant for

    • 5 Comments

    This has been an interesting case where we had an ASP.NET 2.0 application which under load was completely blocked after a few minutes. Since we were talking about a hang/deadlock (as reported from the customer), the !critlist command (you can find the command within the SieExtPub.dll extension) is a good start:

    0:021> !critlist
    CritSec at 7a393800.  Owned by thread 21.  
      Waiting Threads: 6 7 8 9 10 11 12 13 42 72 104 107 108 109 110 111 112 113 114 115 116 
    CritSec at    e194c.  Owned by thread 32.  
      Waiting Threads: 21

    Thread 32 holds a critical section and thread 21 is waiting on it, but thread 21 is also owning another critical section, and about 20 other threads are waiting there… so apparently this is not a real deadlock (21 is waiting on 32 but 32 is not waiting on 21); but what is thread 32 doing?

    0:032> kpL2000
    ChildEBP RetAddr  
    0327fd30 7c822124 ntdll!KiFastSystemCallRet
    0327fd34 77e6baa8 ntdll!NtWaitForSingleObject+0xc
    0327fda4 79e718fd kernel32!WaitForSingleObjectEx+0xac
    0327fde8 79e718c6 mscorwks!PEImage::LoadImage+0x199
    0327fe38 79e7187c mscorwks!CLREvent::WaitEx+0x117
    0327fe48 7a0e288e mscorwks!CLREvent::Wait+0x17
    0327fe9c 7a086e76 mscorwks!Thread::SysSuspendForGC+0x52a
    0327ff88 7a0d867b mscorwks!SVR::GCHeap::SuspendEE+0x16c
    0327ffa8 7a0d8987 mscorwks!SVR::gc_heap::gc_thread_function+0x3b
    0327ffb8 77e66063 mscorwks!SVR::gc_heap::gc_thread_stub+0x9b
    0327ffec 00000000 kernel32!BaseThreadStart+0x34

    It is waiting to start GC but it cannot, because thread 95 has PreEmptive disabled:

    0:032> !threads
    ThreadCount: 69
    UnstartedThread: 0
    BackgroundThread: 69
    PendingThread: 0
    DeadThread: 0
    Hosted Runtime: yes
                                          PreEmptive   GC Alloc           Lock
           ID OSID ThreadOBJ    State     GC       Context       Domain   Count APT Exception
      23    1 1288 0012efb8   1808220 Enabled  1048ebfc:1048efe8 0011f578     0 MTA (Threadpool Worker)
    [...]
      95   35 1114 05424c30   180b222 Disabled 16577334:16579008 00162f38     1 MTA (Threadpool Worker)
      96   36 1b54 05407bf8   180b220 Enabled  145124ec:14512fe8 0011f578     0 MTA (Threadpool Worker)
    [...]

    As you might know, if a thread has PreEmptive GC disabled is because it does not want to be interrupted by the GC; loading a file is a good reason to disable PreEmptive and that’s exactly the case:

    0:095> !clrstack
    OS Thread Id: 0x1114 (95)
    ESP       EIP     
    05e5ecd0 7c82ed54 [HelperMethodFrame_PROTECTOBJ: 05e5ecd0] System.Reflection.Assembly.nLoadFile(System.String, System.Security.Policy.Evidence)
    05e5ef88 793f583f System.Reflection.Assembly.LoadFile(System.String)
    05e5ef9c 04674f82 MyApp.Core.Lib.Managers.LangMan.ResourceFile(System.String, System.String, MyApp.Core.Lib.Managers.ResourceMode)
    05e5efd8 04674f0c MyApp.Core.Lib.Managers.LangMan.ResourceFile(System.String, System.String)
    05e5efe0 04674ee5 MyApp.FPCore.Pages.Page.ResourceFile(System.String, System.String)
    05e5efec 04674d69 FP_Login.formslogin.Page_Load(System.Object, System.EventArgs)
    05e5f028 031bb2b5 [MulticastFrame: 05e5f028] System.EventHandler.Invoke(System.Object, System.EventArgs)
    [...]

    !showstringw from SieExtPub on the first parameter passed to mscorwks!ExplicitBind we can get the name of the file we’re trying to load:

    0:095> kb
    ChildEBP RetAddr  Args to Child              
    05e5e860 7c822124 77e6baa8 00001348 00000000 ntdll!KiFastSystemCallRet
    05e5e864 77e6baa8 00001348 00000000 00000000 ntdll!NtWaitForSingleObject+0xc
    05e5e8d4 79e718fd 00001348 ffffffff 00000000 kernel32!WaitForSingleObjectEx+0xac
    05e5e918 79e718c6 00001348 ffffffff 00000000 mscorwks!PEImage::LoadImage+0x199
    05e5e968 79e7187c ffffffff 00000000 00000000 mscorwks!CLREvent::WaitEx+0x117
    05e5e978 79f90f81 ffffffff 00000000 00000000 mscorwks!CLREvent::Wait+0x17
    05e5e988 79f917ba 7a390970 03c55358 ffffffff mscorwks!CExecutionEngine::WaitForEvent+0x37
    05e5e99c 7a15cfce 03c55358 ffffffff 00000000 mscorwks!ClrWaitEvent+0x17
    05e5e9ac 7a153339 05e5ed7c 00000000 00000000 mscorwks!CSimpleFusionBindSink::Wait+0xe
    05e5e9c8 7a1545d2 03cecd70 0016e3d0 00000200 mscorwks!BindHelper+0x7d
    05e5ec64 7a2d0c7c 05e5ed7c 0016e3d0 00000000 mscorwks!ExplicitBind+0x1a6
    05e5ef80 793f583f 00000000 16577088 1657703c mscorwks!AssemblyNative::LoadFile+0x1de
    [...]
    
    0:095> !showstringw 05e5ed7c 
    D:\MyApp\App\Members\bin\DescMembers.dll

    Looking around in the stack it’s interesting to note that also thread 81 was loading the same file from from ASP.NET temporary folder:

    0:081> kb
    ChildEBP RetAddr  Args to Child              
    0573cf58 7c822124 77e6baa8 000004f4 00000000 ntdll!KiFastSystemCallRet
    0573cf5c 77e6baa8 000004f4 00000000 00000000 ntdll!NtWaitForSingleObject+0xc
    0573cfcc 79e718fd 000004f4 ffffffff 00000000 kernel32!WaitForSingleObjectEx+0xac
    0573d010 79e718c6 000004f4 ffffffff 00000000 mscorwks!PEImage::LoadImage+0x199
    0573d060 79e7187c ffffffff 00000000 00000000 mscorwks!CLREvent::WaitEx+0x117
    0573d070 7a0851cb ffffffff 00000000 00000000 mscorwks!CLREvent::Wait+0x17
    0573d080 79f40e96 00000000 5ed28252 79eaebea mscorwks!SVR::GCHeap::WaitUntilGCComplete+0x32
    0573d0bc 79e797c6 5ed2822a 79eaebea 053ee59a mscorwks!Thread::RareDisablePreemptiveGC+0x1a1
    0573d0e8 79ea91b2 7a3879e0 5ed283e6 79eaebea mscorwks!CrstBase::AcquirePreempLock+0x2e
    0573d124 79eb4132 0573d154 00000000 0573d1b4 mscorwks!PEImage::OpenImage+0x92
    0573d1c0 79eb3d78 053ee598 0573d1f8 0573d1f4 mscorwks!RuntimeOpenImageInternal+0xcd
    0573d208 79eb3cee 053ee598 79eaebb0 00000000 mscorwks!GetAssemblyMDInternalImportEx+0x9d
    0573d21c 79ecf4b9 053ee598 00000000 0573d238 mscorwks!CreateMetaDataImport+0x16
    0573d23c 79ecf462 053ee598 5ed280a6 00000000 mscorwks!CAssemblyManifestImport::Init+0x35
    0573d264 79f2e637 053ee598 0573d27c 000f2a20 mscorwks!CreateAssemblyManifestImport+0x53
    0573d280 7a161a1d 053ee598 03d09c00 0573d9a8 mscorwks!CreateAssemblyFromManifestFile+0x48
    0573d930 7a161e12 0573dcc4 03d09c00 0573d9a8 mscorwks!CAsmDownloadMgr::CreateAssembly+0x8e2
    0573d96c 7a162550 0573dcc4 03d09c00 0573d9a8 mscorwks!CAsmDownloadMgr::DoSetupPushToCache+0x50
    0573dbf4 79f90e65 000f2a20 03d09c00 0573dcc4 mscorwks!CAsmDownloadMgr::DoSetup+0x26a
    0573dc40 79f8f2cc 0573dcc4 00000000 5ed28eb6 mscorwks!CAssemblyDownload::DoSetup+0x7b
    
    0:081> !showstringw 053ee598 
    C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\_mem_bin\bf10f877\41cee659\assembly\dl3\9cf10f5d\0090cb26_8229c601\DescMembers.DLL
    
    0:081> !showstringw 0573dcc4 
    D:\MyApp\App\Members\bin\DescMembers.dll

    Another set of dumps helped to further clarify the situation: thread 94 holds the lock and is trying to load a previously compiled assembly from disk. It is trying to load this in relation to page D:\MyApp\App\Secure\Root\default.aspx and this this is leading to the attempt to load the assembly for global.asax: C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\App_global.asax.ylfot7zw.dll. As part of this assembly load, thread 94 is blocked waiting to acquire a critical section owned by thread 123:

    0:094> !syncblk Index SyncBlock MonitorHeld Recursion Owning Thread Info SyncBlock Owner 140 03f05a84 13 1 03ebd208 714 94 1225a688 System.Web.Compilation.BuildManager ----------------------------- Total 219 CCW 11 RCW 0 ComClassFactory 0 Free 0 0:094> !dso OS Thread Id: 0x714 (94) ESP/REG Object Name 04f7e9e0 790d6654 System.String 04f7eb70 1c571e2c System.Byte[] 04f7ecb8 1c571df0 System.Reflection.AssemblyName 04f7eccc 1c571df0 System.Reflection.AssemblyName 04f7edac 1c571df0 System.Reflection.AssemblyName 04f7edc4 1c571df0 System.Reflection.AssemblyName 04f7edf8 1c571ad0 System.String C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\App_global.asax.ylfot7zw.dll.delete 04f7ee00 1c47a640 System.Object[] (System.Reflection.AssemblyName[]) 04f7ee04 1c5715e8 System.String C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\App_global.asax.ylfot7zw.dll 04f7ee24 1c4775e0 System.Reflection.Assembly 04f7ee28 1c468e58 System.String App_Web_gf9hosur 04f7ee2c 1c428b70 System.Web.Compilation.PreservationFileReader 04f7ee30 121a2004 System.Globalization.CultureInfo 04f7ee34 1c468dbc System.String 110000 04f7ee44 1c1f7e58 System.Web.VirtualPath 04f7ee48 1c428b70 System.Web.Compilation.PreservationFileReader 04f7ee4c 1c4693bc System.Web.Compilation.BuildResultCompiledTemplateType 04f7ee58 1c1f7e58 System.Web.VirtualPath 04f7ee5c 1c428b70 System.Web.Compilation.PreservationFileReader 04f7ee60 1c4693bc System.Web.Compilation.BuildResultCompiledTemplateType 04f7ee7c 1c468cbc System.String fffffff8697ef330 04f7ee84 1c428a68 System.String C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\default.aspx.8cd6fbf4.compiled 04f7ee88 1c428a68 System.String C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\default.aspx.8cd6fbf4.compiled 04f7ee8c 1c1f7e58 System.Web.VirtualPath 04f7ee9c 1c428a68 System.String C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\default.aspx.8cd6fbf4.compiled 04f7eea4 1c428b70 System.Web.Compilation.PreservationFileReader 04f7eea8 1a200b5c System.String .compiled 04f7eeac 1c428970 System.String C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\default.aspx.8cd6fbf4 04f7eeb8 1a200b5c System.String .compiled 04f7eebc 1c428a68 System.String C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\default.aspx.8cd6fbf4.compiled 04f7eec0 1c428b70 System.Web.Compilation.PreservationFileReader 04f7eec4 1225b41c System.Web.Compilation.StandardDiskBuildResultCache 04f7eec8 1c1f7e58 System.Web.VirtualPath 04f7eed8 1c428a68 System.String C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\default.aspx.8cd6fbf4.compiled 04f7eee8 1c1f7e58 System.Web.VirtualPath 04f7eef8 1c1f7e58 System.Web.VirtualPath 04f7ef00 1225a688 System.Web.Compilation.BuildManager 04f7ef08 1c427124 System.String default.aspx.8cd6fbf4 04f7ef0c 1225a7c4 System.Web.Util.SimpleRecyclingCache 04f7ef20 1c427100 System.String 8cd6fbf4 04f7ef2c 1c1f7e58 System.Web.VirtualPath 04f7ef30 1225a688 System.Web.Compilation.BuildManager 04f7ef44 1c1f7e58 System.Web.VirtualPath 04f7ef50 1225a688 System.Web.Compilation.BuildManager 04f7ef54 1c1f7e58 System.Web.VirtualPath 04f7ef70 1c418fb8 System.Web.ApplicationImpersonationContext 04f7ef74 1c418fb8 System.Web.ApplicationImpersonationContext 04f7ef78 1c418fb8 System.Web.ApplicationImpersonationContext 04f7ef88 1c1f7e58 System.Web.VirtualPath 04f7ef9c 1c418fb8 System.Web.ApplicationImpersonationContext 04f7efa8 1c418fb8 System.Web.ApplicationImpersonationContext 04f7efac 1c1f7e58 System.Web.VirtualPath 04f7efc4 1c418fb8 System.Web.ApplicationImpersonationContext 04f7efcc 1c418f88 System.Object[] (System.Type[]) 04f7efd0 1c418fac System.Web.UI.PageHandlerFactory 04f7efe4 1c1f7e58 System.Web.VirtualPath 04f7efe8 1c1f7c2c System.Web.HttpContext 04f7f000 1c1f7c2c System.Web.HttpContext 04f7f004 1444b800 System.RuntimeType 04f7f024 1239e718 System.String System.Web.UI.PageHandlerFactory 04f7f02c 1c393abc System.Collections.Hashtable 04f7f034 1c1f73a4 System.String D:\MyApp\App\Secure\Root\default.aspx 04f7f038 1c1f7c2c System.Web.HttpContext 04f7f03c 1c1f7e58 System.Web.VirtualPath 04f7f050 1c1f7c2c System.Web.HttpContext 04f7f054 1c418fac System.Web.UI.PageHandlerFactory 04f7f058 1c1f7c2c System.Web.HttpContext 04f7f060 1c1f73a4 System.String D:\MyApp\App\Secure\Root\default.aspx 04f7f064 1c1f7e58 System.Web.VirtualPath 04f7f068 1c1f7338 System.String POST 04f7f06c 1c418fac System.Web.UI.PageHandlerFactory 04f7f070 1c4187b0 System.Web.ApplicationImpersonationContext 04f7f074 1c1f87b8 ASP.global_asax 04f7f07c 1c1f73a4 System.String D:\MyApp\App\Secure\Root\default.aspx 04f7f080 1c1f7e58 System.Web.VirtualPath 04f7f084 1c1f7338 System.String POST 04f7f088 1c4187b0 System.Web.ApplicationImpersonationContext 04f7f090 1c1f7c2c System.Web.HttpContext 04f7f0a4 1c1f87b8 ASP.global_asax 04f7f0a8 1c1f87b8 ASP.global_asax 04f7f0ac 1c1f7cd4 System.Web.HttpRequest 04f7f0b0 1c1f7c2c System.Web.HttpContext 04f7f0c0 1c1f73a4 System.String D:\MyApp\App\Secure\Root\default.aspx 04f7f0c4 1c1f7e58 System.Web.VirtualPath 04f7f0c8 1c1f7338 System.String POST 04f7f0cc 1c1f7c2c System.Web.HttpContext 04f7f0d0 1c1f87b8 ASP.global_asax 04f7f0e8 1c411f58 System.Web.HttpApplication+MapHandlerExecutionStep 04f7f0ec 1c1f87b8 ASP.global_asax 04f7f0f0 1c1f7c2c System.Web.HttpContext 04f7f104 1c1f7c2c System.Web.HttpContext 04f7f108 1c1f87b8 ASP.global_asax 04f7f11c 1c1f87b8 ASP.global_asax 04f7f120 1c412258 System.Web.HttpApplication+ThreadContext 04f7f124 1c1f7c2c System.Web.HttpContext 04f7f128 1c1f87b8 ASP.global_asax 04f7f12c 1c412238 System.Web.AspNetSynchronizationContext 04f7f134 1c411cdc System.Web.HttpApplication+ApplicationStepManager 04f7f158 1c1f7c2c System.Web.HttpContext 04f7f15c 1c412218 System.Web.HttpAsyncResult 04f7f160 1c1f87b8 ASP.global_asax 04f7f164 121c489c System.AsyncCallback 04f7f170 1c1f4ff4 System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6 04f7f174 1c1f87b8 ASP.global_asax 04f7f178 1c1f7e6c System.Web.HttpWriter 04f7f180 1c1f7c2c System.Web.HttpContext 04f7f184 121c489c System.AsyncCallback 04f7f188 1c1f7c2c System.Web.HttpContext 04f7f18c 1c1f4ff4 System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6 04f7f190 121c4138 System.Web.HttpRuntime 04f7f1a8 1c1f72dc System.String D:\MyApp\App\Secure\ 04f7f1ac 1c1f4ff4 System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6 04f7f1bc 121a1038 System.String D:\MyApp\App\Secure\ 04f7f1c0 1c1f4ff4 System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6 04f7f1f8 122f2880 System.Web.Hosting.ISAPIRuntime 0:094> !do 1c1f4ff4 Name: System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6 MethodTable: 663a6f84 EEClass: 663a6f04 Size: 256(0x100) bytes (C:\WINDOWS\assembly\GAC_32\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll) Fields: MT Field Offset Type VT Attr Value Name 79105fac 400133a 4 System.DateTime 1 instance 1c1f4ff8 _startTime 7910ead0 400133b c System.Guid 1 instance 1c1f5000 _traceId 7912ab5c 400133c 194 System.String[][] 0 shared static s_HTTPStatusDescriptions >> Domain:Value 001171e8:NotInit 00159f68:1a227064 03ea3b78:122f6384 << 79124304 400133d 198 System.Object[] 0 shared static s_serverVarFromRequestHeaderNames >> Domain:Value 001171e8:NotInit 00159f68:1a2271ac 03ea3b78:122f64cc << 79124304 400133e 19c System.Object[] 0 shared static s_requestHeaderNames >> Domain:Value 001171e8:NotInit 00159f68:1a22725c 03ea3b78:122f657c << 79124304 400133f 1a0 System.Object[] 0 shared static s_responseHeaderNames >> Domain:Value 001171e8:NotInit 00159f68:1a22730c 03ea3b78:122f662c << 790feb48 4001340 1a4 ...ections.Hashtable 0 shared static s_requestHeadersLoookupTable >> Domain:Value 001171e8:NotInit 00159f68:1a227394 03ea3b78:122f66b4 << 790feb48 4001341 1a8 ...ections.Hashtable 0 shared static s_responseHeadersLoookupTable >> Domain:Value 001171e8:NotInit 00159f68:1a22745c 03ea3b78:122f677c << 790fe238 4001b3a 70 System.IntPtr 1 instance 3066598 _ecb 790fe238 4001b3b 74 System.IntPtr 1 instance 2d0 _token 7910ead0 4001b3c a8 System.Guid 1 instance 1c1f509c _traceId 790fa4b0 4001b3d 1c System.String 0 instance 1c1f7338 _method 790fa4b0 4001b3e 20 System.String 0 instance 1c1f7354 _path 790fa4b0 4001b3f 24 System.String 0 instance 1c1f7354 _filePath 790fa4b0 4001b40 28 System.String 0 instance 790d6654 _pathInfo 790fa4b0 4001b41 2c System.String 0 instance 1c1f73a4 _pathTranslated 790fa4b0 4001b42 30 System.String 0 instance 141a4b28 _appPath 790fa4b0 4001b43 34 System.String 0 instance 1c1f72dc _appPathTranslated [...] 0:094> !do 1c1f73a4 Name: System.String MethodTable: 790fa4b0 EEClass: 790fa410 Size: 150(0x96) bytes (C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll) String: D:\MyApp\App\Secure\Root\default.aspx Fields: MT Field Offset Type VT Attr Value Name 790fedf4 4000096 4 System.Int32 1 instance 67 m_arrayLength 790fedf4 4000097 8 System.Int32 1 instance 66 m_stringLength 790fbfcc 4000098 c System.Char 1 instance 44 m_firstChar 790fa4b0 4000099 10 System.String 0 shared static Empty >> Domain:Value 001171e8:790d6654 00159f68:790d6654 03ea3b78:790d6654 << 7912474c 400009a 14 System.Char[] 0 shared static WhitespaceChars >> Domain:Value 001171e8:141a0624 00159f68:141a77f8 03ea3b78:121a2374 << 0:094> !do 1c1f87b8 Name: ASP.global_asax MethodTable: 04c35adc EEClass: 04ff1dd8 Size: 140(0x8c) bytes (C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\App_global.asax.ylfot7zw.dll) Fields: MT Field Offset Type VT Attr Value Name 663a2450 4000dc8 4 ...pApplicationState 0 instance 122e14dc _state 663a2750 4000dc9 8 ...m.Web.HttpContext 0 instance 00000000 _initContext 663a359c 4000dca c ...b.HttpAsyncResult 0 instance 1c412218 _ar 663a3394 4000dcb 10 ...pModuleCollection 0 instance 1c3a648c _moduleCollection 79118214 4000dcc 14 ...ncipal.IPrincipal 0 instance 1c413d68 _savedPrincipal 79105020 4000dcd 78 System.Boolean 1 instance 1 _restorePrincipal 7a74da98 4000de8 18 ....EventHandlerList 0 instance 1c3f7408 _events 663a2aec 4000de9 1c ...ventHandlersTable 0 instance 1c401980 _asyncEvents [...]

     

    0:094> !critlist
    CritSec at 7a38ee88. Owned by thread 123.
    Waiting Threads: 26 66 94

    !dso output already shows objects and strings in the same order as they are stored on the stack and executed, so it’s quite easy to spot interesting file paths, URLs etc… and often that is enough to suggest which files are involved, which page or web service you’re accessing and so on; of course you can always use !do <address> to dig into the stack to be 100% sure you’re on the right path.

    Using the same commands above we found that thread 123 is trying to explicitly load "D:\MyApp\App\Secure\bin\secure.dll" whilst executing "ASP.logout_aspx"; after transitioning from managed to unmanaged code, this thread then gets blocked waiting for garbage collection to complete (because it needs to disable pre-emptive mode in order to acquire a particular lock).

    In summary, the hang was due to a complicated deadlock situation involving “D:\MyApp\App\Secure\Root\default.aspx” being compiled by ASP.NET 2.0 compilation engine. This leads to an attempt to load “C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\00c69763\2d16cf3d\App_global.asax.ylfot7zw.dll” from disk (the default.aspx page has a dependency on the global.asax) which in turn leads to an attempt to acquire a critical section owned by thread 123. Thread 123 is processing the request for “logout.aspx” which involves the execution of “MyApp.Core.Library.Managers.LanguageManager.ResourceFile” which leads to an attempt to load “D:\MyApp\App\Secure\bin\secure.dll” from disk. However this load operation gets blocked waiting for garbage collection to complete as it must acquire a particular lock when GC is not running. The GC was triggered (incidentally) by thread 56, but the GC has not actually commenced yet. The primary GC thread (29) has asked all thread executing managed code to get to a 'safe' point where GC can begin, but that has not happened (and will never happen) because some of those threads are blocked doing assembly load operations and are waiting on critical section.

     

    Finally, the resolution: the customer confirmed the involved assemblies are “resource only” files which they access from everywhere in their quite complex application (basically those were localization assemblies they were explicitly loading when needed to localize their interface, error messages etc…); we strong-named those “sensitive” assemblies and installed them in the GAC: easy solution, when you know which is the problem…

    By the way, there are a couple hotfixes available for this situation (see http://support.microsoft.com/kb/936757/en-us and 946644, KB not yet available but you can ask for the fix to CSS), but did not help in this case.

     

    Carlo

    Quote of the day:
    Tact is the ability to describe others as they see themselves. - Abraham Lincoln
  • Never doubt thy debugger

    autoConfig=”false” effects

    • 3 Comments

    My post on autoConfig triggered and interesting discussion with my colleague Cenk about which is the real effect of this setting; one of his customers asked what to do if they needed to adjust only one or two values and not all the ones discussed in Contention, poor performance, and deadlocks when you make Web service requests from ASP.NET applications, they wanted to be sure they could change only what they needed to fine tune and forget about the rest. The short answer is: you can change only what you need to and leave he rest as is, since we have default values hardcoded to improve performance so you’re safe (provided the values you’re changing make sense in the context, of course).

    If we have a look at System.Web.Configuration.ProcessModelSection with Windbg, we can see all the properties we have in <processModel> and _defaultValue corresponds to the defaults we have in machine.config.comments:

    0:000> !do 06904868 
    Name: System.Web.Configuration.ProcessModelSection
    MethodTable: 6638dcb0
    EEClass: 663a0604
    Size: 64(0x40) bytes
     (C:\Windows\assembly\GAC_32\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll)
    Fields:
          MT    Field   Offset                 Type VT     Attr    Value Name
    7910be50  400014d       30       System.Boolean  1 instance        0 _bDataToWrite
    7910be50  400014e       31       System.Boolean  1 instance        0 _bModified
    7910be50  400014f       32       System.Boolean  1 instance        1 _bReadOnly
    7910be50  4000150       33       System.Boolean  1 instance        0 _bElementPresent
    7910be50  4000151       34       System.Boolean  1 instance        1 _bInited
    648eff64  4000152        4 ...ionLockCollection  0 instance 00000000 _lockedAttributesList
    648eff64  4000153        8 ...ionLockCollection  0 instance 00000000 _lockedAllExceptAttributesList
    648eff64  4000154        c ...ionLockCollection  0 instance 00000000 _lockedElementsList
    648eff64  4000155       10 ...ionLockCollection  0 instance 00000000 _lockedAllExceptElementsList
    648ea704  4000156       14 ...nfigurationValues  0 instance 069048a8 _values
    790fd8c4  4000157       18        System.String  0 instance 00000000 _elementTagName
    648f032c  4000158       1c ...lementInformation  0 instance 00000000 _evaluationElement
    648eee30  4000159       20 ...onElementProperty  0 instance 068fce20 _elementProperty
    648f53d0  400015a       2c         System.Int32  1 instance        0 _fItemLocked
    648f0a48  400015b       24 ...ontextInformation  0 instance 00000000 _evalContext
    648eda8c  400015c       28 ...nfigurationRecord  0 instance 028bea78 _configRecord
    7912d8f8  4000148        8      System.Object[]  0   shared   static s_lockAttributeNames
        >> Domain:Value  023d04b0:NotInit  024169d0:068fcd1c <<
    79101fe4  4000149        c ...ections.Hashtable  0   shared   static s_propertyBags
        >> Domain:Value  023d04b0:NotInit  024169d0:068fcd40 <<
    00000000  400014a       10                       0   shared   static s_perTypeValidators
        >> Domain:Value  023d04b0:NotInit  024169d0:00000000 <<
    790fd0f0  400014b       14        System.Object  0   shared   static s_nullPropertyValue
        >> Domain:Value  023d04b0:NotInit  024169d0:068fce08 <<
    648eee30  400014c       18 ...onElementProperty  0   shared   static s_ElementProperty
        >> Domain:Value  023d04b0:NotInit  024169d0:068fce20 <<
    648eee84  400015d       38 ...ectionInformation  0 instance 069049b8 _section
    648eee30  4001928      74c ...onElementProperty  0   shared   static s_elemProperty
        >> Domain:Value  023d04b0:NotInit  024169d0:06903da8 <<
    7911228c  4001929      748      System.TimeSpan  1   shared   static DefaultClientConnectedCheck
        >> Domain:Value  023d04b0:NotInit  024169d0:06903850 <<
    648ef118  400192a      750 ...ropertyCollection  0   shared   static _properties
        >> Domain:Value  023d04b0:NotInit  024169d0:06904714 <<
    648ef05c  400192b      754 ...igurationProperty  0   shared   static _propEnable
        >> Domain:Value  023d04b0:NotInit  024169d0:06903db4 <<
    648ef05c  400192c      758 ...igurationProperty  0   shared   static _propTimeout
        >> Domain:Value  023d04b0:NotInit  024169d0:06903df4 <<
    648ef05c  400192d      75c ...igurationProperty  0   shared   static _propIdleTimeout
        >> Domain:Value  023d04b0:NotInit  024169d0:06903e38 <<
    648ef05c  400192e      760 ...igurationProperty  0   shared   static _propShutdownTimeout
        >> Domain:Value  023d04b0:NotInit  024169d0:06903e7c <<
    648ef05c  400192f      764 ...igurationProperty  0   shared   static _propRequestLimit
        >> Domain:Value  023d04b0:NotInit  024169d0:06903ed8 <<
    648ef05c  4001930      768 ...igurationProperty  0   shared   static _propRequestQueueLimit
        >> Domain:Value  023d04b0:NotInit  024169d0:06903f74 <<
    648ef05c  4001931      76c ...igurationProperty  0   shared   static _propRestartQueueLimit
        >> Domain:Value  023d04b0:NotInit  024169d0:06903ff8 <<
    648ef05c  4001932      770 ...igurationProperty  0   shared   static _propMemoryLimit
        >> Domain:Value  023d04b0:NotInit  024169d0:06904064 <<
    648ef05c  4001933      774 ...igurationProperty  0   shared   static _propWebGarden
        >> Domain:Value  023d04b0:NotInit  024169d0:069040a4 <<
    648ef05c  4001934      778 ...igurationProperty  0   shared   static _propCpuMask
        >> Domain:Value  023d04b0:NotInit  024169d0:069040e4 <<
    648ef05c  4001935      77c ...igurationProperty  0   shared   static _propUserName
        >> Domain:Value  023d04b0:NotInit  024169d0:06904118 <<
    648ef05c  4001936      780 ...igurationProperty  0   shared   static _propPassword
        >> Domain:Value  023d04b0:NotInit  024169d0:0690414c <<
    648ef05c  4001937      784 ...igurationProperty  0   shared   static _propLogLevel
        >> Domain:Value  023d04b0:NotInit  024169d0:06904180 <<
    648ef05c  4001938      788 ...igurationProperty  0   shared   static _propClientConnectedCheck
        >> Domain:Value  023d04b0:NotInit  024169d0:069041d4 <<
    648ef05c  4001939      78c ...igurationProperty  0   shared   static _propComAuthenticationLevel
        >> Domain:Value  023d04b0:NotInit  024169d0:06904218 <<
    648ef05c  400193a      790 ...igurationProperty  0   shared   static _propComImpersonationLevel
        >> Domain:Value  023d04b0:NotInit  024169d0:0690426c <<
    648ef05c  400193b      794 ...igurationProperty  0   shared   static _propResponseDeadlockInterval
        >> Domain:Value  023d04b0:NotInit  024169d0:069042c0 <<
    648ef05c  400193c      798 ...igurationProperty  0   shared   static _propResponseRestartDeadlockInterval
        >> Domain:Value  023d04b0:NotInit  024169d0:06904304 <<
    648ef05c  400193d      79c ...igurationProperty  0   shared   static _propAutoConfig
        >> Domain:Value  023d04b0:NotInit  024169d0:06904348 <<
    648ef05c  400193e      7a0 ...igurationProperty  0   shared   static _propMaxWorkerThreads
        >> Domain:Value  023d04b0:NotInit  024169d0:069043ac <<
    648ef05c  400193f      7a4 ...igurationProperty  0   shared   static _propMaxIOThreads
        >> Domain:Value  023d04b0:NotInit  024169d0:0690443c <<
    648ef05c  4001940      7a8 ...igurationProperty  0   shared   static _propMinWorkerThreads
        >> Domain:Value  023d04b0:NotInit  024169d0:069044cc <<
    648ef05c  4001941      7ac ...igurationProperty  0   shared   static _propMinIOThreads
        >> Domain:Value  023d04b0:NotInit  024169d0:0690455c <<
    648ef05c  4001942      7b0 ...igurationProperty  0   shared   static _propServerErrorMessageFile
        >> Domain:Value  023d04b0:NotInit  024169d0:069045c8 <<
    648ef05c  4001943      7b4 ...igurationProperty  0   shared   static _propPingFrequency
        >> Domain:Value  023d04b0:NotInit  024169d0:069045fc <<
    648ef05c  4001944      7b8 ...igurationProperty  0   shared   static _propPingTimeout
        >> Domain:Value  023d04b0:NotInit  024169d0:06904640 <<
    648ef05c  4001945      7bc ...igurationProperty  0   shared   static _propMaxAppDomains
        >> Domain:Value  023d04b0:NotInit  024169d0:069046a8 <<
    79102290  4001946      938         System.Int32  1   shared   static cpuCount
        >> Domain:Value  023d04b0:NotInit  024169d0:2 <<
    
    0:000> !do 0690443c 
    Name: System.Configuration.ConfigurationProperty
    MethodTable: 648ef05c
    EEClass: 648eefec
    Size: 52(0x34) bytes
     (C:\Windows\assembly\GAC_MSIL\System.Configuration\2.0.0.0__b03f5f7f11d50a3a\System.Configuration.dll)
    Fields:
          MT    Field   Offset                 Type VT     Attr    Value Name
    790fd8c4  4000277        4        System.String  0 instance 06903c08 _name
    790fd8c4  4000278        8        System.String  0 instance 06903c08 _providedName
    790fd8c4  4000279        c        System.String  0 instance 00000000 _description
    79106894  400027a       10          System.Type  0 instance 068b7554 _type
    790fd0f0  400027b       14        System.Object  0 instance 06904430 _defaultValue
    7a7588dc  400027c       18 ...del.TypeConverter  0 instance 00000000 _converter
    648ef1a0  400027d       2c         System.Int32  1 instance        0 _options
    648f022c  400027e       1c ...tionValidatorBase  0 instance 06904418 _validator
    790fd8c4  400027f       20        System.String  0 instance 00000000 _addElementName
    790fd8c4  4000280       24        System.String  0 instance 00000000 _removeElementName
    790fd8c4  4000281       28        System.String  0 instance 00000000 _clearElementName
    648f022c  4000274       68 ...tionValidatorBase  0   shared   static NonEmptyStringValidator
        >> Domain:Value  023d04b0:NotInit  024169d0:068fcc00 <<
    648f022c  4000275       6c ...tionValidatorBase  0   shared   static DefaultValidatorInstance
        >> Domain:Value  023d04b0:NotInit  024169d0:068fcc14 <<
    790fd8c4  4000276       70        System.String  0   shared   static DefaultCollectionPropertyName
        >> Domain:Value  023d04b0:NotInit  024169d0:068fcbec <<
    
    0:000> !do 06904430 
    Name: System.Int32
    MethodTable: 79102290
    EEClass: 79102218
    Size: 12(0xc) bytes
     (C:\Windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
    Fields:
          MT    Field   Offset                 Type VT     Attr    Value Name
    79102290  40003e9        4         System.Int32  1 instance       20 m_value
    <processModel
        enable = "true" [true|false]
        timeout = "10675199.02:48:05.4775807" [HH:MM:SS]
        idleTimeout = "10675199.02:48:05.4775807" [HH:MM:SS]
        shutdownTimeout = "00:00:05" [HH:MM:SS]
        requestLimit = "2147483647" [number]
        requestQueueLimit = "5000" [number]
        restartQueueLimit = "10" [number]
        memoryLimit = "60" [number]
        webGarden = "false" [true|false]
        cpuMask = "0xffffffff" [number]
        userName = "machine" [String]
        password = "AutoGenerate" [String]
        logLevel = "Errors" [None | All | Errors]
        clientConnectedCheck = "00:00:05" [HH:MM:SS]
        comAuthenticationLevel = "Connect" [None | Call | Connect | Default | Pkt | PktIntegrity | PktPrivacy]
        comImpersonationLevel = "Impersonate" [Default | Anonymous | Delegate | Identify | Impersonate]
        responseDeadlockInterval = "00:03:00" [HH:MM:SS]
        responseRestartDeadlockInterval = "00:03:00" [HH:MM:SS]
        autoConfig = "false" [true|false]
        maxWorkerThreads = "20" [number]
        maxIoThreads = "20" [number]
        minWorkerThreads = "1" [number]
        minIoThreads = "1" [number]
        serverErrorMessageFile = "" [String]
        pingFrequency = "10675199.02:48:05.4775807" [HH:MM:SS]
        pingTimeout = "10675199.02:48:05.4775807" [HH:MM:SS]
        maxAppDomains = "2000" [number]
    />

    Internally to debug managed code we use a Windbg extension which works well with private symbols and has some additional commands the public sos.dll does not have and one of those shows the ThreadPool configuration, very useful to have a quick report of those configuration values.

    After the discussion I found myself thinking how to extract those values from a dump using only public symbols and the public sos extension everyone has access to; two of them are quite easy to find out (you can get a full list of members of ThreadpoolMgr class running mscorwks!ThreadpoolMgr*):

    0:000> ?poi(mscorwks!ThreadpoolMgr::NumberOfProcessors)
    Evaluate expression: 2 = 00000002
    0:000> ?poi(mscorwks!ThreadpoolMgr::MaxLimitTotalWorkerThreads) 
    Evaluate expression: 200 = 000000c8

    From System.Web.RequestQueue we can get _minLocalFreeThreads, _minExternFreeThreads and _queueLimit (RequestQueueLimit):

    Name: System.Web.RequestQueue
    MethodTable: 663aebf0
    EEClass: 663aeb80
    Size: 64(0x40) bytes
     (C:\Windows\assembly\GAC_32\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll)
    Fields:
          MT    Field   Offset                 Type VT     Attr    Value Name
    79102290  4001203       14         System.Int32  1 instance      176 _minExternFreeThreads
    79102290  4001204       18         System.Int32  1 instance      152 _minLocalFreeThreads
    79102290  4001205       1c         System.Int32  1 instance     5000 _queueLimit
    7911228c  4001206       2c      System.TimeSpan  1 instance 0691da2c _clientConnectedTime
    7910be50  4001207       28       System.Boolean  1 instance        1 _iis6
    791087cc  4001208        4 ...Collections.Queue  0 instance 0691da40 _localQueue
    791087cc  4001209        8 ...Collections.Queue  0 instance 0691daf4 _externQueue
    79102290  400120a       20         System.Int32  1 instance        0 _count
    791186fc  400120b        c ...ding.WaitCallback  0 instance 0691dba8 _workItemCallback
    79102290  400120c       24         System.Int32  1 instance        0 _workItemCount
    7910be50  400120d       29       System.Boolean  1 instance        0 _draining
    7911228c  400120e       34      System.TimeSpan  1 instance 0691da34 _timerPeriod
    791127fc  400120f       10 ...m.Threading.Timer  0 instance 0691dbe8 _timer

    I was trying to find a convenient way to get the remaining three values (autoConfig, maxConnection and Number of timers) when I found this post from Tom, and now my quest is over: !DumpThreadConfig is the same command we have in the internal Windbg extension I mentioned and also the output is the same… so apparently you just have to be patient and wait for the next Debugging Tools release smile_regular

    Carlo

    Quote of the day:
    There is nothing new under the sun but there are lots of old things we don't know. - Ambrose Bierce
  • Never doubt thy debugger

    Ups and downs of ASP.NET Performance Counters

    • 1 Comments

    Perfmon is likely the first tool you think to when it comes to monitor some internals and performance of your application and it’s relatively easy to find out information and resources on the Internet; unfortunately this seems to also be a fragile component and every now and then we receive new calls about it: event log spammed with warnings about missing or corrupted counters, values returned are inconsistent or clearly wrong etc…

    Messages like the following can appear after for example installing the .NET Framework 2.0 on your machine:

    The Open Procedure for service "ASP.NET" in DLL "C:\Windows\Microsoft.NET\Framework\v2.0.50727\aspnet_perf.dll" failed. Performance data for this service will not be available. Status code returned is data DWORD 0

    To note that such warnings are logged even if your application is not running.

    Some troubleshooting

    First of all you can try to disable any third party services and processes (for example using msconfig.exe or Autoruns) and see if the problem still reproduces. If it does, you can use How to manually rebuild Performance Counter Library values to try to fix it manually, or try the following:

    • Uninstall the  .NET Framework 2.0
    • If available for Perfc009.dat, Perfh009.dat and Perfh007.dat , Perfh007.dat, backup the registry keys
      • HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services
      • HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\Current Version\Perflib
    • Open a command prompt to your System32 folder
      • REN perf?009.dat *.bak
      • Expand <CD_drive_letter:>\i386\perfc009.DA_ %systemroot%\system32\perfc009.DAT
      • Expand <CD_drive_letter:>\i386\perfd009.DA_ %systemroot%\system32\perfd009.DAT
      • Expand <CD_drive_letter:>\i386\perfh009.DA_ %systemroot%\system32\perfh009.DAT
      • Expand <CD_drive_letter:>\i386\perfi009.DA_ %systemroot%\system32\perfi009.DAT
      • REN perf?007.dat *.bak
      • Expand <CD_drive_letter:>\i386\perfc007.DA_ %systemroot%\system32\perfc007.DAT
      • Expand <CD_drive_letter:>\i386\perfd007.DA_ %systemroot%\system32\perfd007.DAT
      • Expand <CD_drive_letter:>\i386\perfh007.DA_ %systemroot%\system32\perfh007.DAT
      • Expand <CD_drive_letter:>\i386\perfi007.DA_ %systemroot%\system32\perfi007.DAT
    • Open your system registry (be very careful!) to HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Perflib and set
      • Last Counter = 1846 (decimal) or 736 (hex)
      • Last Help = 1847 (decimal) or 737 (hex)
    • Now move to HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services
    • Search for the subkey Performance  in services and set the values
      • First Counter = 0
      • First Help = 0
      • Last Counter = 0
      • Last Help = 0
    • Back to the command prompt (still in System32)
    • findstr drivername *.ini > Perfini.txt
    • open  Perfini.txt  and note  drivername=nnn

    ================  sample  Perfini.txt :==================

    C:\WINDOWS\system32\axperf.ini:drivername=ASP
    C:\WINDOWS\system32\esentprf.ini:drivername=ESENT
    C:\WINDOWS\system32\ftpctrs.ini:drivername=MSFTPSVC
    C:\WINDOWS\system32\iasperf.ini:drivername=IAS
    C:\WINDOWS\system32\infoctrs.ini:drivername=inetinfo
    C:\WINDOWS\system32\mqperf.ini:drivername=MSMQ
    C:\WINDOWS\system32\msdtcprf.ini:drivername=MSDTC
    C:\WINDOWS\system32\ntdsctrs.ini:drivername=NTDS
    C:\WINDOWS\system32\ntfrscon.ini:drivername=FileReplicaConn
    C:\WINDOWS\system32\ntfrsrep.ini:drivername=FileReplicaSet
    C:\WINDOWS\system32\perfci.ini:drivername=ContentIndex
    C:\WINDOWS\system32\perffilt.ini:drivername=ContentFilter
    C:\WINDOWS\system32\perfwci.ini:drivername=ISAPISearch
    C:\WINDOWS\system32\pschdprf.ini:drivername=PSched
    C:\WINDOWS\system32\rasctrs.ini:drivername=RemoteAccess
    C:\WINDOWS\system32\tslabels.ini:drivername=TermService
    C:\WINDOWS\system32\w3ctrs.ini:drivername=W3SVC

    ================  Sample Perfini.txt :==================

    Using this information  unload und load the counter in  System32:

    unlodctr <drivername>
    lodctr <ini file>
    <drivername>  driver's name from  Perfini.txt
    <ini file>  listed  “.ini” files  in  Perfini.txt  

    Example:  ASP Drivername: axperf.ini:drivername=ASP 

    unlodctr ASP
    lodctr axperf.ini

    Or using the following command you can transfer settings from a  working machine to a not working one: 

    lodctr /s:Perf_all.ini
    lodctr Perf_all.ini
    Reboot
    Check Perfdiag 

    Disable not needed counters (as a last resource)

    If nothing else works as you wish or you do not want to waste time troubleshooting and just need the machine up and running without much annoyances you can disable the counters for the problematic service(s), of course at the cost of not having those data if you’ll ever need them.

    For this task you can use Exctrlst.exe utility from the OS Resource Kit (see Exctrlst Overview and Exctrlst UI); it disables the specific counter by adding a registry key similar to the example below. You can add these registry entries under the specific counter entry manually if Exctrlst.exe is not available or desirable

    HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Spooler\Performance
    Value name: Disable Performance Counters
    Value type: REG_DWORD
    Value data: 1

    A few fixes

    Some How To and general articles

     

    Carlo

    Quote of the day:
    The only time people dislike gossip is when you gossip about them. - Will Rogers
  • Never doubt thy debugger

    LogParser & event logs on Vista/Windows 2008

    • 0 Comments

    I was discussing this morning with a customer, here’s a useful reminder to this post if you need to deal with “legacy” .evt logs on Vista/Win2008.

          wevtutil epl application.evt application.evtx /lf:true

    Carlo

    Quote of the day:
    That's the secret to life... replace one worry with another.... - Charles M. Schulz
  • Never doubt thy debugger

    WMI warnings 35 and 40 with ASP.NET

    • 11 Comments

    Over the past couple of weeks I got to almost identical cases where the customer had the event log on their servers “spammed” by the following messages:

    Event Type:      Warning
    Event Source:   WinMgmt
    Event Category:           None
    Event ID:           40
    Date:                <date>
    Time:                <time>
    User:                N/A
    Computer:        <computername>
    Description:
    WMI ADAP was unable to create the object Win32_PerfRawData_ASPNET_2050727_ASPNETAppsv2050727 for Performance Library ASP.NET_2.0.50727 because error 0x80041001 was returned

    Event Type:      Warning
    Event Source:   WinMgmt
    Event Category:           None
    Event ID:           35
    Date:                <date>
    Time:                <time>
    User:                N/A
    Computer:        <computername>
    Description:
    WMI ADAP was unable to load the ASP.NET_2.0.50727 performance library because it returned invalid data: 0x0

    Article How to troubleshoot WinMgmt-based performance counter errors does not help in this case (we do not have Event ID 37, 41 or 61).

    The fix for this problem was scheduled to be added in .NET 3.5 SP1 (as also explained in this post) then we decided to ship a standalone hotfix too: the KB article is not yet available but you can request the fix (which by the way resolved both my cases) to CSS as usual, asking for 951683.

     

     

    Carlo

    Quote of the day:
    Always do right. This will gratify some people and astonish the rest. - Mark Twain
Page 1 of 1 (5 items)