Event Handlers Gone Wild

Toddca — Wed, 06 Sep 2006 00:10:00 GMT

Today I ran across an issue that my good buddy Tess hit with one of her customers a while back.

The application in question here is an ASP.NET web application that is exhibiting fairly high memory usage after almost a day of usage (approx 400-500 MB). Just as in Tess's case the number of event handlers was really high as you can see from this snip of the !dumpheap -stat command:

MD,Count,Size,Name
=====================================
0x024cd82c 876,697 24,547,516 System.ResolveEventHandler
0x019e4300 156,448 29,216,964 System.String
0x000cf740 238 66,842,468 Free
Total 1,464,956 objects, Total size: 182,475,044

Note that when working managed high memory issues I always start with !dumpheap -stat to get the landscape.

The customer stated that they were registering for the AppDomain's AssemblyResolve event in the global.asax and provided the code that showed what they were doing. Now the code below is incorrect and we will get back to that in a bit however even being wrong it still could not account for over 867,000 event handlers being added to AppDomain's event.

public HttpApplication() : base() {
AppDomain.CurrentDomain.AssemblyResolve += new ResolveEventHandler(MyAssemblyResolver.ResolveAssembly);
}

So given the time the application had been up (approx 21hrs) there must be some common code that is getting called to register for these events elsewhere. I found the code in a class they use to derive all of their WebForm classes from; a custom BasePage class. Of course we are having them remove that code and that should get them back in much better shape.

Now how do we fix the global.asax code. Well the problem here is that a new HttpApplication is created each time one is needed by a web application and this means you could have more than one handler registered. Now if you have a really low volume site you will only ever see one of these in most cases, however as your traffic gets heavier additional HttpApplications will be spun up to handle the concurrent requests. Think of the number of HttpApplication objects as your high watermark for the number of concurrent requests your site has seen.

So the fix is to only register for this event handler once and doing this through a static constructor of the HttpApplication would probably be the best approach (though not the only). Since the CLR ensures that static constructors are only called once per AppDomain this will ensure we do not register for the handler multiple times.

static HttpApplication() {
AppDomain.CurrentDomain.AssemblyResolve += new ResolveEventHandler(MyAssemblyResolver.ResolveAssembly);
}

AppPool Running ASP.NET 2.0 May Crash On Process Exit

Toddca — Wed, 01 Mar 2006 19:43:00 GMT

Ran into an interesting issue the other day while working with a customer and I thought I would share what we learned.

Problem
Customer's IIS AppPool running on Windows Server 2003 SP1 with ASP.NET 2.0 RTM installed would crash anytime the AppPool would shutdown.

Debugging
I received a user mode dump file of the crashing process and found the following stack:

0:024> kb
ChildEBP RetAddr Args to Child
0125fd18 6a2a2fef 01892710 800703e3 00000000 0x64006f
0125fd30 79f2f3b0 000003e3 00000000 01894090 webengine!CorThreadPoolCompletionCallback+0x35 0125fd94 79ecb00b 00000000 00000000 80a78be3 mscorwks!ThreadpoolMgr::intermediateThreadProc+0x49
0125fd98 00000000 00000000 80a78be3 00000008 mscorwks!ThreadpoolMgr::intermediateThreadProc+0x49

0:024> r
eax=800703e3 ebx=00000001 ecx=01892710 edx=01894090 esi=01892fd0 edi=000003e3
eip=0064006f esp=0125fd1c ebp=000b9790 iopl=0 nv up ei ng nz na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00010282
0064006f ?? ???

Obviously eip is totally bogus here and hence the crash.

ASP.NET uses Threadpool completion callbacks for a number of things including File change notification callbacks and performance counters.

Looking at the stack I found an ERROR_OPERATION_ABORTED error 3 DWORDs down.

0:024> dd esp
0125fd1c 6a2a2fef 01892710 800703e3 00000000
0125fd2c 01894090 00000000 79f2f3b0 000003e3
0125fd3c 00000000 01894090 09feda8b 00000000

Looking a little bit further down the stack I found the OVERLAPPED_COMPLETION structure which is returning that it was canceled. Not surprising since we are shutting down.

0:024> dt OVERLAPPED_COMPLETION 01894090
+0x000 Internal : 0xc0000120 STATUS_CANCELLED -> The I/O request was canceled.
+0x004 InternalHigh : 0
+0x008 Offset : 0
+0x00c OffsetHigh : 0
+0x008 Pointer : (null)
+0x010 hEvent : (null)
+0x014 pCompletion : 0x01892710 ICompletion

When we take a look at the ICompletion interface we can see our EIP 4 DWORDS within the VTable. So maybe we should not be trying to call our callback when the status is STATUS_CANCELLED but this should be legal. So we still do not know why we are getting this status in the first place.

0:024> dt 0x01892710 ICompletion
+0x000 __VFN_table : 0x01892fd0
0:024> dd 0x01892fd0 l4
01892fd0 01893078 00700070 0043005f 0064006f <<<<this is our EIP

The next step in the investigation was to try to ascertain what this callback was originally for. This was a bit of a fishing expedition in that I needed to try to account for all the IO Completion calls. Fortunately my customer was able to allow me access to his machine so I could debug this live or this would have been really hard to track down. I eventually narrowed it down to the performance counter callback. The performance counter block was NULL and this indicated a problem in initializing the performance counters. A bit of live debugging narrowed the problem down to a permissions problem accessing the registry at HKLM\CurrentControlSet\Services\ASP.NET_2.0.50727\Names. This registry key (along with others) are ACLed in such a way that only a small set of users/groups are allowed to access and write to that key.

Solution
The problem we found was that the NETWORK_SERIVCE account which was running the AppPool was not a member of the IIS_WPG group which is required to access and read the performance registry key. Adding the NETWORK_SERVICE to this group resolved the issue. Needless to say we are tracking this issue and will correct it.

Update
We now have an official KB on this one: http://support.microsoft.com/Default.aspx?scid=kb;en-us;918041&spid=8940&sid=513

Todd's Blog : ASP.NET Debugging

Event Handlers Gone Wild

AppPool Running ASP.NET 2.0 May Crash On Process Exit