Symptom:

According to the dump file, there are dozens of application domains for one asp.net web application.

Troubleshooting:

Asp.net starts new appdomain and unloads old one in various scenarios, for example:

·         Machine.Config, Web.Config or Global.asax are modified

·         The bin directory or its contents is modified

·         The number of re-compilations (aspx, ascx or asax) exceeds the limit specified by the <compilation numRecompilesBeforeAppRestart=/> setting in machine.config or web.config  (by default this is set to 15)

·         The physical path of the virtual directory is modified

·         The CAS policy is modified

·         The web service is restarted

·         (2.0 only) Application Sub-Directories are deleted (see Todd’s blog http://blogs.msdn.com/toddca/archive/2006/07/17/668412.aspx  for more info)

This blog article describes the behavior for more details: http://blogs.msdn.com/tess/archive/2006/08/02/asp-net-case-study-lost-session-variables-and-appdomain-recycles.aspx

It will fail to unload the old AppDomain if you are stuck inside unmanaged code which performs an uninterruptable blocking operation.

A System.CannotUnloadAppDomainException will occur and can be found in the dump.

Below is a typical call stack when AppDomain will fail to be unloaded, System.Net.Sockets.Socket.Receive is doing uninterruptable blocking operation:

Child-SP         RetAddr          Call Site

000000000362d410 00000642749e90b0 DomainNeutralILStubClass.IL_STUB(IntPtr, Byte*, Int32, System.Net.Sockets.SocketFlags)

000000000362d4f0 00000642749e9174 System.Net.Sockets.Socket.Receive(Byte[], Int32, Int32, System.Net.Sockets.SocketFlags, System.Net.Sockets.SocketError ByRef)

…….

000000000362dd00 000006427f62ff5d System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(System.Runtime.Remoting.Proxies.MessageData ByRef, Int32)

 

Resolution:

To fix the issue you should prevent the code from calling inside unmanaged code which performs an uninterruptable blocking operation when AppDomain is unloading.

We did a test to use a background thread calling System.Net.Sockets.Socket.Receive always. When reload new AppDomain, the same System.CannotUnloadAppDomainException occurs.

A recycling of application pool process w3wp.exe will force IIS to unload all AppDomains. It can be considered as a workaround.

Regards,

Juntao Zhu