I got two cases with different products:
1. ASP.NET 2.0 application stops responding intermittently.
2. WF 4.0 workflow service reported the following exception:
Message: Async Callback threw an exception.
Message: The execution of an InstancePersistenceCommand was interrupted because another valid InstanceHandle holds a lock on instance ‘xxxxxxxx', indicating that a non-stale copy of the instance is already loaded. The loaded copy of the instance and its associated InstanceHandle should be used or unloaded.
Although they had the different symptoms, the cause is the same. I used “ASP.NET 2.0 application stops responding intermittently” issue as the sample.
As usual we captured hang dump files for w3wp.exe while the ASP.NET application stops responding with Debug Diagnostic Tool v1.2.
We still use windbg to analyze the dump files. In the dump files most of the threads were executing in timer callback and waiting for the response from a HTTP request:
0:027>.loadby sos mscorwks
OS Thread Id: 0x1744 (27)
Child-SP RetAddr Call Site
000000000e05f890 0000064274a88157 System.Threading.WaitHandle.WaitOne(Int64, Boolean)
000000000e05f8d0 0000064274a92081 System.Net.LazyAsyncResult.WaitForCompletion(Boolean)
000000000e05f930 0000064274a91974 System.Net.Connection.SubmitRequest(System.Net.HttpWebRequest)
000000000e05f9c0 0000064274a90f38 System.Net.ServicePoint.SubmitRequest(System.Net.HttpWebRequest, System.String)
000000000e05fa30 0000064274a94a71 System.Net.HttpWebRequest.SubmitRequest(System.Net.ServicePoint)
000000000e05fa90 00000642802394c3 System.Net.HttpWebRequest.GetResponse()
000000000e05fb20 0000064275004fab ASP.global_asax.KeepAlive(System.Object, System.Timers.ElapsedEventArgs)
000000000e05fb80 00000642782f174b System.Timers.Timer.MyTimerCall
000000000e05fbf0 000006427834d966 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
000000000e05fc40 000006427f600dd2 System.Threading._TimerCallback.PerformTimerCallback(System.Object)
At first we may think:
1. The remote web server where the HTTP request was sent to had some performance issue.
2. Increase the maxconnection parameter according to KB821268.
However we found there were a lot of timer objects if we ran the command !threadpool.
CPU utilization 8%
Worker Thread: Total: 200 Running: 200 Idle: 0 MaxLimit: 200 MinLimit: 2
Number of Timers: 3891
All worker threads were used up. That seemed to be abnormal.
According to the call stack, the timer event handler is KeepAlive method in global.asax. We reviewed the code of global.asax and found the timer was created in HttpApplication.Init:
public override void Init()
global_asax._timer = new Timer((double) (60000 * 10));
global_asax._timer.Elapsed += new ElapsedEventHandler(this.KeepAlive);
Obviously the developer thought he would create a timer once and let the timer call KeepAlive every 10 minutes.
Unfortunately HttpApplication.Init will be called immediately when each HttpApplication instance is created.Instances of this HttpApplication class are created in the ASP.NET infrastructure, one instance is used to process many requests in its lifetime but it can process only one at a time. Thus member variables can be used to store per-request data.
It is *not* called once during the application’s lifetime. If we want to initialize something once, we need to use Application_OnStart instead of HttpApplication.Init.
The cause of WF 4.0 workflow service issue is the same. The initialization code for WF Instance Store and InstanceHandle were put in HttpApplication.Init method which was be called multiple times. The initialization code should only be executed once.
If you want to initialize something once during an application's lifetime, use Application_OnStart.
Xin Jin from APGC DSI Team