Thoughts on Application Pool Recycling and Application Availability

re: Thoughts on Application Pool Recycling and Application Availability

Brian Weis — Fri, 27 Jan 2006 05:14:19 GMT

This is the reason why i will give you an example.

EX: We have a company that has 10 users connect to a web-based application. And when we look in task manager at the W3WP.exe it shows a memory usage of around 160 MB. We had all of the users log off all within 1/2 an hr of each other but the
memory usage does not drop. Is this memory really used up or is just waiting for another app that needs it.

The memory does not clear up within task manager until we run a recycle.

This is why we want to run recycling every few hours.

re: Thoughts on Application Pool Recycling and Application Availability

David Wang — Fri, 27 Jan 2006 05:24:28 GMT

Brian - Your intuition is on target; just missing some details. Let me explain...

Server applications, contrary to client applications, expect to run for a long period of time. Thus, a lot of server-side application employ caching to improve performance over time, at the expense of memory, for heavily used traffic patterns which may span across users.

For example, response to /default.aspx may be cached in memory such that after 1 user accesses it and pre-compile/caches it, the other 9 users immediately retrieves the URL without much processing. This is a huge performance win at the expense of spending memory to cache it.

In other words, the fact that after 10 users connect to an application, log off, and after half an hour the memory still remains used is insignificant. You have to accept that this may be the expected "working set" of your application.

What you should be concerned about is this -- suppose 10 users repeatedly log on/off and use the application over a period of time doing the same sort of thing and the memory used seems to increase without bound -- that is a sign of memory leak and must be addressed.

In other words, until you determine *through experimentation* the "working set" memory of that application, you cannot distinguish between "expected memory caching" (i.e. "working set") and "aggressive memory caching" (i.e. "memory leak"). And if you cannot distinguish between the two, arbitrary numbers like 160MB or invoking periodic recycling simply does not make sense.

It may be that for 10 users, your application's working set is 200MB after 1 hour and stays steady thereafter -- in which case you are prematurely worrying about nothing by looking at things in 30 minutes.

Sorry, there is no magic bullet nor calculation to automatically determine the proper metrics or need for recycling - it is all specific to the application and requires human experience to decide.

//David

re: Thoughts on Application Pool Recycling and Application Availability

jason lee — Wed, 08 Feb 2006 07:04:54 GMT

Hi David,

We have built a Client Server application that uses webservices to
handle exchanges between the Client and the Server. (Think web-based application, but with a Win32 client instead of a browser.) The system is stateless, so the above discussion of lost state does not apply to us. However, we have been puzzled to find that when the worker process recycles, that often causes our client to freeze up, sometimes temporarily (1 to 5 minutes), and sometimes fatally. It seems that either the old process is not completing its request queue before termination, or the new process is not picking up the new requests.

To investigate this, we used a free utility called Process Explorer to
access the threads of the worker process. We noticed that, in those cases where the clients were unaffected by the recycling, the new worker process immediately listed a thread to the webservice DLL; in contrast, in cases where the client hung temporarily, the new worker process failed to list a thread to the webservice DLL at first, and after a few minutes, the expected thread would appear and the clients would unfreeze.
We also found that if we killed the worker process's thread to the
webservice DLL, sometimes a replacement thread would immediately appear and the client would continue undisturbed, but sometimes the thread would not reappear and the client would freeze fatally.

Can you tell us why recycling might cause the symptoms we're observing?

Thank you very much

re: Thoughts on Application Pool Recycling and Application Availability

David.Wang — Wed, 08 Feb 2006 14:31:24 GMT

jason - I think you need to look through the code in your custom client and web service to figure out what you are observing. What you are describing do not look like IIS issues but probably result from miscommunication between your custom application's expectations and IIS configuration. This is because the problem is always with how the user application deals with recycling - you can always configure IIS6 to not recycle.

For example, you say "... the old process is not completing its request queue before termination" - which is perfectly possible and by-design on a recycle. If a worker process had a long running user application and a recycle was initiated, if the application within the worker process failed to shutdown/finish within the "Recycle Wait Time Period" of 90 seconds by default (for example, a lengthy synchronous network call to SQL that took >90 seconds), IIS will go ahead and kill the worker process *without* finishing that request. Recycling makes a best effort to process the request, but if you have bad requests, IIS will kill them to be defensive. I have no idea whether your Custom Client can deal with "network data loss" in such a case, nor whether your Custom webservice allows such long requests to happen... but they can explain your temporary hangs.

You also say "... the new process is not picking up the new requests" - which does not make sense because following a recycle, any outstanding requests to the application pool are queued and go to the new worker process and your web service decides whether to work on it or not. It is simply not possible that the new worker process is *not* picking up the requests - if that was real, lots of people would have complained about missing requests by now. So, I suspect that either your Custom Client is not making those new requests until later, or your Custom webservice is not processing them for some reason.

Also, I ignore your observations regarding killing the worker process's thread using Process Explorer because that is hypothetical and not real coded behavior.

Finally, we may be looking at a startup/shutdown bug within the user application code. This happens far more frequently than people admit (because people always think their code is perfect), and it happens frequently when people run code on IIS6 which may have haphazardly worked on IIS5. This is because recycling is basically like shutdown/restart of IIS5 (except no client connection loss), and that event rarely happens prior to IIS6. But on IIS6 with recycling enabled, it can happen all the time, and startup/shutdown bugs in user code quickly show up under pressure.

Absent more concrete information like source code, it sounds like if a work-item was lost over the recycle due to it taking too much time, the custom Client has problems dealing with it. It may make retry-requests after 5 minutes and "recover", or maybe it never recovers. But in either case, the problem is not with the recycling but how the user application deals with the recycling, and the reasons there are quite arbitrary and endless.

//David

re: Thoughts on Application Pool Recycling and Application Availability

Good — Thu, 16 Mar 2006 13:14:54 GMT

Really nice thoughts!

Do you have any ideas about intermittent performance problems? Computing resource are fine, and when slowness is found, recycling will "solve " the problem.

Someone suggested "perhasp app pool is running out of threads", where can I find this setting or measure it then?

Cheers

re: Thoughts on Application Pool Recycling and Application Availability

SP — Mon, 10 Apr 2006 09:27:12 GMT

Hi,
We are facing a wierd problem with the worker process recycling. The web application dlls are deployed on a NAS. When the NAS connection is lost, the server gives the appropriate errors. The worker process gets recylced. The strange thing is, if we try to reconnect or refresh the web page, a new request is given to the old worker process which is getting recycled. We could observe it when we debugged the application. This causes the further requests to fails since the application does not get initialized properly. We could see the application_start event for the old worker process, which should not be the case. And when this happens all the subsequent requests fail. What might be the cause? Is this a problem with the application or the IIS giving requests to wrong process? We have tried to put in a kill call for the current process after a custom terminate routine. This seems to solve the problem. But is it a valid solution?

re: Thoughts on Application Pool Recycling and Application Availability

David.Wang — Mon, 10 Apr 2006 11:33:59 GMT

SP - How does losing the NAS connection trigger the recycling of the worker process? That does not sound like a built-in IIS6 behavior.

In particular, it sounds like your triggering of the "recycling" is not known to IIS nor HTTP.SYS, so further requests still get routed to the old worker process.

When IIS triggers the recycling of a worker process, an atomic "switch" of the Application Pool Queue happens such that new requests get routed to new worker processes while the old worker process finishes up on the existing requests. This prevents what you claim from happening.

Right now, it sounds like you are running custom recycling-related code on the server and you are observing bugs in it, not IIS6.

//David

re: Thoughts on Application Pool Recycling and Application Availability

SoP — Mon, 10 Apr 2006 12:11:10 GMT

hmm. I have a doubt with the process recycling then. I think my understanding about the process recycling is wrong. In our case when the connection with the NAS is lost, we get an Application_End event at global.asax.cs. Once the connection is restored an Application_start is received to the same worker process. Is this behaior correct? My understanding was when we receive Application_End event, worker process (w3wp.exe) will be killed by IIS and a new w3wp.exe will be spawned at the time of next request. Is my understanding correct? I am a novice with the IIS process recycling concepts and hence not able to understand the behavior that I am seeing. Any help will be much appreciated.
Thanks,
SP

re: Thoughts on Application Pool Recycling and Application Availability

David.Wang — Mon, 10 Apr 2006 12:44:14 GMT

SoP - What you describe sounds like Application Domain Unloading, which is an ASP.Net specific concept that has no correlation to IIS worker process recycling.

IIS process recycling concepts are pretty simple - worker process starts up only when servicing requests, and you configure a whole bunch of metrics on when the worker process goes away - such as by time, by # requests, by CPU, on-demand, etc. These metrics are generic and know NOTHING about Application Framework specific concepts like Application_End/Application_Start, etc.

What is happening in your case is this - When ASP.Net detects that an assembly is missing or "refreshed", it has to restart the AppDomain to reload/init that assembly - so it unloads the Application Domain (thus you get an Application_End event), and on next new request to an unloaded AppDomain, you get an Application_Start event. This is pretty much by-design behavior, and your application should work well with it. If it doesn't, it is usually a problem in your application.

In other words, Application_End event (an ASP.Net concept) has no correlation to worker process recycling (an IIS concept). Can you point me to the documentation that gave you this understanding?

//David

re: Thoughts on Application Pool Recycling and Application Availability

SoP — Mon, 10 Apr 2006 12:57:01 GMT

Thanks a lot for this information. Not any documentation in particular. I think I got confused going through the worker process recycling and application domain concepts. Thanks a again for the information.
SoP

re: Thoughts on Application Pool Recycling and Application Availability

Liam — Thu, 20 Apr 2006 04:55:11 GMT

We've just had heaps of trouble with what we thought was either the worker process being recycled or application_end being called and it now looks like neither were the cause of our cache issues. for whatever reason, the cache just kept losing entries randomly - although the higher the load, the more frequent the losses. We've completely given up trying to figure out why and stopped using the cache altogether in favour of a statis class initialised in global asax that exposes a hashtable...it never fails.

re: Thoughts on Application Pool Recycling and Application Availability

David.Wang — Thu, 20 Apr 2006 11:22:32 GMT

Liam - I do not know what cache you are referring to, but in general, caches all have insertion and eviction policies, so can you enumerate the exact insertion and eviction policies of the cache and what does the cache do when it is "full".

In other words, with high load, maybe you are caching so many cold objects that the hot ones get evicted somehow -- and this appears as "losing entries randomly" to you. What is obvious to you to cache is NOT so obvious to the cache - depends on its insertion/eviction policies. Thus, you are basically wasting time trying to figure out WHY when you cannot enumerate the insertion/eviction policy of the cache.

So, it is not that the static class is better or never fails - it is that you happen to know its insertion/eviction policy.

//David

Loosing ASP.NET Session State in IIS6

Dennis J. Bottjer | MVP ASP.NET — Wed, 09 May 2007 17:52:08 GMT

Last year we deployed a new underwriting system. Since the Fall of 2006 we have been steadily releasing

re: Thoughts on Application Pool Recycling and Application Availability

Topcat — Thu, 21 Jun 2007 03:25:56 GMT

David, assuming you are receiving a lot of concurrency or that you have a memory leak what is the highest you can safely set your application pool w3wp.exe process to recycle and still prevent out of memory errors? I know this sounds like a dangerous question, but assuming I can't fix the poor code, I want to know how far I can push the limit. Let's assume I have 32GB of physical memory. Is the limit around 900MB physical or can I go higher?

re: Thoughts on Application Pool Recycling and Application Availability

David.Wang — Sun, 24 Jun 2007 10:19:15 GMT

Topcat -

You can just set any arbitrary private-bytes-consumed limit, determined through experimentation. 900MB is just as arbitrary as 2000MB.

Otherwise, you will have to figure out:

1. What memory pool is the leak in and what is the limit of that memory pool

2. What is the rate of leakage (either in terms of time or #-requests)

And you calculate:

---- = units to hit limit

And you make sure to either recycle by time or by #-requests at about 80% of the limit.

//David

re: Thoughts on Application Pool Recycling and Application Availability

Jim — Tue, 26 Jun 2007 14:22:41 GMT

Great posts and helping me to understand a current problem we have. I'm somewhat of a newbie to this btw.

One of our Sharepoint sites (just a webapp in IIS) stops functioning after an overnight app pool recycle ie. cannot find resource or locate page. We can resolve this by renaming web.config or moving/replacing a dll file, hitting site getting an error, renaming web.config and on next hit all seems well. I'm guessing the rename of web.config is causing something in IIS to recompile the webapp hence why it works.

What I'm not sure about is how to properly diagnose what may be causing this problem? Any help gratefully received.

re: Thoughts on Application Pool Recycling and Application Availability

David.Wang — Mon, 02 Jul 2007 03:14:54 GMT

Jim - what Application Pool are you recycling, and do the applications in that Application Pool support recycling.

Changing web.config or changing a DLL file in /bin causes ASP.Net to recompile that web application on next request.

My guess is that your problem happens because of the overnight Application Pool recycling of an application that does not support it, and that causing ASP.Net to recompile the web application resets whatever internal linkages that are broken by the unexpected Application Pool recycling.

FYI: This really has very little to do with IIS and everything to do with diagnosing the behaviors of an arbitrary application, such as Sharepoint. If you truly want to get to the bottom of things, then you should hire a professional to help - like Microsoft PSS or appropriately skilled consultant.

//David

re: Thoughts on Application Pool Recycling and Application Availability

Jim — Tue, 03 Jul 2007 16:56:41 GMT

Thanks David - I'll take a look at the application support for recycling app pools that you mention. The particular site which is having problems includes some workflow/infopath functionality. We have several other MOSS sites on the same server which are fine after the overnight app pool recycle and don't use this functionality.

We haven't done any separation of apps across different app pools yet but this seems a sensible next test to try.

I'm sure you are right about Sharepoint causing the problem. I'll take your advice and engage a pro to help.

many thanks

Jim

re: Thoughts on Application Pool Recycling and Application Availability

Brett — Thu, 12 Jul 2007 20:40:02 GMT

David - Thank you for all of the information. I have a couple of questions. I'm kind of new at IIS configuration so I hope they make sense.

1) Is there a standard app pool recylce schedule that you use as part of a preventative maintenance plan? Is it necessary to recycle the pools every 8, 12, or 24 hours if you do not see issues? Can/should you base a recycle only on memory usage?

2) Do you have a recommendation for setting memory limits for application pools? If you assume there are no known memory leak issues aside from normal wear and tear what percentage of available memory should you use? The environment I'm asking about is a SharePoint 2007 front end sever with nothing else running on the box.

Any insight you can offer would be appreciated.

Brett

re: Thoughts on Application Pool Recycling and Application Availability

Paula DiTallo — Mon, 24 Sep 2007 20:37:12 GMT

We're experiencing a symptom in one of our ASP .NET applications that could use some direction before analysis. The symptom is a user will start a session with the application and begin working, shortly, the performance degrades to such a slow pace that work in the session becomes impossible-or the user leaves a working session to take a break, etc. -- then tries to create a new session, but the session will not connect. We've been restarting the application pool to address both conditons, however we know this isn't the solution. From an administrative perspective, all I'm seeing in the logs are idle timouts, etc.

Any advice on what else to look for to help resolve this issue over here would be very helpful.

re: Thoughts on Application Pool Recycling and Application Availability

Okisan — Fri, 06 Jun 2008 03:34:27 GMT

Thanks for very useful article.

My application pool stop sometimes. I wish to restart the application pool automatically if it stop. How can I do?

Thanks.

re: Thoughts on Application Pool Recycling and Application Availability

Hi guys — Fri, 01 Aug 2008 00:06:58 GMT

Just a comment about the freeze in the apps,

if I understood, your application is connected to webservices.

If you restart the appPool, you propably will require the web app to recompile on the first access to it, that is how .net framework works on websites, and a webservice is a kind of a website.

I think it is normal... You application is just waiting the ready state of your webservices.

Regards,

André Luiz Sobreiro

MCPD

re: Thoughts on Application Pool Recycling and Application Availability

David.Wang — Sat, 02 Aug 2008 23:42:20 GMT

Okisan - if the application pool stops, it means that some catestrophic error happened in the application running in the application pool. You need to follow the instructions in my "AppPool Crash Diagnosis" link from all my blog entries to diagnose the cause.

Until you determine and resolve the crash in your application, your application pool will continue to periodically stop.

//David

re: Thoughts on Application Pool Recycling and Application Availability

Stormy — Sat, 06 Sep 2008 00:20:52 GMT

IIS6 and 7 I believe come with a default recycle value set, and you should always consider increasing or extending those values based on your available memory, expected load, number of processes and processors on your server, etc. Microsoft has an algorythm for that increase, I believe, to max that out.

If your app is poorly written...not closing objects and releasing threads, using too much memory per session, heavy user load event design, etc, you can crash the AppPool more than expected. Your only work around is to get a faster or bigger server, and bump up the app pool defaults, or look at your processes and see if threads or objects are not getting collected and disposed in a timely fashion.

re: Thoughts on Application Pool Recycling and Application Availability

Richard — Mon, 15 Sep 2008 05:47:08 GMT

Hi everybody

I have a little question... In our environment we have a Web services based on XML files, when we edit an xml file for a specific web site, we apply an iisreset command to take the new configuration.

What happens if we apply an application pool recycle on the web site? Does the web service take the new configuration?

Thanks

re: Thoughts on Application Pool Recycling and Application Availability

David.Wang — Thu, 18 Sep 2008 05:42:37 GMT

Richard - the answer completely depends on how your Web Service reads configuration.

It could require no action -- as example, how changes in web.config are picked up by ASP.Net. Or it could require actions like iisreset or application pool recycl. It all depends on the Web Service's code.

//David

re: Thoughts on Application Pool Recycling and Application Availability

Richard — Sat, 20 Sep 2008 02:20:40 GMT

Ok. Thank you David. I'll check that.

re: Thoughts on Application Pool Recycling and Application Availability

terri hawkins — Tue, 03 Feb 2009 22:38:52 GMT

I am new to IIS management, and could use some help please.

I have a web server which is running several ASP applications which have memory leaks. I am trying to set up the application pool recycling to free the memory without having our site go down. We established a maximum of 450K of memory before we have a problem, so I set the application pool to recycle at that point.

Generally, this works, I can look at the event logs and see where the application pool recycled without incident (it occurs roughly 3 times per day), however, sometimes I end up having the site go down (our customers receive out of memory errors) and have to sign in and recycle it myself.

Here is my question (although any thoughts on the above are also welcome), if I reduce the amount of memory it can use, do you think that would solve the problem? Or would it just occur more often? Is this normal behavior for a recycle? Is it just that there are so many processes running at that time that it takes too long to startup and shutdown (the out of memory errors always seem to occur during the day).

I was also wondering if I could leverage both the virtual and physical memory limits… is there a rule of thumb about that, or would it not help?

Any assistance is greatly appreciated.

terri

re: Thoughts on Application Pool Recycling and Application Availability

jon — Thu, 26 Feb 2009 23:13:03 GMT

Dave,

You come off as a complete dick, chill out man-

John

re: Thoughts on Application Pool Recycling and Application Availability

jon — Thu, 26 Feb 2009 23:14:11 GMT

Dave,

Sorry, did'nt know this comment section wasn't moderated, I retract my earilier staement.

re: Thoughts on Application Pool Recycling and Application Availability

David.Wang — Mon, 02 Mar 2009 16:12:23 GMT

terri hawkins - the correct solution is to fix the ASP applications to not have memory leaks. What you need to do is to force these applications to be properly stress and scale tested before being put in production, or else your life will be an endless firedrill to find a bandaid to someone else's bugs in production. If you cannot affect the development process to improve this mess, your life will be miserable and there is no way I can help.

Realistically, your are really not doing IIS management. You are basically baby-sitting someone else's poor application on the server you are responsible for. And you can either keep taking that abuse, or strike back at the source of the abuse and improve application quality to truly improve your life so that you can actually do IIS management.

Configuration Application Pool Recycling is just a temporary bandaid to attempt to work around a bug. It may/not work.

Figuring out the Application Pool recycling metric is more an art than science -- it completely depends on the behavior of the bug being worked around. The reason I say that it is an art and not science is because if you knew the nature of the leak, you would probably be able to fix the leak. It is because you don't know the source/type of leak that you have to "guess" -- hence art and not science.

It is impossible for me to give concrete answers to your questions because they depend on the behavior of your ASP applications. You need to closely monitor the server to understand how the error comes about... but if you knew that, you probably understand the nature of the leak and would not be guessing.

For example, if you reduce the limit for memory based recycling, they should occur more frequently, but that is no defense against an application suddenly spiraling out of control using memory. Also, what makes you say that "there are so many processes at that time that it takes too long to do X/Y"? Where's the concrete proof? How is that related to out-of-memory?

Hopefully, you see that everything is a possibility. It all depends on the application and the nature of the bug itself. Without diagnosing the bug, it is nearly impossible to provide a rule-of-thumb to work around the bug. If there was a rule-of-thumb, then it means that all bugs work the same way, and that is simply untrue.

//David

re: Thoughts on Application Pool Recycling and Application Availability

Dinu John — Tue, 24 Mar 2009 11:01:50 GMT

I have a web application that deployed in windows 200 server IIS6.0, running in its own application pool. We are using some page level caching also, while accessing the site with more than 50 users, abrupt session expiry is happening and we could understand that the application pool recycling is happening and the cache value is also cleared. The memory usage at that time is 40%. Could you please give your thoughts on what may be the problem?

re: Thoughts on Application Pool Recycling and Application Availability

Vishal Deokar — Wed, 03 Jun 2009 10:12:21 GMT

Hi,

I have the following doubt.

Is there any way that the worker process can determine at what time it is going to be recycled or can it calculate how many minutes are there before the app pool recycle the worker process?

Any method that gets called or any indication that the worker process can listen to so that it come to know before hand that recylce process is to begin shortly.

Regards

Vishal

re: Thoughts on Application Pool Recycling and Application Availability

Kirtesh Jain — Fri, 19 Jun 2009 17:27:34 GMT

Hi David,

I am running an Insurance web application on .NET 2.0 this application has 1000 s of user every day and huge volume.

In the application we show all the data for insurance policies on a grid (paged properly)

Now the problem is .. once in a weeks time we receive errors navigating the page containing grid :

Errors e.g.

1.) System. System.Web.HttpException: Multiple controls with the same ID 'PnlState' were found. FindControl requires that controls have unique IDs. at System.Web.UI.Control.FillNamedControlsTable(Control namingContainer, ControlCollection controls) at System.Web.UI.Control.FillNamedControlsTable(Control namingContainer, ControlCollection controls) at System.Web.UI.Control.FillNamedControlsTable(Control namingContainer, ControlCollection controls) at

2.) System. System.Web.HttpException: Multiple controls with the same ID 'PnlPolicyNumber' were found. FindControl requires that controls have unique IDs. at System.Web.UI.Control.FillNamedControlsTable(Cont

Now when we recyle the application pool the site is back available and user can browse through grid.

Please suggest some solution so that where we can fix in application or some setting in IIS so that we do not get this errors.

re: Thoughts on Application Pool Recycling and Application Availability

Cristian — Wed, 24 Jun 2009 16:56:12 GMT

Hi Kirtesh,

could be a cache problem

Take a look at http://support.microsoft.com/kb/834608

Cheers.

re: Thoughts on Application Pool Recycling and Application Availability

Kirtesh — Wed, 24 Jun 2009 21:02:18 GMT

Hi Cristian,

I checked the url posted by you but it seems to be applicable for .NET framework 1.1 . I am using Framework 2.0 . Can I still use the hotfix ?

re: Thoughts on Application Pool Recycling and Application Availability

Stuart — Wed, 19 Aug 2009 16:45:47 GMT

David,

A ton of very useful information on this post/thread (you are the top hit for "iis application pool recycling"). Thank you for taking the time to maintain the thread (I see the posts go back over three years).

However, after digging through it all, I still have a question as I have been refactoring a moderate volume (~150k pages/day) web site to address pervasive memory issues. The recycle issue is resolved, but we have a new issue now.

Our initial problem was multiple previous developers used Session rather liberally for everything, so our process would roll over about every 10 minutes under load because of the Session growth. This also caused a loss of state on every recycle which created a different set of problems.

Over a period of about four months, we slowly but surely found and addressed Session offenders. In most cases, we were able to move state to move the data to the HTTP Cache. The caching design was essentially two ideas; (1) for data objects, there is now only 1 copy used by all users rather than N copies for N users and (2) collections are not populated with copies of data objects but instead contain a set of IDs that we can manually dereference to data objects in cache. This brought our memory usage well within limits and the CPU overhead seems to be negligible.

To address the potential loss of what state was left in session, we moved to ASP State server. As a side note, our original sessions were so large, they quickly generated SQL IO errors in production I assume because it couldn't push enough data fast enough for all the users (we found 50Mb+ sessions in the DB from that initial attempt). Anyway, we keep viewstate in Session now, along with a handful of other things, and its working great so far.

We recently pushed the last of our caching optimizations and we ran for most of the day at 15% CPU, about 4k objects totaling 150Mb in HTTP Cache (estimated), and a process memory that stabilized at 350Mb in task manager. I considered this our new baseline.

I would note that we have relatively aggressive HTTP Cache timeouts, so that 4k entries roll over relatively frequently. In some cases our process will spike up to 600Mb according to the Peak Memory Usage column in task manager. This seems kind of high for our caching, but I cant rule that out (i.e. I haven't caught the process with a high value like this and checked the cache).

With that as a back drop, on to the new problem which we really started noticing this week with the last of the code pushes.

We run very consistent all day long with stable CPU, load, and memory, but pretty much anytime after say, 3 hours of load, the CPU can snap up to about 80% and stay there (its immediate, not gradual). When this happens, there is no noticeable change in memory or load and the site usually becomes very slow (we get email alerts regarding CPU and timeouts). I have seen it recover from this condition before without recycling, if left alone long enough, but sometimes it eventually wanders up to 100% where I usually stop waiting to see what happens and recycle. Resetting the application pool manually snaps us right back down to our new baseline (15% CPU/350Mb process) as soon as the recycle completes.

It seems we are accumulating something over time that is eventually showing in CPU utilization. On this note, I belabored the caching stuff because I felt we might be rolling so much through the cache that its fragmented badly enough to become expensive after some period of time, but I really have no basis for that notion.

I suppose with the StateServer we could schedule recycles based on page hits, but I was really hoping to see the system run all day with the same app pool...

Any thoughts on our situation?

Thanks in advance for your time,

Stuart

re: Thoughts on Application Pool Recycling and Application Availability

Athi S — Wed, 16 Dec 2009 17:22:20 GMT

Hi David,

We have a website, which has global.asax file, which contains Seesion_End as well as Application_End events. These are firing in normal scenarios. But, if IISreset happens, it doesnt... Is it the behavior or anything wrong in our side? If its the behaviour, is there any other alternative to know when IIS is reset? Ultimately, we want to log some data (in the backend) when iisreset happens.

Thanx in advance.

Athi