ASP.NET Case Study: Lost session variables and appdomain recycles

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Snorrk — Wed, 02 Aug 2006 15:52:09 GMT

Great post!

Soci blog » Blog Archive » Mikor ??s mi??rt indul ??jra egy asp.net app?

Soci blog » Blog Archive » Mikor ??s mi??rt indul ??jra egy asp.net app? — Wed, 02 Aug 2006 16:13:59 GMT

PingBack from http://soci.hu/blog/index.php/2006/08/02/mikor-es-miert-indul-ujra-egy-aspnet-app/

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Jason — Wed, 02 Aug 2006 17:03:16 GMT

Great post, Tess. Thanks so much for sharing this info!

Why do app domain recycle (when you don't want them to)

ÎÜñ|‹ø\/\/ñ [ÐëÞrëçã†ëð]'s Blog — Wed, 02 Aug 2006 18:54:33 GMT

Well, if you have ever wondered, Tess has a great post listing out the various reasons why. Well worth

Link Listing - August 2, 2006

Christopher Steen — Thu, 03 Aug 2006 07:35:31 GMT

My
ASP.NET 2.0 Tips, Tricks, Recipes and Gotchas "Highlights Page" [Via:
ScottGu ]
Sending...

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Ran Davidovitz — Thu, 03 Aug 2006 11:00:02 GMT

Great post indeed, Thank you.

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Pedro Rui Silva — Thu, 03 Aug 2006 12:24:29 GMT

Great post it helped my a lot, we were having many timeout's and we didn't have a clue on why they were happening. Thank's.

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Yaroslav Vysotsky — Thu, 03 Aug 2006 20:20:48 GMT

Great! Very useful post. Hope that will help us to solve a problem. Thanks!

Interesting Finds: August 2

Jason Haley — Fri, 04 Aug 2006 06:26:01 GMT

August 9th ASP.NET Link-Listing

ScottGu's Blog — Wed, 09 Aug 2006 11:31:49 GMT

Below are some nice articles and links I&rsquo;ve found on the web over the last week that I enjoyed,

re: ASP.NET Case Study: Lost session variables and appdomain recycles

JasonBSteele — Wed, 09 Aug 2006 14:46:19 GMT

Tess,

This is a really useful post and has given me several pointers.

However, I am having a problem where my users are loosing their session and the folowing messag eis appearing in th event log:
Event code: 4005
Event message: Forms authentication failed for the request. Reason: The ticket supplied has expired.

Doing a bit of googling reveals that a few other people are having this problem but nobody has really come up with a good explanation why. Perhaps this could be the subject of a future blog? /emote Fails miserably at charming smile

Also of possible relevance is that my web application is organised with each of my ciustomers having their own virtual folder which is pointing at a particular version of the product. So I have folders v1.1, v1.2 etc with virtual folders CompA..CompS pointing to v1.1 and CompT..CompZ pointing to v1.2. When I make a new release I create a new release folder and then repoint each company virtual folder as and when they are ready for the new release.

What are the pros and cons for this? Am I wasting loads of memory by having an AppDomain for each company when they could all be using a single AppDomain?

Thanks for your time,
Jason

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Rodel E. Dagumampan — Thu, 10 Aug 2006 06:57:39 GMT

A very nice post. keep them coming.

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Ramachandran — Fri, 11 Aug 2006 12:46:09 GMT

I want to know the internals of how the ASP.NET request is processed from the begining ( the user typing the url) to the end ( Appdomain unload). Is there any material or links. This post is really useful and if there are many posts like this then that will be great :)

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Ken Egozi — Sat, 12 Aug 2006 20:26:37 GMT

I'm having a Session clearing problem with a project I'm running.
I'll try yo investigate on you leads (web.config change by some service) but I think that it's due to another thing. The website allows a lot of file uploads, through simple http requests, so my guess is that it is caused by inflation in the memory used by the AppDomain. Do you think that this can cause AppDomain restarts?

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Tess — Mon, 14 Aug 2006 14:24:57 GMT

If it is due to memory usage, it would probably be caused by an out of memory exception, but then you wouldnt only see an appdomain restart, you would see the whole process restart (likely with a stopped unexpectedly eventlog entry). Could it be that as part of the file upload you copy the results into the bin directory or similar? I would try to log the restart reason and go from there.

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Tess — Mon, 14 Aug 2006 14:35:03 GMT

Ramachandran,

I am not sure if this is what you are looking for, but here is an article on the page lifecycle, from when the request comes in to the ASP.NET layer to the unload of the page object

http://msdn2.microsoft.com/en-us/library/ms178472.aspx

It doesn't cover what happens before ASP.NET gets a hold of it, and it doesnt cover the appdomain unload since that is not really part of a request, but a part of the whole application, so again, I am not sure if it is what you are looking for, but it is pretty good...

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Tess — Mon, 14 Aug 2006 14:51:16 GMT

JasonBSteele,

I hadn't seen the issue you describe before but a quick internal search revealed that you might be running into this issue which has to do with URL rewrites:

http://support.microsoft.com/kb/911300

The NullReferenceException mentioned in there might be what causes your tickets to seem expired.

If it was my own app, and I could reproduce, i would probably log exceptions to see what I could find.

I am not sure if you have any alternative ways of setting it up (without getting too deep into your particular architecture) but each appdomain has a certain overhead, and depending on how things are linked it might be quite large if you keep loading copies of the same dlls.

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Troy Hall — Wed, 23 Aug 2006 20:53:48 GMT

In my case, I was using the Enterprise Library Logging Block to write to the web application root directory (locally, on my dev machine only!). Once I reconfigured to stop this logging, and reset the app domain manually (resaving web.config) the session reset went away. THANKS!!! This page gave me the clues to move on.

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Tess — Thu, 24 Aug 2006 09:23:05 GMT

Awesome, glad it helped:)

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Rich S. — Thu, 14 Sep 2006 20:57:17 GMT

I enjoyed the article, and was hopeful when I started tracking application restarts. I found that my application isn't restarting when the session is unexpectedly abandoned.

Can you recommend a course of debugging when it is only the session that seems to be intermittently and [seemingly] inconsistently restarting?

TIA,

Rich

re: ASP.NET Case Study: Lost session variables and appdomain recycles

JasonBSteele — Tue, 19 Sep 2006 18:11:48 GMT

Tess,

I wouldn't say that my application has the characteristics suggested in http://support.microsoft.com/kb/911300
It doesn't use URL re-writing, it simply uses several virtual folders (which often point to thesame physical folders).

However, further investigation has revealed that the ticket expired event does not appear to coincide with the session loss.

I followed your advice and swiched on the health monitoring logging and found that the session loss was caused by a "Application is shutting down. Reason: Configuration changed"

However, nothing is touching the files you mentioned (no indexing, virus checking backups etc.). I have used FileMon and it also indicates that nothing is happening to the files.

I know of one other person is having the same problem, see http://www.velocityreviews.com/forums/t294620-application-restarting-configuration-has-changed.html

who has been trying to resolve this with MS support for quite some time now.

Have you heard of this issue, or can you suggest a way of investigating it further?

Thanks,
JasonBSteele

re: ASP.NET Case Study: Lost session variables and appdomain recycles

JDelmerico — Tue, 03 Oct 2006 15:55:59 GMT

JasonBSteele & Tess, I can confirm the same behavior here. I am seeing app domain restarts no apparent cause under the label of "Application is shutting down. Reason: Configuration changed." according to the ASP.NET 2.0 health monitoring API. I also ran Filemon and confirmed there were no writes affecting of config file around the time of the restart. The only file activity affecting the web.config file is the aspnet_wp itself, and only reads.

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Sudeep — Thu, 12 Oct 2006 16:01:43 GMT

JDelmerico,

In ASP.NET 2.0, any change to the contents of the application folder can cause an appdomain restarts. Check out Toddca's site:

https://blogs.msdn.com/toddca/archive/2006/07/17/668412.aspx

Does filemon show any files from the application folder being *touched* ?

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Tony Bunce — Fri, 20 Oct 2006 22:42:30 GMT

How can I get a dump file when my AppDomain restarts? I am logging asp.net app restarts via the method described by ScottGu and am getting some weird results and want to dig deeper.

_shutDownMessage=Overwhelming Change Notification in C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Config

nothing is changing in that directory (watched it with filemon) and even turned off NTFS last access time to see if that would help and it didn't.

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Jethro — Wed, 25 Oct 2006 06:29:04 GMT

I can confirm the same behavior here. I am seeing app domain restarts no apparent cause under the label of "Application is shutting down. Reason: Configuration changed." according to the ASP.NET 2.0 health monitoring API. I also ran Filemon and confirmed there were no writes affecting of config file around the time of the restart. The only file activity affecting the web.config file is the aspnet_wp itself, and only reads.

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Tess — Wed, 25 Oct 2006 09:01:00 GMT

If you receive the Overwhelming Change Notifications and noone is touching the files I would recommend that you contact Microsoft Support. It might be a problem in the framework that you are running into where ASP.NET reports an error during filechangesmonitoring as "overwhelming change notifications", normally the way to troubleshoot is to set breakpoints on filechanges notification functions in the webengine to see what error occurred.

re: ASP.NET Case Study: Lost session variables and appdomain recycles

abbstl — Wed, 25 Oct 2006 20:59:17 GMT

JasonBSteele -

Did you ever figure out the cause of your issue?

re: ASP.NET Case Study: Lost session variables and appdomain recycles

JohnWilley — Fri, 10 Nov 2006 18:12:03 GMT

We're having frequent recycles with event message "Reason: Configuration Changed". Like several other folks, we've confirmed with filemon that nothing but w3wp.exe is touching the application folder.

Wondering if anyone found the problem??

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Payal — Thu, 01 Feb 2007 15:18:01 GMT

Great article..and helped me a lot..but i have one other issue related to session variable..My application is in asp.net 1.1..when i login in to the application i can see my details properly..but after some time in between i m able to see other person's details instead of mine..and this problem is not happening frequently.. It is happening some times..can you please tell me reason and the solution. I have stored user information in one session variable..depending on that i have shown user details.

This problem we are getting on live server only..not on other servers.

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Tess — Thu, 01 Feb 2007 15:27:39 GMT

Hi Payal,

Pretty much every time I have seen this it has been one of two things.

1. The thing that is being put in session scope is comming from a cached or static object of some sort without proper locking mechanisms so while you are reading, some other thread/user has already written new data to it.

2. A router or load balancer is used and the users come from the same ISP getting the same IP address so the router gets confused and returns the session cookie to the wrong user

The second scenario is a misconfiguration in a router, and is by far much less frequent than the first scenario.

re: ASP.NET Case Study: Lost session variables and appdomain recycles

payal — Thu, 01 Feb 2007 15:43:31 GMT

Hi Tess,

thanks a lot.. i think i m getting this problem because of second issue because i have already checked first option but session scope is not comming from any cached or static object.

and if that is the case, i should get the same issue on my other servers.

Previously for this same site..we are getting above issue i.e. lost of session variable due to appdomain recycles.. that time ur article helped me lot..as we found on live server there are two anti-viruses are running and causing the problem.

re: ASP.NET Case Study: Lost session variables and appdomain recycles

BorisL — Tue, 27 Mar 2007 20:58:14 GMT

Hi Tess,

We are getting the same problem as JasonBSteele:

Event code: 4005

Event message: Forms authentication failed for the request. Reason: The ticket supplied has expired.

I have a question about this: I have FormsAuthentication enabled, so shouldn't the ticket expiration (the cookie) be handled regularly by FormsAuthentication subsystem?

We have this problem only on IIS5, it can not be reproduced on IIS6 :-(

Any suggestions would be greatly appreciated...

Boris

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Jethro — Fri, 30 Mar 2007 09:52:49 GMT

Hi, there are several Apps in IIS 6.0 on Windows 2003, the config is:

</credentials>

</forms>

</authentication>

the others are :

authentication mode="Forms">

</authentication>

Now, only the first app log event every minute as:

Event code: 4005

Event message: Forms authentication failed for the request. Reason: The ticket supplied has expired.

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Tess — Fri, 30 Mar 2007 10:56:36 GMT

I think the most common reason for this error is that the appdomain restarted, and since you are posting these questsions in connection to a post about appdomain recycling does that mean that you do see in perfmon that the appdomain has recycled?

If it has, add logging to see why it recycled

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Boris — Fri, 06 Apr 2007 22:48:28 GMT

Hi Tess

I solved the "Forms authentication failed for the request. Reason: The ticket supplied has expired. " problem by providing permanent valudation and decription keys in machine.config. The problem was caused by generation of new random validation/decription keys when appdomain restarted...

Here the article about how to create keys...

http://support.microsoft.com/default.aspx?scid=kb;en-us;Q312906

Boris

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Glenn Slayden — Tue, 10 Apr 2007 09:30:57 GMT

Wow, after 8 hours of debugging the buck stopped here. You hit the nail on the head with this great article. In my case (ASP .NET 2.0, C#), a static generic List object was sometimes getting reinitialized across a System.Diagnostics.Process.Start() spawn of Perl.exe. As you point out, the AppDomain is being unloaded because Perl changes various files in a subdirectory.

I haven't tried this yet, but would a fix be to create a different application for those subdirectories (using IIS manager), thus excluding them from the main AppDomain and its notifications?

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Tess — Tue, 10 Apr 2007 10:12:19 GMT

Hi Glenn,

I'm happy that the post helped:) The optimal setup would be if the files that were updated weren't in the application path at all, but i suspect that they are supposed to be accessible from the web so in that case if you set the subdir up as an application, only that application would restart when the files are updated and a new request comes in to that application which would save the main app from restarting.

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Glenn Slayden — Wed, 11 Apr 2007 02:47:04 GMT

Tess, thanks for your reply.

Unfortunately, my idea does not work; creating an application for the subdirectory does not exclude it from the parent application's FCN (file change notification).

There are not many other options. It may be possible to create a directory junction (as discussed by Toddca) but from my reading it seems like they are somewhat delicate.

What does seem to work--and is not delicate enough--is setting <httpRuntime waitChangeNotification="2147483647" maxWaitChangeNotification="2147483647" /> in the site's web.config, but then we lose the whole auto-deployment feature, and it becomes difficult to re-JIT the site without a full-blown manual 'iisreset'. With this option, to force the AppDomain to reinitialize I tried having a button on the site which causes it to call HttpRuntime.UnloadAppDomain() or AppDomain.Unload(AppDomain.CurrentDomain) on itself (even from a QueueUserWorkItem), but had little success. I can't see a way to do it from the IIS Manager snap-in, either.

Glenn

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Glenn Slayden — Wed, 11 Apr 2007 06:35:35 GMT

Another thing to note is that the delays imposed by waitChangeNotification and maxWaitChangeNotification are in effect even for explicit calls to HttpRuntime.UnloadAppDomain(). In fact, the function blocks for the entire wait duration!

This is not what I would expect. The result is that when using a large delay value, the function is not usable and there's little chance of recycling your AppDomain.

I confirmed this result with varying delay wait values 'w': the the call to UnloadAppDomain() blocked for 'w' seconds, after which the AppDomain was cheerfully unloaded.

Glenn

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Colin Hale — Wed, 11 Apr 2007 14:17:32 GMT

Hi,

Back in Feb, there was a post from Payal. He was experiencing a problem whereby users were getting other users session variables.

I am having the same problem under .net1.1

Changing the software to 'cookieless' - to display the sessionID in the url, we can see that the sessionIDs are changing to someone else's.

In your response, you give 2 possible scenarios

I think I can rule the first one out, as no shared components are involved No reused or 'cached' components are being stored in the session, only a simple class structure defined within the asp.net project, instantiated as an object and stored in the session within an ascx.vb file.

As for scenario #2, what tool(s) would you recommend to determine the ip address of the incoming request?

Thanks,

Colin.

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Colin Hale — Fri, 13 Apr 2007 13:23:51 GMT

Just for closure:

We had to contact Microsoft. They gave 3 or 4 scenarios, but it turned out that one of our aspx pages had the directive <%@ OutputCache %>. This caused the page to be cached, hence the second user will get the first users page (and details taken from the first session).

What was weird (and a bit of a red fish) was the 2nd user getting the 1st users session ID.

After removing this directive - it all started to work correctly.

Colin.

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Glenn Slayden — Thu, 19 Apr 2007 02:09:01 GMT

And for closure on mine:

It looks like the directory junction approach, while bizarre, will work for my application and I will eventually have to use it.

For now though, I have been testing <httpRuntime waitChangeNotification="2147483647" maxWaitChangeNotification="2147483647" /> on the production site for a week and this works as well, with the caveat that this disables auto-deploy and you must do a manual iisreset to force updating or re-JIT.

Glenn

re: ASP.NET Case Study: Lost session variables and appdomain recycles

joe — Mon, 23 Apr 2007 01:18:43 GMT

Hi There,

I am having a similar problem, what other possible causes did MS give you.

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Colin Hale — Wed, 25 Apr 2007 13:00:22 GMT

MS response was as follows:

Everything we have checked so far results to be fine, so there is only one possibility left which can lead to this problem: caching.

We are probably sending to the second client a page (or data) being cached by the first client.

There are 3 way this could happen:

1) using OutputCache on those pages: please, open the page directive section of all the pages in relation with the problem and ensure you are not using OutputCache.

2) caching sensitive data using the Cache[] object. Please, make sure you aren’t

3) having a proxy/firewall between server and clients which caches pages: please make sure this does not happen. Could you try to reproduce the problem using 2 different boxes as client, both directly connected to the server (no proxy between)?

Hope that helps,

Colin.

re: ASP.NET Case Study: Lost session variables and appdomain recycles

mkraley — Sat, 28 Apr 2007 01:04:59 GMT

I too am having mysterious "Reason: Configuration Changed" application restarts. Filemon confirms that there are no writes taking place either in the application tree or in the Framework folder tree. The first mention of anything relevant in Filemon is a CLOSE of web.config, which I suspect is just the beginning of the restart, not the cause.

Has anyone made progress on deducing why this happens?

...Mike

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Colin Hale — Tue, 08 May 2007 12:23:34 GMT

Have you tried excluding the virt folder from any anti-virus s/w?

Lost session variables and appdomain recycles

Tue, 29 May 2007 19:03:36 GMT

In ASP.NET Case Study: Lost session variables and appdomain recycles beschreibt Tess was passiert wenn

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Minnie — Sat, 16 Jun 2007 04:49:30 GMT

I have a question. How can we capture session timeout in IIS/ASP.NET?

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Salman Banani — Tue, 26 Jun 2007 04:14:56 GMT

Great article, really helped. My antivirus was casuing my appdomain to recycle.

Thanks man.

ASP.NET Hang scenario: Kernel Debugging for Dummies

If broken it is, fix it you should — Mon, 27 Aug 2007 10:52:21 GMT

A local radio show has started giving out cookie points to their interviewees for personal development,

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Rob Kent — Fri, 05 Oct 2007 13:47:30 GMT

Using ASP.NET 2, we're seeing Appdomain recycles pretty frequently with a low load - just a few users on our test system. The reason given in the eventlog is 'HostingEnvironment caused shutdown', which looks to me like a memory issue.

Looking at memory, we see big chunks being taken every time another user logs in and the wp process gets to 2.5G, at which point I guess it decides to recycle itself.

I can't find any reason for that amount of memory being used and on my development laptop, that doesn't happen. We are using Ajax, and WCF to communicate with the data layer. Does anyone have any experience of that combination and whether there are known performance/memory issues.

Cheers, Rob

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Tess — Mon, 08 Oct 2007 15:49:34 GMT

Since you have a pretty well defined scenario where you see memory going up everytime someone logs in, an idea would be to

attach windbg or visual studio for that matter,

load up sos,

do !dumpheap -stat just before the log in

hit g to go

!dumpheap -stat after the log in

compare the outputs to see what types of objects have increased the most and investigate those.

Alternatively you could hook up CLR Profiler and look there at what objects have been instantiated while you were logging in.

Thanks

Tess

re: ASP.NET Case Study: Lost session variables and appdomain recycles

George — Wed, 10 Oct 2007 23:28:44 GMT

We've been experiencing this same issue for quite some time now. Thanks to your blog entry here, I was able to confirm that it is indeed a configuration change which causes our worker process to recycle. Specifically, I see the "Application is shutting down. Reason: Configuration changed." message in the event log now.

I figured it would be fairly simple to track down the culprit (indexing service is off, and virus scanner is explicitly set to ignore any directories that contain config files), but that has proven to not be the case. I left FileMon running overnight (filtering on *.config) and waited until the worker process recycled. Unfortunately, it looks as if nothing accessed (read or write) a config file immediately before the recycle occured. The sequence of events is as follows:

Event code: 1002

Event message: Application is shutting down. Reason: Configuration changed.

Event time: 10/4/2007 5:01:27 PM

Event time (UTC): 10/4/2007 9:01:27 PM

Event ID: de5016e0e27944dc82f644b975c01d38

Event sequence: 140208

Event occurrence: 1

Event detail code: 50004

Process information:

Process ID: 6052

Process name: w3wp.exe

Account name: NT AUTHORITY\NETWORK SERVICE

and from the FileMon log:

5:00:24.164 PM w3wp.exe:6052 FASTIO_QUERY_OPEN D:\AppRoot\main\developments\web.config NOT FOUND Attributes: Error

5:01:13.917 PM w3wp.exe:6052 FASTIO_QUERY_OPEN D:\AppRoot\main\reports\web.config NOT FOUND Attributes: Error

5:01:13.917 PM w3wp.exe:6052 IRP_MJ_DIRECTORY_CONTROL D:\AppRoot\main\reports\ NO SUCH FILE FileBothDirectoryInformation: web.config

5:01:13.917 PM w3wp.exe:6052 FASTIO_QUERY_OPEN D:\AppRoot\main\reports\web.config NOT FOUND Attributes: Error

5:01:27.448 PM w3wp.exe:6052 FASTIO_QUERY_OPEN C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Config\machine.config SUCCESS Attributes: A

5:01:27.448 PM w3wp.exe:6052 IRP_MJ_DIRECTORY_CONTROL C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Config\ SUCCESS

FileBothDirectoryInformation: machine.config

5:01:27.448 PM w3wp.exe:6052 IRP_MJ_CREATE C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Config\machine.config SUCCESS Options: Open Access:

00020000

5:01:27.448 PM w3wp.exe:6052 IRP_MJ_QUERY_SECURITY C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Config\machine.config BUFFER OVERFLOW

Repeat 1000x as the restart finishes. The curious thing about this is twofold:

1) FileMon reports the same PID for the worker process before, during, and after the restart. A new process should be spun up after the restart, correct?

2) There are multiple worker processes running on this machine, but only one ever restarts. This all but rules out global web.confg/machine.config changes.

Is there anything else other than .config files that ASP.NET considers to be configuration files? Any ideas would be greatly appreciated.

Thanks for your time,

George

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Tess — Mon, 15 Oct 2007 11:02:59 GMT

Hi George,

Not sure why the config changes are being reported, but to answer your questions

1) The workerprocess doesnt restart only the application domain inside it so the PID should stay the same. It will only change if the whole process restarts.

2) No, only .config files are considered configuration files.

I do seem to recall some issues reported in the early stages of 2.0 with false config restarts because of exceptions under certain conditions but i dont recall the exact details. You might just want to check that you are on the latest rollups/service packs for the framework if you haven't checked already.

re: ASP.NET Case Study: Lost session variables and appdomain recycles

George — Tue, 23 Oct 2007 23:08:25 GMT

Hi Tess,

Thanks for the clarification.

It looks like the framework is indeed up to date. The only other thing I can think of is that we recently began writing data to a directory inside of the application's directory tree, although Filemon confirms there are no writes in the moments leading up to the recycle. Maybe this is tangentially related to the overwhelming changes (http://support.microsoft.com/?id=913297) issue?

I'll keep after it and post back here if I discover anything new.

Thanks again,

George

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Jethro — Fri, 07 Dec 2007 07:27:44 GMT

the perfmon counter ASP.NET v…/Application Restarts always ==0 for 2 hours, BUT the eventLog is full of this event, see Event occurrence: 36401 ! Help ！！！

Event code: 4005

Event message: Forms authentication failed for the request. Reason: The ticket supplied has expired.

Event time: 12/7/2007 11:17:06 AM

Event time (UTC): 12/7/2007 4:17:06 AM

Event ID: 57a0de50fe674a3fb5ab61c36938c1a7

Event sequence: 85310

Event occurrence: 36401

Event detail code: 50202

re: ASP.NET Case Study: Lost session variables and appdomain recycles

salsam — Fri, 14 Mar 2008 16:38:29 GMT

I have also facing same problem. during normal execution of the site my session information getting lost.

Event log is full of such errors.

Event code: 4005

Event message: Forms authentication failed for the request. Reason: The ticket supplied has expired.

Event time: 14/03/2008 13:19:26

Event time (UTC): 14/03/2008 13:19:26

Event ID: d9f0333c2eed46e0b0207da69d2ea70e

Event sequence: 154

Event occurrence: 5

Event detail code: 50202

If some on find solution of this problem please do post here.

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Tess — Fri, 14 Mar 2008 16:43:37 GMT

Have you gone through and verified if your appdomain is recycling (in performance monitor) that is the most common case for lost session variables.

Did you follow hte article and log application restarts in the eventlog? The solution will depend on why the application/appdomain is restarting...

re: ASP.NET Case Study: Lost session variables and appdomain recycles

salsam — Fri, 14 Mar 2008 17:03:37 GMT

If you mean, copying 'Application Lifetime Events ' in web.config file. then yes i have copied it to webconfig but didnt got any of known reason as yet, like it is being happened becos of 'Configuration changed.'

do i have to restart webserver in order to make below mention changes come in to effect.

<add name="Application Lifetime Events Default" eventName="Application Lifetime Events"

provider="EventLogProvider" profile="Default" minInstances="1"

maxLimit="Infinite" minInterval="00:01:00" custom="" />

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Tess — Fri, 14 Mar 2008 17:45:08 GMT

No you shouldnt have to, the best thing is probably to contact support and have someone look at it in more detail

re: ASP.NET Case Study: Lost session variables and appdomain recycles

salsam — Fri, 14 Mar 2008 18:05:18 GMT

<add name="Application Lifetime Events Default" eventName="Application Lifetime Events"

provider="EventLogProvider" profile="Default" minInstances="1"

maxLimit="Infinite" minInterval="00:01:00" custom="" />

Even after adding this , still i m getting this in event log.

Event code: 4005

Event message: Forms authentication failed for the request. Reason: The ticket supplied has expired.

Event time: 14/03/2008 13:19:26

Event time (UTC): 14/03/2008 13:19:26

Event ID: d9f0333c2eed46e0b0207da69d2ea70e

Event sequence: 154

Event occurrence: 5

Event detail code: 50202

re: ASP.NET Case Study: Lost session variables and appdomain recycles

salsam — Tue, 18 Mar 2008 02:36:51 GMT

hi Tess,

it is now confirmed that in my case appdomain is not recycling. beoc since one day i have received even single event of appdomain recycle. so problem is somewhere else. i have also update machine.cofig file with custom generated keys but still having same errors in evenlog.

please suggest some solution

thanks

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Tess — Tue, 18 Mar 2008 10:20:56 GMT

I would have to look at it in a lot more detail to see what is going on and unfortunately I can't really do that in this blog setting so the best is probably to contact support and have someone look at it for you. sorry:(

My presentation on Troubleshooting ASP.NET production issues at Developer Summit 2008

If broken it is, fix it you should — Fri, 11 Apr 2008 14:24:35 GMT

The purpose of my presentation was to show some common pitfalls and of course to show off windbg and

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Salsam — Thu, 08 May 2008 17:29:46 GMT

Hi guys,

I was working on the issue where event viewer showing the log "Forms authentication failed for the request. Reason: The ticket supplied has expired.

Finally able to get to the roots of this message. here what it was happening to me.

I was working on web application which is using Form Authentication and Sessions to store the user specific profiles.

Here is my setting in web.config file.

</forms>

</authentication>

Now problem was before i didn't had form authentication timeout, which had default of 30 mins. So when request is made after being idle for 30 mins, Form authentication was throwing the user back to login page. beacuse after 30 mins the ticket supplied with the request gone expired. and this also logged warrning that user was thrown out beacuse of expire ticket passed.

other problem i was having that Form Authentication was unable to redirect the user back to login page and end up with page no found error. reason for that was, missing <authorization> tags.

ForAuthentication requires deny anonymous access in the Authorization tags in order to redirect the user back to login page in case if authorization failed or timeout.

So in order to fix these issues i have stick in the Authorization tags. and along with that, i have also set the Form Authentication timeout same as session timeout.

But this still give the warraning in Event viewer beacuse if session and Form Autentication Cookie is expiring at same time. and user makes any request after the timout first things which triggered is Form Authentication and it will throw the user to login page and log the event in Event viewer.

To do quick fix i have increased Form Autentication timeout to 240 mins by leaving the session timout 60 just to make session timout before Form Authentication timeout. So when session timeout after 60 mins Code is already there to take the user back to login page so user can re loggin.

If any one has batter solution to this problem, please do share it

thanks

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Adam — Tue, 13 May 2008 12:05:43 GMT

I just thought I would share another scenario I came across, not that its because of application pool recycling but because it looked like it at first.

I had an application that kept losing its session state and the actual reason was I had a maximum number of worker processes allowed set to 5.

Because the application used inproc session state the session state was lost each time it jumped a worker process. The setting is in the properties of the Application Pool.

Thanks

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Tess — Tue, 13 May 2008 12:12:30 GMT

Absolutely, the same is of course true for web farms.

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Simone Busoli — Wed, 14 May 2008 01:45:13 GMT

Hi Tess,

I was wondering why the waitChangeNotification and maxWaitChangeNotification settings support has been dropped in IIS 7. It was the only mechanism available for hot swapping applications on the fly.

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Tess — Wed, 14 May 2008 08:55:22 GMT

I honestly had no clue that that was the case, and it surprises me, because that is a framework setting, not an IIS setting.

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Simone Busoli — Wed, 14 May 2008 10:05:20 GMT

Tess, look here, point 25.

http://mvolo.com/blogs/serverside/archive/2007/12/08/IIS-7.0-Breaking-Changes-ASP.NET-2.0-applications-Integrated-mode.aspx

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Simone Busoli — Wed, 14 May 2008 10:08:32 GMT

Oh, and obviously, this occurs in the Integrated Pipeline mode only.

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Tess — Wed, 14 May 2008 10:24:20 GMT

I would have to look at this in more detail and unfortunately I do not have an IIS 7 machine set up at the moment, but the bullet item says:

__________

25) IIS 7.0 always restarts ASP.NET applications when changes are made to the application’s root web.config file. Because of this, waitChangeNotification and maxWaitChangeNotification attributes have no effect.

IIS 7.0 monitors changes to the web.config files as well, and cause the ASP.NET application corresponding to this file to be restarted without regard to the ASP.NET change notification settings including the waitChangeNotification and maxWaitChangeNotification attributes in the system.web/httpRuntime configuration sections.

_____________

My comments: Web.config changes have always been monitored and do restart the application domain. Off the bat I can't see how this disables waitChangeNotifications other than for changes in web.config.

One of the biggest reasons you would use them is to be able to update multiple dlls at a time without restarts, and from this comment I am not sure that that would change. The only thing that this comment seems to suggest is that waitChangeNotifications are not honored for web.config changes.

I might be wrong though as I haven't tested it personally.

ASP.NET Performance issue: Large number of application restarts due to virus scanning

If broken it is, fix it you should — Wed, 14 May 2008 10:44:18 GMT

Issues caused by frequent application restarts are pretty common. If you are interested in what causes

re: ASP.NET Case Study: Lost session variables and appdomain recycles

simone_b — Wed, 14 May 2008 12:44:31 GMT

Yes, it happens to ignore the settings only when changes to the web.config file occur, but it represents a major issue since replacing the web.config file is exactly what happens when doing hot swap of applications.

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Tess — Wed, 14 May 2008 14:06:58 GMT

Still that would only mean one appdomain recycle, which is what you would see anyways with the maxWaitChangeNotification, so I am not sure I understand the issue...

re: ASP.NET Case Study: Lost session variables and appdomain recycles

simone_b — Wed, 14 May 2008 14:21:45 GMT

This is the issue, supposing that I don't set any waitChangeNotification or maxWaitChangeNotification:

I trigger a batch update, using robocopy for example, which takes all files from the source directory and copies them over the files in the old directory, where the application is running.

1. some - but not all - files from the source bin folder are copied to the destination - the application restarts immediately

2. the web.config file is copied - the application restarts immediately

3. the rest of the content of the bin folder is copied, the application restarts again.

This is the issue, supposing that I set waitChangeNotification = 10 and maxWaitChangeNotification to a big value (meaningless in this example):

1. some - but not all - files from the source bin folder are copied to the destination - this takes much less than 10 seconds - the application doesn't restart

2. the web.config file is copied - the application restarts immediately

3. the rest of the content of the bin folder is copied, the application waits 10 seconds from the last copy then restarts.

Here we have 2 restarts, one of them probably error prone, since not all assemblies were copied and might be incompatible to each other.

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Tess — Wed, 14 May 2008 14:36:40 GMT

ok, then you would either have to do the robocopy first, and then copy the web.config or do the update when the site is not live...

re: ASP.NET Case Study: Lost session variables and appdomain recycles

simone_b — Wed, 14 May 2008 14:54:06 GMT

Yes, that's the workaround I came up with, but that's not as optimal as it was before, and mostly I can't see the reason why it should have been changed in IIS7, it worked so well in the previous versions..

The 21 most popular blog posts

If broken it is, fix it you should — Wed, 21 May 2008 16:45:26 GMT

I started this blog 2.5 years ago today, mostly because I felt that the same types of issues came up

The root cause for application performance problems might seem to be totally unrelated

Green Dragon's Lair — Sun, 25 May 2008 01:12:01 GMT

I just read a post on If broken it is, fix it you should , where he's debugging a performance problem

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Mike Driest — Tue, 19 Aug 2008 23:32:42 GMT

Hello Tess,

On Thursday, February 01, 2007 7:27 AM you wrote:

-------------------------------------------------------------------------

Hi Payal,

Pretty much every time I have seen this it has been one of two things.

2. A router or load balancer is used and the users come from the same ISP getting the same IP address so the router gets confused and returns the session cookie to the wrong user

The second scenario is a misconfiguration in a router, and is by far much less frequent than the first scenario.

-------------------------------------------------------------------------

We have a similar issue with one of our Internet facing ASP.NET 2.0 Applications with only 1 customer/computer where we're seeing 3 Public IP Addresses hit our web servers (2 node with NLB) in a matter of 3.5 minutes. These 3 Public IP Addresses from the Client are in the same Subnet and are literly .101, .102 and .103. Our ASP.NET App uses .NET Forms Authentication and I think it's possible that the session is being responded to via the Wrong Client Public IP Address. The user is logged out shortly after they login to the application and try to execute a Search.

Our code barely uses caching and other customers/users don't have this issue from other corporate networks, computers or ISPs. We've used the user's credentials on PCs on the Public Internet and our Corporate computers successfully without errors or being logged out.

Would you agree that the issue lies within the client/customer network infrastructure (firewalls, routers, etc)?

Thank you,

Mike Driest

Questions on application domains, application pools and unhandled exceptions

If broken it is, fix it you should — Wed, 20 Aug 2008 11:18:34 GMT

I got an email with some questions around application domains, application pools and unhandled exceptions

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Craig Hubbard — Fri, 19 Sep 2008 03:54:50 GMT

Thanks for this post, the recycling issue has become quite an issue for our DotNetNuke implemenations of recent times (Sept 2008) and this post will help me to investigate further - great stuff!

Common reasons for the session loss issue in ASP.NET applications

Akshay's Weblog — Mon, 29 Sep 2008 15:54:16 GMT

I have handled a lot of session related issues till now; I felt the need to key in a few checkpoints

ASP.NET v2.0 - AppDomain recycles, more common than before

Scott Forsyth's Blog — Wed, 22 Oct 2008 16:59:44 GMT

I stumbled across a very interesting find yesterday morning and Todd Carter confirmed it by sending a

Troubleshooting appdomain restarts and other issues with ETW tracing

If broken it is, fix it you should — Thu, 06 Nov 2008 13:33:14 GMT

You learn something new every day… I was working on an issue with appdomain restarts caused by page re-compilations

[ASP.NET]Good reference about troubleshooting ASP.NET application restart problem

Steven Cheng's MSDN Notes — Mon, 02 Feb 2009 12:17:07 GMT

Application restart is a common issue in ASP.NET web application. Here is a very good blog entry providing

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Måns Tånneryd — Tue, 17 Feb 2009 12:29:20 GMT

Nice post! But, I have a problem with app domain restarts that does not seem to be covered in your post. My web application runs smoothly on a win2003 server in iis 6 but when I deployed it on a win2008 server with iis 7 it started hickupping. After about 60 seconds the appdomain restarts due to "_shutDownMessage=IIS configuration change". No configuration files have been changed! What is happening though is that the web application is downloading files using ftp on a background thread and placing them in a directory in the app_data folder. I seems unreasonable to me that adding files in the app_data folder should cause the application to restart. Calling an "IIS configuration change" doesn't make any sense to me either!

Any ideas?

/Måns

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Tess — Tue, 17 Feb 2009 18:23:06 GMT

Hi Måns,

Just adding or removing files from there should not restart the app, but if something creates or deletes subfolders, even in the app_data folder then you will see app restarts with the iis config change message

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Nick Crowther — Tue, 28 Apr 2009 17:54:11 GMT

Just fixed my session loss problem (deleting folders in App_Data). Thanks Tess (& Måns :), useful blog.

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Siddhartha Bhuyan — Tue, 19 May 2009 16:34:17 GMT

Nice article Tess. Got a really good understanding about application domain recycling.

I have been working on a content management website since the past few weeks and my part involves the file handling functionalities.

Our website actually has to create files, folders etc. in a 3-level hierarchy and also manipulate them based on what the user does.

While everything was working fine, I kept getting puzzled as to why, everytime a directory was renamed (moved) successfully, I used to lose everything in the session... Debugging only revealed that the session was getting invalidated.

Only after I read this and Scott Forsyth's article, did the problem become clear to me.

But I have to admit that this is a sticky problem. And there is no easy way to work around this one.

Keep writing!

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Surat Jewpojaroen — Tue, 07 Jul 2009 12:06:49 GMT

Thanks for sharing knowledge.

It's really good article.

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Codru — Mon, 17 Aug 2009 13:33:31 GMT

Hi!

I have a question, maybe you could help me. For example, if I try to submit a job (request to the server) after a recycle, my job is lost - no trace of it, no errors are logged, nothing but the information in event viewer that the application has recycled. I understand why this happens, but I would like to know if there's by any chance a way of showing a message to the user that something happened and he will be logged out when submitting the job? Or maybe after he logged back in?

I cannot see a way of doing this, since all the dlls need to be reloaded, but I thought it's worth a shot to ask :)

Thanks for the article, it's really good and revealing :)

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Tess — Mon, 17 Aug 2009 14:43:07 GMT

i dont really see a way that you could do that... unless perhaps you keep some session variables and detect that they are lost for example

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Josh — Fri, 21 Aug 2009 22:37:58 GMT

Great article Tess! I have been troubleshooting recycle events for a few days. A combination of virus scanning and mis-used app pools in IIS6 were the trouble. Thanks again.

re: ASP.NET Case Study: Lost session variables and appdomain recycles

csaxena — Fri, 18 Sep 2009 01:27:01 GMT

Great blog Tess. I have learnt so much from it. I have a few questions. Is there any disadvantage of setting the value for maxWaitChangeNotification and waitChangeNotification to be really high (like 10 hours)? We are seeing a lot of Application restarts in our DotNetNuke environment and we are not making any changes. Is it because the different user controls on different pages are getting compiled and causing the application to be reloaded.

What is the disadvantage (other than the memory footprint) of increasing numRecompilesBeforeAppRestart to a really high number like 500 or 1000?

Thanks !

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Tess — Fri, 18 Sep 2009 10:18:39 GMT

csaxena,

There are basically two disadvantages.

1. Setting numRecompilesBeforeAppRestart to something really high, as you pointed out will mean that you store more copies of stale dlls in the process leading to higher memory usage. That is pretty much the only drawback you will see from that...

2. setting maxWaitChangeNotification and waitChangeNotification to something really high like 10 hrs means that changes made during that time that require appdomain restarts wont take effect. For example if you modify the web.config, the old settings will be "cached" in that case for 10 hrs and new ones wont take effect until the appdomain is restarted. Same with dll updates. As long as you are aware of that and restart the appdomain manually (by restarting the process) when changes that you need to see go through immediately are made, you'll be ok. In general I really wouldn't recommend setting this higher than 5-10 minutes though...

re: ASP.NET Case Study: Lost session variables and appdomain recycles

frankosaurus — Sun, 06 Dec 2009 22:15:05 GMT

Great article Tess...

I have an ASP.NET application (in IIS7) where content editors make regular updates to .resx files.

The problem is that this eventually causes the app domain to recycle.

With changes to .aspx pages, I can set compilationMode="never" in web.config and I avoid the problem of excess recompiles.

Is there a solution for .resx files??

I would prefer not to set numRecompilesBeforeAppRestart to a high value (or 0) because I don't want to use up RAM -- I need RAM for ASP.NET cache.

re: ASP.NET Case Study: Lost session variables and appdomain recycles

Tess — Mon, 07 Dec 2009 10:07:57 GMT

unfortunately i dont think there is, but even if there was, that would mean that the updates wouldnt go through until you manually recycle. So given that, perhaps it is better to have the content editors edit to a staging location and copy them all in one go at some point.

re: ASP.NET Case Study: Lost session variables and appdomain recycles

srinivasan — Sun, 13 Dec 2009 15:49:14 GMT

''Sec 1. a.dll and b.dll are update to v 1.00.12 - appdomain unload started (any pending requests will finish before it is completely unloaded)

Sec 2. Request1 comes in and loads a new appdomain with 2 out of 7 of the dlls updated

Sec 3. c.dll is updated - appdomain unload started (any pending requests will finish before it is completely unloaded)

Sec 4. d.dll is updated

Sec 5. Request2 comes in and loads a new appdomain, now with 4 out of 7 dlls updated

Sec 6. e.dll and f.dll is updated - appdomain unload started (any pending requests will finish before it is completely unloaded)

Sec 7. f.dll is updated

Sec 8. Request3 comes in and loads a new appdomain with all 7 dlls updated

So, many bad things happened here…

First off you had 3 application domain restarts while you probably thought you would only have one, because asp.net has no way of knowing when you are done.''

-----------------------------------

In the above scenario, don't you feel there are more than 3 appdomain restarts, but you've stated there are 3. Am I getting this wrong ?