HOWTO: Understand and Diagnose an Application Pool Crash

re: HOWTO: Understand and Diagnose an Application Pool Crash

Ian Cox — Tue, 30 Aug 2005 14:40:51 GMT

Great post David. I am one of those whose problems you quoted at the start. Everything you say here makes a lot of sense.

re: HOWTO: Understand and Diagnose an Application Pool Crash

Oliver Ogg — Thu, 22 Sep 2005 16:33:24 GMT

Thanks for a well written and informative post.

re: HOWTO: Understand and Diagnose an Application Pool Crash

David Wang — Mon, 05 Dec 2005 21:52:34 GMT

I had gotten a comment about this blog entry:

Hi there, just read your blog here:

http://blogs.msdn.com/david.wang/archive/2005/08/29/HOWTO_Understand_and_Diagnose_an_AppPool_Crash.aspx

Would you say that this following error would fit into this category of a "crash" and your diagnosis techniques in this blog would be recommended? The worker process seems to restart, but I am consistently getting this crash. The primary software appliation I am running is called ACT, using a web front-end known as WiredContact.

ERROR MESSAGE:

Faulting application w3wp.exe, version 6.0.3790.1830, faulting module mfc42.dll, version 6.6.8063.0, fault address 0x00022659.

--> Yes, this exactly matches one of the patterns I gave for what a "crash" would look like on IIS6.

If you attach a debugger onto the w3wp.exe or use a tool like IIS State or DebugDiag it will immediately identify the code that is actually crashing when it happens, so you can start looking for support from the correct party.

//David

re: HOWTO: Understand and Diagnose an Application Pool Crash

Kirk Francis — Fri, 09 Dec 2005 05:40:07 GMT

Well done David, this information was well thought out and quite helpful to me. Like others, I've looked for a direct resolution to an Event Log message to no avail. Things are better with IIS 6.0 no doubt, but with that come the complexities of arriving at a root cause.

re: HOWTO: Understand and Diagnose an Application Pool Crash

Randy — Fri, 06 Jan 2006 18:52:50 GMT

Well done David!! Hopefully DebugDiag will be a final release.. but until then I have found another helpful article which talks about running the debug symbols and what not

http://support.microsoft.com/kb/911359/EN-US/

Keep up the good work.. now about the IIS 7.0 Beta ;)

re: HOWTO: Understand and Diagnose an Application Pool Crash

David.Wang — Mon, 13 Feb 2006 17:39:10 GMT

Woohoo... my second 10K+ blog entry...

You all seem to like howto diagnosis entries. :-)

//David

re: HOWTO: Understand and Diagnose an Application Pool Crash

Peter Stathakos — Sat, 04 Mar 2006 03:09:45 GMT

Hi David,

Thanks for the information, I will try attaching the IIS debug tools and see if it helps to diagnose my issue.

re: HOWTO: Understand and Diagnose an Application Pool Crash

Ahmed El-Rasheedy — Thu, 09 Mar 2006 00:43:10 GMT

Hi David,
We are running JRun/ColdFusion Multi-Server Cluster (clustered using the Application server) on several Win 2003 machines. The machines that are having the error are running Windows 2003 SP1 while a third machine which does not have the Windows SP1 is not having the error. Here are the symptoms:
Users get "Service Unavailable" in their browser and we see the following error and warnings in the EventViewer.

Event Type: Warning
Event Source: W3SVC
Event Category: None
Event ID: 1011
Date: 3/7/2006
Time: 7:21:55 PM
User: N/A
Computer: SLIOBTWEB2
Description:
A process serving application pool 'DefaultAppPool' suffered a fatal communication error with the World Wide Web Publishing Service. The process id was '5844'. The data field contains the error number.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 6d 00 07 80 m..?

Then after 5 warning message as above we get the below error. I found the number five in the IIS Application Pool (it is a setting to shut down w3wp.exe after 5 errors in f5 minutes)
Event Type: Error
Event Source: W3SVC
Event Category: None
Event ID: 1002
Date: 3/7/2006
Time: 7:22:03 PM
User: N/A
Computer: SLIOBTWEB2
Description:
Application pool 'DefaultAppPool' is being automatically disabled due to a series of failures in the process(es) serving that application pool.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

I captured a debug dump and I am not sure what to do with it or who should I submit it to. Can you please let me know.
thanks,

re: HOWTO: Understand and Diagnose an Application Pool Crash

David.Wang — Thu, 09 Mar 2006 01:07:15 GMT

Ahmed - as I mentioned in the blog entry:
you want to either analyze the stack trace log yourself, pay someone to perform an analysis (for example, Microsoft PSS) or post to a newsgroup like microsoft.publit.inetserver.iis to see if anyone will do a free analysis.

You have a classic "something is crashing in the application pool" and if you captured the stack trace of the crash (and not before/after the crash), you should be able to see in DebugDiag logfile what is at fault and who to follow up with.

At this point, you only know that on WS03SP1 machines the issue can happen, but you do not know whether it is SP1 or some application's reaction to SP1 that is at fault. The stack trace of the crash helps identify which situation.

//David

HOWTO: Basics of IIS6 Troubleshooting

David Wang — Wed, 15 Mar 2006 00:35:37 GMT

I recently sat down and thought a little about the typical user experience when troubleshooting IIS6,...

re: HOWTO: Understand and Diagnose an Application Pool Crash

Sudeep — Mon, 20 Mar 2006 17:58:47 GMT

Hi David Please help me by telling how to proceed with my problem. I know why my IIS is crashing but i dont know how to fix it. Whenever i try to access a custom ISAPI dll from the browser, IIS is crashing, worker process identity is network service, IIS does not crash if i use local system account. Obviously it is an access violation problem. I am trying to give more access rights to network service rather than using local system account but with no success. I dont want my worker process identity to be local system account. Could u guide me how to go about giving only the required access to network service. Any help from anyone is appreciated. Thank you.

re: HOWTO: Understand and Diagnose an Application Pool Crash

Sudeep — Thu, 23 Mar 2006 13:39:26 GMT

does anyone how to debug com dlls in vb environment for IIS 6 , windows server 2003? it is unlike 2000 where we unregister dll and run the project and put a breakpoint and can go ahead debugging. i could not find anywhere the architecture of how this is happening also...

re: HOWTO: Understand and Diagnose an Application Pool Crash

Chris Vandecaveye — Fri, 24 Mar 2006 17:51:37 GMT

Hi David,
Of course your article makes sense. Your article describes how I would normally handle a crash (or hang). But now I'm in a situation where I cannot apply this procedure. What is happening?
We have an ASP.NET (1.1) application. We use a pool of 8 w3wp's in IIS 6.0. Clients are using ie 5.5 to connect to the application. Every now and then, we cannot connect to the application anymore from certain clients while other clients are still working. Even more, starting the application on the webserver works fine, starting the same application on another server in that domain works fine. The clients do not get any message (blank white screen). There are no events in the eventlog. IIS is still working because we can still connect to other applications on the same webserver. We added a lot of debugcode in our application, but we don't even reach the first line of code! So, the error must be situated between the incomming request and our application.
I would greatly appreciate some help here, and yes we already placed a call at Microsoft without any result until now.
I find it more important to get some suggestions on how to debug/monitor this issue, rather than have the solution without me nowing what I am doing.
Thanks in advance!

re: HOWTO: Understand and Diagnose an Application Pool Crash

Chris Vandecaveye — Fri, 24 Mar 2006 17:53:00 GMT

re: HOWTO: Understand and Diagnose an Application Pool Crash

David.Wang — Fri, 24 Mar 2006 22:30:37 GMT

Chris - I would look at HTTPERR logs for the affected client's IP to see if the request from the affected client even got to the web server machine, HTTP.SYS, and IIS6.

http://blogs.msdn.com/david.wang/archive/2005/12/31/HOWTO_Basics_of_IIS6_Troubleshooting.aspx

If the request got to the web server and HTTP.SYS did not report it dropped in HTTPERR logs, then we need to isolate which w3wp.exe is having issues with it. You are using a pool of 8 w3wp.exe which makes it impossible to track down which w3wp.exe is handling the failing request (HTTP.SYS round-robins requests by connection when using Web Garden).

You can either attach debugging tools to all w3wp and hope for something to happen that you can identify with the debugger, or make the Application Pool use only one w3wp.exe.

Using one w3wp.exe will affect availability since other applications using the same Application Pool will also be out-of-commission, so please adjust debugging accordingly.

//David

re: HOWTO: Understand and Diagnose an Application Pool Crash

Chris Vandecaveye — Mon, 27 Mar 2006 16:42:19 GMT

Thanks for your reply David.
I will suggest using a second server for availability purpose so we can use one w3wp.exe instead of the pool of 8.
Some thing I do notice but i do not have the knowledge to interprete it or pinpoint the problem.
- In the http log I see a lot of 'Timer_ConnectionIdle' and sometimes 'Connection_Dropped' but the application is still working. Is this normal? I will pay special attention to these logs in case of connection problems.
- The use of virtual memory for each w3wpc.exe is high +/- 400MB but stable, while the available physical memory continues to drop from +/- 700MB available to 5MB available. When physical memory is at 5MB available, it goes back to 700MB available.

re: HOWTO: Understand and Diagnose an Application Pool Crash

David.Wang — Tue, 28 Mar 2006 13:06:16 GMT

Chris - Explanation of HTTPERR Log entries:
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/http/http/types_of_errors_logged_by_the_http_server_api.asp

Sounds like some sort of process recycling is kicking in to free up available physical memory.

//David

re: HOWTO: Understand and Diagnose an Application Pool Crash

Sudeep — Tue, 28 Mar 2006 15:43:54 GMT

re: HOWTO: Understand and Diagnose an Application Pool Crash

Sudeep — Tue, 28 Mar 2006 15:56:22 GMT

I moved asp website from 2000 to 2003, and my iis server was crashing, upon using debug diagnostics tool, it says "first chance exception which is Visual Basic Runtime exception with VB Error #: 53 ....
Also ntdll.dll is crashing... can anyone tell me why this is happening or how to debug vb dll in 2003 as this exception does not occur on a 2000 machine?

here is the stack trace for your perusal............ this is when ntdll.dll crashed due to access violation.......

Entry point w3tp!THREAD_MANAGER::ThreadManagerThread
Create time 3/22/2006 1:45:14 PM
Time spent in user mode 0 Days 0:0:1.656
Time spent in kernel mode 0 Days 0:0:0.62

Function Arg 1 Arg 2 Arg 3 Source
ntdll!RtlpLocateActivationContextSection+c 00070000 00000000 00000002
ntdll!RtlpFindNextActivationContextSection+64 00d3308c 00d330c4 00d330b0
ntdll!RtlpFindFirstActivationContextSection+41 00d3308c 00d330c4 00d330b0
ntdll!RtlFindActivationContextSectionString+91 00000000 00000000 00000002
ntdll!LdrpCheckForLoadedDll+f7 000a8990 00d33208 00000000
ntdll!LdrpLoadDll+1b3 00000000 000a8990 00d33724
ntdll!LdrLoadDll+198 000a8990 00d33724 00d33704
kernel32!LoadLibraryExW+1b2 7ffd5c00 00000000 00000000
kernel32!LoadLibraryExA+1f 01b03118 00000000 00000000
kernel32!LoadLibraryA+b5 01b03118 00d33c88 00d33794
RtCppUtils!ImgHelp::ImgHelp+6f 01a3a5d0 00000000 00000000
RtCppUtils!RTCppLog::RTCppLog+f0 c0000005 00d33cd0 e06d7363
msvcrt!_CallSETranslator+97 00d33eec 00d342e0 00d33f08
msvcrt!__CxxExceptionFilter+1d1 00d33eec 00d342e0 00d33f08
msvcrt!__CxxExceptionFilter+58f 00d33eec 00d342e0 00d33f08
msvcrt!__InternalCxxFrameHandler+b2 00d33eec 00d342e0 00d33f08
msvcrt!__CxxFrameHandler+28 00d33eec 00d342e0 00d33f08
ntdll!ExecuteHandler2+26 00d33eec 00d342e0 00d33f08
ntdll!ExecuteHandler+24 00d33000 00d33f08 00d33eec
ntdll!KiUserExceptionDispatcher+e 00d33000 00d33f08 00d33eec
ntdll!RtlEnterCriticalSection+19 00000020 00d34228 77bd1d69
msvcrt!_lock_file+33 00000000 01a3a5d0 00d34acc
msvcrt!fprintf+18 00000000 01b010c8 01621d69
RtCppUtils!RTBaseLog::logMessage+17e 00d34420 00d3438c 00250168
RtCppUtils!RTCppLog::log+dd 00000001 00d34420 00d34788
RtCppUtils!RTCppLogPtr::log+1e 00000001 00d34420 01a3a5d0
RtCppUtils!RTCppLog::RTCppLog+1f1 c0000005 00d348b0 e06d7363
msvcrt!_CallSETranslator+97 00d34acc 00d34ec0 00d34ae8
msvcrt!__CxxExceptionFilter+1d1 00d34acc 00d34ec0 00d34ae8
msvcrt!__CxxExceptionFilter+58f 00d34acc 00d34ec0 00d34ae8
msvcrt!__InternalCxxFrameHandler+b2 00d34acc 00d34ec0 00d34ae8
msvcrt!__CxxFrameHandler+28 00d34acc 00d34ec0 00d34ae8
ntdll!ExecuteHandler2+26 00d34acc 00d34ec0 00d34ae8
ntdll!ExecuteHandler+24 00d34000 00d34ae8 00d34acc
ntdll!KiUserExceptionDispatcher+e 00d34000 00d34ae8 00d34acc
ntdll!RtlEnterCriticalSection+19 00000020 00d34e08 77bd1d69
msvcrt!_lock_file+33 00000000 01a3a5d0 00d356ac
msvcrt!fprintf+18 00000000 01b010c8 01621ad9
RtCppUtils!RTBaseLog::logMessage+17e 00d35000 00d34f6c 00250164
RtCppUtils!RTCppLog::log+dd 00000001 00d35000 00d35368
RtCppUtils!RTCppLogPtr::log+1e 00000001 00d35000 01a3a5d0
RtCppUtils!RTCppLog::RTCppLog+1f1 c0000005 00d35490 e06d7363
msvcrt!_CallSETranslator+97 00d356ac 00d35aa0 00d356c8
msvcrt!__CxxExceptionFilter+1d1 00d356ac 00d35aa0 00d356c8
msvcrt!__CxxExceptionFilter+58f 00d356ac 00d35aa0 00d356c8
msvcrt!__InternalCxxFrameHandler+b2 00d356ac 00d35aa0 00d356c8
msvcrt!__CxxFrameHandler+28 00d356ac 00d35aa0 00d356c8
ntdll!ExecuteHandler2+26 00d356ac 00d35aa0 00d356c8
ntdll!ExecuteHandler+24 00d34000 00d356c8 00d356ac
ntdll!KiUserExceptionDispatcher+e 00d34000 00d356c8 00d356ac
ntdll!RtlEnterCriticalSection+19 00000020 00d359e8 77bd1d69
msvcrt!_lock_file+33 00000000 01a3a5d0 00d3628c
msvcrt!fprintf+18 00000000 01b010c8 01621849
RtCppUtils!RTBaseLog::logMessage+17e 00d35be0 00d35b4c 00250168
RtCppUtils!RTCppLog::log+dd 00000001 00d35be0 00d35f48
RtCppUtils!RTCppLogPtr::log+1e 00000001 00d35be0 01a3a5d0
RtCppUtils!RTCppLog::RTCppLog+1f1 c0000005 00d36070 e06d7363
msvcrt!_CallSETranslator+97 00d3628c 00d36680 00d362a8
msvcrt!__CxxExceptionFilter+1d1 00d3628c 00d36680 00d362a8
msvcrt!__CxxExceptionFilter+58f 00d3628c 00d36680 00d362a8
msvcrt!__InternalCxxFrameHandler+b2 00d3628c 00d36680 00d362a8
msvcrt!__CxxFrameHandler+28 00d3628c 00d36680 00d362a8
ntdll!ExecuteHandler2+26 00d3628c 00d36680 00d362a8
ntdll!ExecuteHandler+24 00d35000 00d362a8 00d3628c
ntdll!KiUserExceptionDispatcher+e 00d35000 00d362a8 00d3628c
ntdll!RtlEnterCriticalSection+19 00000020 00d365c8 77bd1d69
msvcrt!_lock_file+33 00000000 01a3a5d0 00d36e6c
msvcrt!fprintf+18 00000000 01b010c8 016215b9
RtCppUtils!RTBaseLog::logMessage+17e 00d367c0 00d3672c 00250168
RtCppUtils!RTCppLog::log+dd 00000001 00d367c0 00d36b28
RtCppUtils!RTCppLogPtr::log+1e 00000001 00d367c0 01a3a5d0
RtCppUtils!RTCppLog::RTCppLog+1f1 c0000005 00d36c50 e06d7363
msvcrt!_CallSETranslator+97 00d36e6c 00d37260 00d36e88
msvcrt!__CxxExceptionFilter+1d1 00d36e6c 00d37260 00d36e88
msvcrt!__CxxExceptionFilter+58f 00d36e6c 00d37260 00d36e88
msvcrt!__InternalCxxFrameHandler+b2 00d36e6c 00d37260 00d36e88
msvcrt!__CxxFrameHandler+28 00d36e6c 00d37260 00d36e88
ntdll!ExecuteHandler2+26 00d36e6c 00d37260 00d36e88
ntdll!ExecuteHandler+24 00d36000 00d36e88 00d36e6c
ntdll!KiUserExceptionDispatcher+e 00d36000 00d36e88 00d36e6c
ntdll!RtlEnterCriticalSection+19 00000020 00d371a8 77bd1d69
msvcrt!_lock_file+33 00000000 01a3a5d0 00d37a4c
msvcrt!fprintf+18 00000000 01b010c8 01621329
RtCppUtils!RTBaseLog::logMessage+17e 00d373a0 00d3730c 00250168
RtCppUtils!RTCppLog::log+dd 00000001 00d373a0 00d37708
RtCppUtils!RTCppLogPtr::log+1e 00000001 00d373a0 01a3a5d0
RtCppUtils!RTCppLog::RTCppLog+1f1 c0000005 00d37830 e06d7363
msvcrt!_CallSETranslator+97 00d37a4c 00d37e40 00d37a68
msvcrt!__CxxExceptionFilter+1d1 00d37a4c 00d37e40 00d37a68
msvcrt!__CxxExceptionFilter+58f 00d37a4c 00d37e40 00d37a68
msvcrt!__InternalCxxFrameHandler+b2 00d37a4c 00d37e40 00d37a68
msvcrt!__CxxFrameHandler+28 00d37a4c 00d37e40 00d37a68
ntdll!ExecuteHandler2+26 00d37a4c 00d37e40 00d37a68
ntdll!ExecuteHandler+24 00d37000 00d37a68 00d37a4c
ntdll!KiUserExceptionDispatcher+e 00d37000 00d37a68 00d37a4c
ntdll!RtlEnterCriticalSection+19 00000020 00d37d88 77bd1d69
msvcrt!_lock_file+33 00000000 01a3a5d0 00d3862c
msvcrt!fprintf+18 00000000 01b010c8 01621099
RtCppUtils!RTBaseLog::logMessage+17e 00d37f80 00d37eec 00250168
RtCppUtils!RTCppLog::log+dd 00000001 00d37f80 00d382e8
RtCppUtils!RTCppLogPtr::log+1e 00000001 00d37f80 01a3a5d0
RtCppUtils!RTCppLog::RTCppLog+1f1 c0000005 00d38410 e06d7363
msvcrt!_CallSETranslator+97 00d3862c 00d38a20 00d38648
msvcrt!__CxxExceptionFilter+1d1 00d3862c 00d38a20 00d38648
msvcrt!__CxxExceptionFilter+58f 00d3862c 00d38a20 00d38648
msvcrt!__InternalCxxFrameHandler+b2 00d3862c 00d38a20 00d38648
msvcrt!__CxxFrameHandler+28 00d3862c 00d38a20 00d38648
ntdll!ExecuteHandler2+26 00d3862c 00d38a20 00d38648
ntdll!ExecuteHandler+24 00d37000 00d38648 00d3862c
ntdll!KiUserExceptionDispatcher+e 00d37000 00d38648 00d3862c
ntdll!RtlEnterCriticalSection+19 00000020 00d38968 77bd1d69
msvcrt!_lock_file+33 00000000 01a3a5d0 00d3920c
msvcrt!fprintf+18 00000000 01b010c8 01620e09
RtCppUtils!RTBaseLog::logMessage+17e 00d38b60 00d38acc 00250168
RtCppUtils!RTCppLog::log+dd 00000001 00d38b60 00d38ec8
RtCppUtils!RTCppLogPtr::log+1e 00000001 00d38b60 01a3a5d0
RtCppUtils!RTCppLog::RTCppLog+1f1 c0000005 00d38ff0 e06d7363
msvcrt!_CallSETranslator+97 00d3920c 00d39600 00d39228
msvcrt!__CxxExceptionFilter+1d1 00d3920c 00d39600 00d39228
msvcrt!__CxxExceptionFilter+58f 00d3920c 00d39600 00d39228
msvcrt!__InternalCxxFrameHandler+b2 00d3920c 00d39600 00d39228
msvcrt!__CxxFrameHandler+28 00d3920c 00d39600 00d39228
ntdll!ExecuteHandler2+26 00d3920c 00d39600 00d39228
ntdll!ExecuteHandler+24

and so on..............

re: HOWTO: Understand and Diagnose an Application Pool Crash

David.Wang — Tue, 28 Mar 2006 16:53:52 GMT

Sudeep - looks like something is in infinite recursion. In which case it is less interesting that ntdll.dll crashed (when you see ntdll.dll "crash" you should always think "what did I do wrong to cause it" and not "it's Microsoft's problem that ntdll.dll crashed").

Since you are using VB, you must make sure to get the latest VB Runtime and patches because otherwise VB doesn't work properly when launched by IIS6 on Windows Server 2003.

Debugging Principles hasn't changed on Windows Server 2003. Procedures may have changed since process models change, but that's a fact of life. The Debugging Principles haven't changed since NT 3.51.

//David

re: HOWTO: Understand and Diagnose an Application Pool Crash

Sudeep — Wed, 29 Mar 2006 09:04:36 GMT

Absolutely i agree with you david, I bet there is a problem with one of the dlls we are using... i have run debug diagnostics tool a lot of times, what i get to see is there is a first chance exception occuring in one of the dlls used by us everytime we access the page which uses that dll...

does a first chance exception handled properly by the application also get notified by the debugger? or does it mean there is definitely an unhandled exception in our vb dll? also is there a chance that many first chance exceptions eventually result in a crash ? or we can be sure that first chance exceptions never crash IIS?

Should i treat exception occuring due to vb dll (first chance exception) and the other exception which is crashing ntdll (second chance exception) as seperate exceptions or is there a chance they are interlinked?

re: HOWTO: Understand and Diagnose an Application Pool Crash

Sudeep — Thu, 30 Mar 2006 11:17:41 GMT

Also David, I have the latest sp which is sp6 for visual studio 6.0 and latest vb runtime installed on my machine and still facing the problem of not being able to debug vb com dll and also i have sp1 for my server 2003 os which is the latest.

Are you referring to any specific patch that will do the job? do you have the url to the patch you are referring to?

re: HOWTO: Understand and Diagnose an Application Pool Crash

David.Wang — Thu, 30 Mar 2006 14:02:00 GMT

Sudeep - I would not speculate about any of the exceptions seen in the debugger other than the unhandled one which will cause the debugger to break in. Exceptions are a normal part of computing, and handled ones are usually ok (at least, the code handling it thinks it is ok). For example, when you try/catch an exception you will see it show up as first-chance - nothing wrong with that.

So, you just need to care about the unhandled exceptions, and if you are paranoid, any handled exceptions that are "swallowing up" otherwise unhandled exceptions and preventing early diagnosis. But, there is little you can do about that one.

As for the exceptions - you have no way to determine if they are linked or not. You can only debug the crash, determine what criteria caused that crash, and go backwards from there.

Second-chance exception means that someone already caught the first-chance exception but did not handle it - which eventually leads to second chance.

I do not know if Visual Studio 6.0 works on Windows Server 2003. It is certainly no longer a supported product.

I have never debugged Visual Basic. I just know that the VB Runtime had some hacks in it to turn on RIM and UE for badly written VB COM components that would otherwise hang a server like IIS. The hack expectedly breaks on IIS6 because of the new process model (process name is w3wp.exe and not inetinfo.exe or dllhost.exe), so it had to be re-hacked to work with IIS6. I don't know if you have that version or not, and I don't know where you'de get it. Sorry - I just don't work with VB much.

//David

re: HOWTO: Understand and Diagnose an Application Pool Crash

TJ — Thu, 30 Mar 2006 19:09:20 GMT

Great article. I attached the debugger and was able to generate a dump from the URL that was experiencing the hang - however, I am unable to open the dump:

IISAnalysis.asp Could not open specified dump file.

Any thoughts?

Thoughts on IIS Memory Recycling for 3rd party Applications

David Wang — Tue, 11 Apr 2006 12:40:44 GMT

Sigh... it seems that the Application Health Monitoring features added in IIS6 are merely used by VARs...

re: HOWTO: Understand and Diagnose an Application Pool Crash

Matthew — Fri, 14 Apr 2006 17:25:23 GMT

What should I do if DebugDiag crashes everytime I try to install it??? :>

No really, that is happening to me. The only answer at this point is to format c:

re: HOWTO: Understand and Diagnose an Application Pool Crash

David.Wang — Fri, 14 Apr 2006 22:17:07 GMT

Matthew - are you sure that DebugDiag is actually crashing when you install it?

The installer does not invoke DebugDiag binaries in any way, so I do not see how it can crash, and installer programs themselves rarely crash.

Since the installer is an MSI, you can always launch it as:
msiexec.exe /I iisdiag.msi /L*v iisdiagInstall.log

And view the verbose installation log file iisdiagInstall.log afterwards to see where the install failure happened.

Sure, you can always format c: . Debugging and Troubleshooting is not for everyone, and if you can rebuild server state faster than you can troubleshoot, debugging is irrelevant.

//David

HOWTO: Understand and Diagnose an Application Pool Crash

dudu — Sun, 07 May 2006 06:22:30 GMT

TrackBack From:http://dudu.cnblogs.com/archive/2006/05/07/393073.html

re: HOWTO: Understand and Diagnose an Application Pool Crash

gavin milward — Fri, 12 May 2006 11:41:30 GMT

it this similar situations when the network service does not have necessary permissions. I use regmon from sysinternals to monitor registry access.

filter it for the w3wp.exe service and highlight access deinied errors

when you get access denied from the network services, this usually happens with access to certificates. open the registry key and add read permissions for the network service account.

then once all accessd enieds are found schedule an iis restart to pick up the new permissions.

This fixed all my similar problems but just recently an update patch reset all my permissions for certificates, so I am yet to decide whether the network service is the best to use.

re: HOWTO: Understand and Diagnose an Application Pool Crash

David.Wang — Sun, 14 May 2006 10:36:50 GMT

gavin - I think your problem comes from some application running on IIS that is accessing certificates - because IIS itself (and the process identity, Network Service by default) does not have any need to access certificates.

//David

Understand and Diagnose an Application Pool Crash

IIS 6 and Application pool — Mon, 15 May 2006 20:37:34 GMT

http://blogs.msdn.com/david.wang/archive/2005/08/29/HOWTO_Understand_and_Diagnose_an_AppPool_Crash.asp...

re: HOWTO: Understand and Diagnose an Application Pool Crash

tyrven — Thu, 25 May 2006 01:12:49 GMT

Unfortunately, IIS Diag requires the Windows 2003 DDK which is no longer available for public download, but now requires a MSDN subscription (or you have to buy the CD and wait for it to arrive). Annoying.

Anatomy of an Application Pool Crash

David Wang — Thu, 22 Jun 2006 15:00:38 GMT

Question:
Ok, I read through David Wang's Troubleshooting crashes thing and got the DebugDiag and I...

re: HOWTO: Understand and Diagnose an Application Pool Crash

Oded — Thu, 20 Jul 2006 10:35:36 GMT

We have experienced the same problem:
Event Type: Error
Event Source: W3SVC
Event Category: None
Event ID: 1002
Date: 7/19/2006
Time: 3:15:55 AM
User: N/A
Computer: xxx
Description:
Application pool 'DefaultAppPool' is being automatically disabled due to a series of failures in the process(es) serving that application pool.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Event Type: Warning
Event Source: W3SVC
Event Category: None
Event ID: 1011
Date: 7/19/2006
Time: 3:15:55 AM
User: N/A
Computer: xxx
Description:
A process serving application pool 'DefaultAppPool' suffered a fatal communication error with the World Wide Web Publishing Service. The process id was '1700'. The data field contains the error number.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:

After a thourough investigation, we have realizes that it was the result of DDoS (distributed denial of service) attacks, which where launched on our site every now and then.
No solution yet...

re: HOWTO: Understand and Diagnose an Application Pool Crash

John Smith — Tue, 12 Sep 2006 00:21:07 GMT

I am very sorry to say this, David. But this is one of the worst technical article I have ever read in my life.
You could say these words in one page, but you used six, which makes this article just a waste of time.

re: HOWTO: Understand and Diagnose an Application Pool Crash

Ruth Horvath — Wed, 20 Sep 2006 01:13:16 GMT

I just found this thread extremely interesting. We too have an application running on Windows Server 2003 SP1 using IIS 6. I've been trying to run load on our application. At first, I saw that the w3wp.exe process memory usage escalated, even if we only opened 50 windows running the application and closed them, the memory usage still climed. Then I found out about a "known" memory leack and "known" null pointer exception. Oh well...
Now I'm trying to run load again (that I know had worked a few weeks back) and things are worse. I used to be able to login 1000 users to the application and run for 10 hours. Now I don't even last 20-30 minutes and only get in 200-400 users. In looking at the events, I see the Faulting application w3wp.exe....., The VB Application identified by the event source logged this Application CPU_Mon.... there is some other stuff, but then the whole server reboots. I've just downloaded the DebugDiag tool and set up rules for hangs and crashes - but it doesn't look like I have that right; I do have a w3wp...txt file created, but inside there's an error about the Symbol file could not be found and the stack unwind information not available.... HELP !

re: HOWTO: Understand and Diagnose an Application Pool Crash

Krish — Thu, 28 Sep 2006 11:51:03 GMT

Sorry David this is not for you.
Few words for John Smith,
John, Genius like you can understand things in few words, but people like us need some explanation. So If you can't encourage forums like this please don't discourage. I would have appreciated if you could have posted that one page of your's here.

Also I don't want to write something like this here, but I don't want to see another john discouraging like this.

David please continue your great work.

re: HOWTO: Understand and Diagnose an Application Pool Crash

rightioho — Wed, 04 Oct 2006 04:06:19 GMT

so... i get the same issue with MICROSOFT sharepoint portal services,... running on MICROSOFT windows operating system,... and web service provided by MICROSOFT IIS 6.0,... supplemented with MICROSOFT .NET 1.1,... and MICROSOFT COM+ services... there seems to be a common occurence here...

re: HOWTO: Understand and Diagnose an Application Pool Crash

Yatish — Mon, 09 Apr 2007 14:46:02 GMT

This info helped me alot , while resolving some sticky issues.I am working as tech support engineer and whnever I got any windows issue I prefer this site and mostly your blogs.

re: HOWTO: Understand and Diagnose an Application Pool Crash

Ayoob — Wed, 06 Jun 2007 15:21:43 GMT

Excellent post David, straight to the point it helped a lot, troubleshooting and debugging waaay more fun than just apply random fixes, format ;)

re: HOWTO: Understand and Diagnose an Application Pool Crash

Manpreet — Tue, 26 Jun 2007 07:35:42 GMT

Is Installing DebugDiag on server going to be having any performance issues when i use the debugger with the application? how much is the setback going to be?

re: HOWTO: Understand and Diagnose an Application Pool Crash

amit — Fri, 29 Jun 2007 22:25:06 GMT

And what is your suggestion when devs dont have access to production servers to install debuggers? When production servers are handled by a "production support" team who are clueless about these things but only know to guard the servers against any modifications, with their life. So that they don't have to do any extra work. With the UAT server in my control, I don't see the problem , because its not loaded properly or because the combination of events that causes a crash in the real world doesn't occur.

Is it too much work to add a dump capability so IIS can WRITE A CRASH LOG when it crashes??

re: HOWTO: Understand and Diagnose an Application Pool Crash

David.Wang — Mon, 02 Jul 2007 02:43:08 GMT

Manpreet - there is a (usually small) performance degredation, but it varies depending on how much debug output the application produces.

It is customary practice to not run debuggers on production machines, so it should only be set up to help debug your issue and then removed.

//David

re: HOWTO: Understand and Diagnose an Application Pool Crash

David.Wang — Mon, 02 Jul 2007 03:04:36 GMT

amit - my suggestion would be to reproduce the problem on a controlled test-server. If you cannot reproduce the issue in test, then that illustrates an inadequacy in your testing procedures.

Since NT4, Windows has always had Dr. Watson, which will write a crash dump for any process, including IIS, when it exits due to an unhandled exception.

Thus, IIS will never add capability to privately capture and write a crash log when code inside of it crashes. It is a web server, not a development/debugging platform. It simply does not make sense to duplicate any of the existing debugging infrastructure that the user merely needs to configure to use.

It sounds like your frustration is more with the production support team, not with IIS nor debugging facilities.

However, relying on this mechanism assumes that the issues of interest are only crashes (unhandled exceptions) and that the issue survises until JIT debugger activates Dr. Watson.

If you need to debug something else, then you need to attach a debugger and wait for the right exception.

//David

re: HOWTO: Understand and Diagnose an Application Pool Crash

Sam — Sat, 14 Jul 2007 04:14:24 GMT

OMG, when will you people learn! Why not get a "real" webserver so you don't have to go through this agony. For Christs sakes, "Attach a debugger and wait" because that's all you can do??? Screw IIS!!

re: HOWTO: Understand and Diagnose an Application Pool Crash

Blitzenn — Mon, 16 Jul 2007 21:20:42 GMT

Sam, that's like say you should stop being stupid and get a push mower because the tire is flat on your rider.

Can your Web server automatically dump me to a standby data server when it recognizes the first one is not available and determine that both are synchronized across 60 + databases all in real time and do all of that out of the box (no code required)? I think not. If you want to bash things, please research your argument prior to spouting off.

re: HOWTO: Understand and Diagnose an Application Pool Crash

Jack — Thu, 19 Jul 2007 16:23:53 GMT

David's article is not right to the point, he is trying to by pass the issue and not knowing how to fix it, it's a wast of time reading his article!

re: HOWTO: Understand and Diagnose an Application Pool Crash

David.Wang — Sat, 21 Jul 2007 14:09:46 GMT

Jack - you are absolutely correct. This article is not about how to fix a crash. It is about understanding and diagnosing a crash so that one can assign responsibility to get that crash fixed.

If you are looking for ways to fix the crash, the answer is simple -- identify the cause of the crash, which is usually either user configuration error or software bug, and either get the right configuration setting or get the source code/compiled binary for the bug fix.

There are many causes for a crash, so it is quite useless to search for how to fix a crash. You need to first identify the cause of the crash to narrow down the search for a fix. That is exactly what this blog entry tries to teach.

Most users just want their issue fixed but unwilling to learn how to fix their problems. It is the proverbial "give a man a fish, he avoids hunger that night. Teach a man to fish, he is never hungry again". I'm sorry, but I am only teaching how to fish. You will have to compensate me to give you a fish.

//David

re: HOWTO: Understand and Diagnose an Application Pool Crash

Rich — Wed, 22 Aug 2007 14:45:00 GMT

Hi David,

Just wanted to say thanks very much for this article, it was just what I needed to track down and fix a bug with a 3rd party app on a clients server and I am sure the methodology is someting that I will use again and again. Have passed this link onto all our engineers.

Many thanks

Rich

re: HOWTO: Understand and Diagnose an Application Pool Crash

Brian Milinazzo — Wed, 29 Aug 2007 20:54:53 GMT

This seems like the logical approach to take. I am also getting the application pool error, so I have installed the Debug Diag tool and created a generic crash rule but I am not sure what specific exceptions to list in the advanced settings if any at all is this.

Can someone post a setup that should catch the error and provide the debuging information?

Background:

This error is occuring once every two weeks and requires a complete restart of the server. Restarting IIS and the appliations do not work the website remains unavailable. After the server comes backup we get the application pools errors on the console and the info messages are generated in the event log.

The error log before the restart is:

Faulting application w3wp.exe, version 6.0.3790.1830, faulting module cognosisapi.dll, version 8.2.1640.0, fault address 0x00004cfc.

Info logs is basically the same it just has the "report queued message" at the beginning:

Reporting queued error: faulting application w3wp.exe, version 6.0.3790.1830, faulting module cognosisapi.dll, version 8.2.1640.0, fault address 0x00004cfc.

Thank you,

Brian

re: HOWTO: Understand and Diagnose an Application Pool Crash

Hong — Fri, 16 Nov 2007 03:07:18 GMT

Hi David,

I am also getting the same application pool error on production server which is not really accessible to dev. However, I have a minidumper running in my application dll loaded by w3wp. Previously when the app dll crashed, we got minidump files for debugging. but this time with this "faulting application w3wp.exe..." error, no minidump files has been generated.

So I was wondering if there is any separate IIS process running, in additional to the one running my user app dll, that could cause the Application Pool Crash error. If that's the case, the crashing process is out of the control of my minidumper, as the dumper only sits in my app dll and monitors the "current process" running the app dll.

How many processes are started for IIS in general? Is there only single one, w3wp.exe?

Thanks,

Hong

re: HOWTO: Understand and Diagnose an Application Pool Crash

David.Wang — Sun, 18 Nov 2007 12:01:21 GMT

Hong - IIS6 has two different runtime modes that affect this answer.

In IIS5 Compatibility Mode, user code can be running in inetinfo.exe if Low Isolation, some dllhost.exe if Medium Isolation, and any number of dllhost.exe if High Isolation.

In IIS6 Worker Process Isolation mode, all user code run inside of a w3wp.exe process, and the number of such w3wp.exe processes depend on the number of functional Application Pools as well as each Application Pool's "Maximum Processes" setting.

There are no other IIS processes running user code that can crash w3wp.exe like that. I suspect your minidumper was not activated or functional for the w3wp.exe process that actually crashed.

//David

re: HOWTO: Understand and Diagnose an Application Pool Crash

Pichitchai — Thu, 06 Dec 2007 12:05:29 GMT

Hi David

I want to know

Why this error occur ?

Event Type: Warning

Event Source: W3SVC

Event Category: None

Event ID: 1011

and

Event Type: Warning

Event Source: W3SVC

Event Category: None

Event ID: 1009

It occurs often but IIS6.0 not crash immediate

re: HOWTO: Understand and Diagnose an Application Pool Crash

Praveen — Wed, 12 Dec 2007 16:30:59 GMT

Hi David,

We aree facing the similiar problem posted here.. with the following error message.

A process serving application pool 'AppPool' suffered a fatal communication error with the World Wide Web Publishing Service. The process id was '5448'. The data field contains the error number.

I used DebugDiag tool to debug and analyzed the dump file using the same tool. and I am seeing the Memory Heap corruption errors while connecting to some COM DLLs. (SAS 9.1.3)

In w3wp__PID__4940__Date__12_03_2007__Time_07_23_17AM__422__Second_Chance_Exception_C0000005.dmp the assembly instruction at ntdll!ExpInterlockedPopEntrySListFault in C:\WINNT\system32\ntdll.dll from Microsoft Corporation has caused an access violation exception (0xC0000005) when trying to read from memory location 0x0004000d on thread 52

When I click on thread 52 it takes me to the following.

Thread 52 - System ID 488

Entry point SASWMan!DllUnregisterServer+b95d

Create time 12/3/2007 7:23:15 AM

Time spent in user mode 0 Days 0:0:0.0

Time spent in kernel mode 0 Days 0:0:0.0

Function Arg 1 Arg 2 Arg 3 Source

ntdll!ExpInterlockedPopEntrySListFault 04ff08f8 072ff4fc 7c82a0b8

ntdll!RtlAllocateHeap+14e 04ff08f8 05c8d3e0 0000005c

ntdll!RtlAllocateHeap+e2 04ff0000 00000000 0000005c

SASComb!DllCanUnloadNow+d705 0000005c 1001139a 0000005c

SASComb!DllCanUnloadNow+d727 05c89ab0 066183ec 00000001

SASComb+1d8f 05c89ab0 066183ec 065facac

SASWMan!DllUnregisterServer+89ef 072fff2c 04e74e60 04e74ce0

SASWMan!DllUnregisterServer+7ec0 0607ec44 00000000 00000000

SASWMan!DllUnregisterServer+b9bc 04e74d80 00000000 00000000

kernel32!GetModuleHandleA+df 04e45388 04e74d80 00000000

and the heap corruption is as below

Heap 58 - 0x04ff0000

Heap Name SASComb!DllCanUnloadNow+149924

Heap Description This heap is used by SASComb

Reserved memory 1.06 MBytes

Committed memory 596.00 KBytes (54.78% of reserved)

Uncommitted memory 492.00 KBytes (45.22% of reserved)

Number of heap segments 2 segments

Number of uncommitted ranges 1 range(s)

Size of largest uncommitted range 492.00 KBytes

Calculated heap fragmentation 0.00%

The DLLs listed above SASomb and SASWman are SAS DLLs and I am not sure whether this error is related to SAS or .netframework 2.0 or IIS or...

We already contacted SAS Support and couldn't get much help from them and all SAS DLLs are latest ones which we are using.

We are connecting to SAS on Unix server from .Net 2.0 framework using ASP.Net (using VB.Net) with IIS 6.0 on webserver.

Before migrating to 2.0 (in 1.1) we didn't face this issue, but this error is occuring after every 5 minutes after we converted to 2.0

Could you please help we have a release this weekend..and Thank you in advance.

re: HOWTO: Understand and Diagnose an Application Pool Crash

Praveen — Tue, 25 Dec 2007 18:27:01 GMT

Is this site still active or nobody cares??

re: HOWTO: Understand and Diagnose an Application Pool Crash

David.Wang — Wed, 26 Dec 2007 00:27:41 GMT

Praveen - I believe SAS Support needs to look at this issue because you are seeing memory corruption, which causes the crash.

Now, the memory corruption happens while the SAS code tries to allocate memory. Since it is *highly* unlikely that another piece of code corrupts the exact piece of memory that SAS is about to use, it is *highly* likely that SAS code itself is using bad memory.

This means that SAS support is responsible because it is a logical bug in their code. If you cannot get them to diagnose/support their own code, then you need to rethink using their software.

//David

re: HOWTO: Understand and Diagnose an Application Pool Crash

David.Wang — Wed, 26 Dec 2007 00:32:18 GMT

Praveen - Memory corruption like this can be diagnosed with PageHeap from Application Verifier, supplied by Microsoft.

http://www.microsoft.com/downloads/details.aspx?familyid=bd02c19c-1250-433c-8c1b-2619bd93b3a2&displaylang=en

You can use it to try to track down *when* something corrupts a piece of memory in the SASComb heap because that's the culprit that needs to be fixed.

//David

re: HOWTO: Understand and Diagnose an Application Pool Crash

Andy Brierley — Thu, 27 Dec 2007 17:51:44 GMT

I am having problems with w3wp.exe on a dedicated Win 2003 server - it randomlt shoots up to 99% CPU usage... and, yes, it does have all the latest patches and I can find no way to resolve the issue - restarting IIS sometime 'cures' the problems and on other occasios, it doesn't.

Our site is running good old ASP (not .NET) connecting with one big MS Access DB and several other smaller DBs. I have checked all open connections and all my code checks out - there are no w3wp.exe errors in the App. log but just occasional errors from W3Ctrs "taking too long to refresh the W£SVC counters."

Any thoughts?

Thanks in anticipation

//Andy B

re: HOWTO: Understand and Diagnose an Application Pool Crash

Praveen — Sat, 29 Dec 2007 21:18:32 GMT

Thanks David for your reply,

We'll try to contact SAS again, with your reply,

but I have a question.

Actually we have a wrapper class to connect to SAS, before we migrated to 2.0 framework, there used to be lot of connections we were making using this class and then closing the connection as soon as we are getting the data...

that is connect..get the data and close the connection..

Now this was very time consuming, and as part of performance enhancements we made, removed all the connections..restored only one connection through out the session and reusing it.

Then we started getting the above mentioned application pool crash..

so now we have a situation here..so is it because..we are using (re-use) the connection through out the session (which sas com bridge dlls might not support) or is it because we converted the code to 2.0..or is it becuase of the combination..

Then as the first step..we put back all the connections..that is no reuse...open get the data and close..each and every time almost..

in .net framework 2.0 code...

still we saw the error (and the app pool crashed)..

then we thought..it's not because..of the connections...

Then we converted the code back to 1.1 framework..with all the connections (that is no re-use)..

...then..then..then..

it looks like..that the error started disappearing now...but still testing..to see..if the app pool crashes..but mostly..it's not crashing..

so it looks like the combination of 1.1 framework and putting back the connections..that worked...but I don't know why it is so..

David..could you please share your thoughts on this??...because...we need to convert to 2.0 and we need better performance..both (or either) of them..are not possible..with the current scenario.

Thanks

Praveen.

re: HOWTO: Understand and Diagnose an Application Pool Crash

David.Wang — Mon, 31 Dec 2007 11:54:05 GMT

Andy B - clearly the problem is not with IIS code, no matter how your code checks out, because IIS does not have sporadic performance issues.

Perhaps your ASP code has problems with MTA (default change from prior IIS versions) and needs to switch to STA. Several other people resolved their CPU spikes with this change.

Another culprit is your Access DB because MDBs are notoriously single-threaded and easily cause contention like the 100% CPU spike.

In other words, I believe these CPU spikes are caused by problems/assumptions within user code, so the problem is not with w3wp.exe but rather user code no matter the process it is in...

//David

re: HOWTO: Understand and Diagnose an Application Pool Crash

David.Wang — Mon, 31 Dec 2007 12:08:23 GMT

Praveen - the only way to understand "why" it works is to understand why it fails.

Personally, I do not think it is a problem with .Net Framework. Maybe .Net Framework 1.1 memory layout masked a bug in the SAS DLL so that it is always exposed in .Net 2.0.

Clearly, that would not be a problem with .Net Framework 2.0, but from your perspective the SAS DLL is not usable under .Net Framework 2.0, thus it seems like an issue with .Net Framework 2.0.

And the only way to distinguish between the possibilities is to diagnose the crash.

From my perspective, the crash is caused by SAS DLL, and maybe .Net Framework 1.1 is your only work-around. I would not expend effort to try to figure out how to get a work-around on .Net Framework 2.0 because that does not resolve the actual issue.

//David

re: HOWTO: Understand and Diagnose an Application Pool Crash

srinivas — Thu, 03 Jan 2008 16:59:53 GMT

we are experiencing issue with app pool on one of the serevr.

Event viewer shows the following errors for system log:

1.A process serving application pool 'RTIAppPool' suffered a fatal communication error with the World Wide Web Publishing Service. The process id was '5668'. The data field contains the error number.

we are getting the same error for different process ids.. then we are getting the following error:

"Application pool 'RTIAppPool' is being automatically disabled due to a series of failures in the process (es) serving that application pool."

and app log shows the following error:

Faulting application w3wp.exe, version 6.0.3790.3959, stamp 45d6968e, faulting module kernel32.dll, version 5.2.3790.4062, stamp 46264680, debug? 0, fault address 0x0000bee7.

re: HOWTO: Understand and Diagnose an Application Pool Crash

David.Wang — Sun, 06 Jan 2008 14:10:13 GMT

srinivas - you are running an unstable application in IIS that is repeatedly crashing and has been disabled.

You should follow the instructions in this blog entry to use the correct tools and methodology to detect and identify the issue to get it fixed.

This is almost certainly a bug in your code or a 3rd party component used by the application serviced by IIS.

//David

re: HOWTO: Understand and Diagnose an Application Pool Crash

Roman — Sun, 06 Jan 2008 14:58:17 GMT

Hi David,

How to debug "w3wp", if "application pool crash" occurs at moment of start "w3wp"?

I was trying to add key in registry

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\W3WP.EXE]

"Debugger"="C:\\Program Files\\Debugging Tools for Windows\\ntsd.exe -server npipe:pipe=w3wp"

but i can't connect to debugger,

thus following messages in "system" event log (order by date and time):

"A process serving application pool 'MyPool' terminated unexpectedly. The process id was '7016'. The process exit code was '0x80004005'."

...

"Application pool 'AppPoolASPNet11' is being automatically disabled due to a series of failures in the process(es) serving that application pool."

Thanks...

re: HOWTO: Understand and Diagnose an Application Pool Crash

Roman — Sun, 06 Jan 2008 15:06:03 GMT

correction: the name of application pool is 'AppPoolASPNet11' everywhere.

Sorry...

re: HOWTO: Understand and Diagnose an Application Pool Crash

David.Wang — Sun, 06 Jan 2008 17:59:07 GMT

Roman - how did you verify that your settings work?

Did you try it on something else like "notepad.exe" instead of "w3wp.exe"?

I am not certain why you need to do this because no application crash happens at the start of w3wp.exe. One can always intervene and attach a debugger prior to any application code execution on IIS.

However, in all cases correct debugger settings are critical, so I suggest establishing that first. You can probably progress to the solution on your own so I would like to give you the satisfaction of doing so, given only hints.

//David

re: HOWTO: Understand and Diagnose an Application Pool Crash

Brian — Mon, 07 Jan 2008 21:32:58 GMT

It seems a shame that to troublshoot IIS errors like this, you have to install a tool to gather more information, hoping that the crash happens again. PLus, some of these tools can create disk quota issues with large dumps. It would be great if more information could be gathered at actual error time.

re: HOWTO: Understand and Diagnose an Application Pool Crash

David.Wang — Tue, 08 Jan 2008 10:49:17 GMT

Brian - Sure, it is possible to gather this information at actual error time using built-in tools to Windows -- that is exactly how we troubleshoot IIS and Windows software in general, and it works great.

However, you would have to endure a 20+% hit in performance. Is that acceptable to you?

The existing tools have low system impact and work for the vast majority of users because that's what they want.

Now, disk quota issues sounds like your problem, not a problem with the tools. You certainly cannot suggest that debugging bypasses quotas... because that would be a security vulnerability.

//David

re: HOWTO: Understand and Diagnose an Application Pool Crash

Nibor — Fri, 22 Feb 2008 13:32:08 GMT

Hi,

I'm running into a weird problem when I try to run IISState.

Some background information first:

A customer has a .NET application connecting to webservices on an IIS 6 webserver. The last few weeks the application randomly goes very, very slow, and we can see w3wp.exe running on around 30-40% cpu usage. After a while (30-40 mins) cpu usage goes down, and the application is "fast" again.

However, the last few days the webserver has gone crazy, and cpu usage has been much higher, up to 100% during the night when no one connects to the application at all, and when they do connect, cpu usage actaully went down to 80-85%. And all of a sudden it drops tp 0% before going all the way up again.

Running an iisreset solved the problem for a while, before it re-occured again. The interesting thing is that after a full restart of the server, the users of the application was all happy, whilst I could see the w3wp.exe still running on 99% cpu usage.

So, I thought I should try out IISState to see what is hogging the cpu. When I ran it, it got to thread 12 or 13, before it stopped, hung, and eventualy restarted w3wp.exe (to my amusement this actually "solved" the problem on the server, since cpu load has been on 1-2% since then).

What is logged is the usual "A process serving application pool 'DefaultAppPool' failed to respond to a ping. The process id was '3356'."

I've tried to download both IISState 3.0 and 3.3.1, and run into the same problem with both.

What the IISState log says is:

Thread ID: 11

System Thread ID: ba0

Kernel Time: 0:0:0.0

User Time: 0:0:0.0

Thread Type: Managed Thread. Possible ASP.Net page or other .Net worker

SaveModuleToMemory failed

Failed to load SOS data.

No valid SOS data table found.

Begin System Thread Information

# ChildEBP RetAddr

00 019dfea8 00000000 ntdll!KiFastSystemCallRet

GetContextState failed, 0x8007001F

Thread ID: 12

System Thread ID: 0

Kernel Time: 0:0:0.0

User Time: 0:0:0.0

GetContextState failed, 0x8007001F

I'm not really sure what to think, since we never had any actual crashes or timeouts on the server until I ran IISState, so it seems that somehow IISState initiates it.

Any ideas?

-Nibor

re: HOWTO: Understand and Diagnose an Application Pool Crash

David.Wang — Fri, 22 Feb 2008 16:41:46 GMT

Nibor - There is no issue with IIS State here.

What is happening is that you used IIS State to break into a live w3wp.exe to take a trace. Meanwhile, IIS is still configured to periodically "ping" the w3wp.exe of its Application Pools. Since IIS State is hogging this w3wp.exe trying to take a trace, the w3wp.exe cannot respond to the ping, thus IIS deemed the w3wp.exe as hung and initiated a recycle of the w3wp.exe as well as log the event log entry.

My guess is that the .NET application is badly written and causes a lot of Garbage Collection, which would cause CPU to spike sporadically for long periods of time. Most people schedule periodic restarts of the .NET Application pools to work around their performance problem, if their design/environment allows it.

//David

re: HOWTO: Understand and Diagnose an Application Pool Crash

Johnathan — Thu, 06 Mar 2008 00:30:38 GMT

It is good to see you are still following this post. I found it a good article. However, I would hope that these tools would come with IIS out of box along with best practices documentation.

I am seeing w3wp.exe crashes under load tests and I am starting to debug them. Is there a ‘common mistakes’ article out there to help with common web services?

Since Server 2008 is about out I wanted to know if this process gets easier with the next version of IIS.

Thanks.

re: HOWTO: Understand and Diagnose an Application Pool Crash

David.Wang — Thu, 06 Mar 2008 14:45:34 GMT

Jonathan - I believe that tools and best practices documentation are living documents that should not be tied to be "out of the box".

Frankly, people use web servers in such diverse ways and environments that any tool or comprehensive documentation quickly goes out of date or will be so large that it is not worthwhile/efficient to maintain.

Most of the tools and techniques are general problem-solving techniques that apply to everything, not just IIS. I highly recommend that you learn the techniques because it helps out everywhere.

By learning how to troubleshoot, not just pattern match against "common mistakes" articles or "best practices" documentation, you will be able to help yourself, and not constantly rely on someone else to provide you the information.

Personally, I think the process gets both easier and harder with the next version of IIS in Windows Server 2008, IIS 7. On the one hand, you have Failed Request Tracing, which combined with IIS logs, Event Logs, and HTTP.SYS logs, is all one needs to troubleshoot most issues on IIS7 without needing a debugger or turn on ETW tracing. On the other hand, IIS7 is so generic and easy to misconfigure that you have tons more ways to accidentally shoot yourself.

I suspect that most people will initially find IIS 7 harder to use/configure because it will be easy to break and misconfigure, but after people get comfortable with IIS7 basics, they will learn to appreciate the improvements and benefits and see how is gets easier.

//David

re: HOWTO: Understand and Diagnose an Application Pool Crash

Allen — Thu, 03 Apr 2008 16:54:38 GMT

Hi David, and thanx for posting such a wonderful article.

Well, my problem is also somewhat similar but couldn't understand or diagnose the problem as such because the site is hosted by discountasp.net. They don't even give me event logs, memory dumps as i asked for it multiple times, reason being told as it is a shared hosting.

Your response would be much appreciated.

Thanks.

re: HOWTO: Understand and Diagnose an Application Pool Crash

David.Wang — Sat, 19 Apr 2008 07:28:38 GMT

Allen - unfortunately, without cooperation from the hoster, it is impossible to troubleshoot.

For all you know, your problems may be caused by another customer since it is shared hosting. But, you should not be able to debug someone else's application to figure it out (security issue), so debugging on a shared hosting environment by the customer for systemic issues is practically impossible.

//David

re: HOWTO: Understand and Diagnose an Application Pool Crash

Carl Cook — Mon, 28 Apr 2008 14:22:45 GMT

Hi David,

Thanks for your replies to this post... you have given a great starting point to help me fix a problem I have been recently assigned.

I am troubleshooting some WCF services run within IIS 6 on a Windows 2003 R2 server. We are getting the app pool crashing out, with the 'typical' messages in the event log ("faulting application w3wp in mscorwks.dll", "communications failure", "process unexpectedly terminated", etc). I ran IISState/WinDbg to produce a stack trace, but this gave me nothing out of the ordinary (all the stacks looked fairly normal, no stack overflows, access exceptions, deadlocks, etc). .Net logging/tracing shows nothing out of the ordinary. Performance monitoring showed thread usage was fine, no memory leaks, etc. Http logs gave nothing out of the ordinary either.

Today, I ended up stepping through one of the WCF services with a debugger (attaching from my dev machine to the live server as a remote process), and the debugger returned a System.ExecutionEngineException from the DataContractSerializer.WriteObject method. This exception is uncatchable... all I can do is continue execution in the debugger... at which point the w2wp.exe process gets terminated. [According to a few recent posts, this might be a CLR issue (i.e. the dynamically generated serializer is not handling nullable fields correctly, then crashing out).]

So, one lesson I have learnt is that stepping through the worker process with a debugger is one of the easiest ways that I have found to find the real fault within IIS.

However, what you would you now recommend as an approach to fix this problem? Approach Microsoft for a potential hotfix? Code around the problematic method? Install .Net 3.5 (on the hope that the problem has been addressed)? Try to disassemble the System.* libraries to find the root cause? Move to Windows 2008 Server with either IIS 7 or WAS, and just hope that this fixes the issue? Or something else?

Many thanks!

-Carl.

re: HOWTO: Understand and Diagnose an Application Pool Crash

Kalpesh Joshi — Thu, 29 May 2008 17:18:15 GMT

I have Win 2008 64bit with iis7. I have developed a 32bit application in .net. I have create an additional apppool named online.

I have select "Enable 32bit Application" as TRUE.

Now whenever i brows my application this apppool has been stopped with searies of failure.

If I set "Enable 32bit Application" as FALSE then i get JET 4.0 Errors.

The event log contains this data

A listener channel for protocol 'http' in worker process '5780' serving application pool 'OnlinePool' reported a listener channel failure. The data field contains the error number.

Event ID 5139 and then stope the apppool with event id 5002.

re: HOWTO: Understand and Diagnose an Application Pool Crash

David.Wang — Fri, 30 May 2008 21:31:15 GMT

Carl - requesting for a hotfix is the best approach if you do not want to change .Net version. And then temporarily code around it.

I would not upgrade .Net version nor OS on the "hope" that it contains the fixes -- if it is fixed in the later version, a hotfix usually exists for the prior version. However, if you do not report it, it has no chance of getting fixed.

//David

re: HOWTO: Understand and Diagnose an Application Pool Crash

Harry McHaffie — Thu, 26 Jun 2008 11:40:47 GMT

David,

Please Help! We opened a case with Microsoft in an effort to resolve our crash problem. WinDbg was used to trace the crash. The specific .dll and function that was causing a stack overflow was discovered. Microsoft asked for the debug version of that .dll. We went to the company that supplies a C API for their product that contains the problem .dll. This company will not provide the debug version of the .dll. Are we at a dead end?

We have Server 2003sp2, SQL Server 2005sp2, ESRI SDE 9.1sp2, VS2008 with a Solution containing a C++ project that makes calls to the "SDE C API" (SDE C API contains the problem DLL), a CSharp Wrapper project that wraps the C++ code and a VB Web Service project that calls the CSharp Wrapper code.

Running the Web Service from within VS2008 everything works great! We call a method which returns the expected data.

We publish the web service to the same machine VS2008 runs the solution successfully on. Run IE, pull up the page to test the Web Service, select the method and provide one parameter, then Invoke the method. It throws "An unhandled win32 exception occured in W3WP.exe". We know from the debug/trace that the offending function is in the sdesqlsrvr91.dll, the name of the function that causes a Stack Overflow is SE_LowerCase. Our code makes no direct calls to this function. The SDE C API functions that we are making calls to within our C++ code are probably making a call to the SE_LowerCase function.

Our event log reflects the following:

A process serving application pool 'DefaultAppPool' suffered a fatal communication error with the World Wide Web Publishing Service. The process id was '3212'. The data field contains the error number.

A process serving application pool 'DefaultAppPool' terminated unexpectedly. The process id was '3440'. The process exit code was '0x0'.

Is there anything else we can do?

re: HOWTO: Understand and Diagnose an Application Pool Crash

David.Wang — Sun, 29 Jun 2008 14:10:48 GMT

Harry - unfortunately, your issue is with the 3rd party software. There is little Microsoft or anyone else can do about it.

It sounds like that software does not work on Windows Server 2003 and IIS6 because applications are launched in different processes with different permissions and filepaths for debug by VS2008 and for-production by IIS6.

You will need to verify that this 3rd party software is supported on Windows Server 2003 and if so, what are its required configuration. For example, this 3rd party product could have code searching up its stack for the process that launched it, and it is confused by finding w3wp.exe or "\\?\" in the filepath.

In short, these would all be bugs in the 3rd party software if they say they support Windows Server 2003.

//David

re: HOWTO: Understand and Diagnose an Application Pool Crash

Dave — Wed, 09 Jul 2008 22:32:06 GMT

Hi David,

This is a great article that really helped me understand how to go about diagnosing an application crash.

I am experiencing an issue where DEP stops one of my worker processes. As a result, our site goes down and our web app along with it. I have this generic entry in the application event log:

Faulting application w3wp.exe, version 6.0.3790.1830, faulting module unknown, version 0.0.0.0, fault address 0x00ef0184.

I have downloaded and installed Debug Diagnostics on the server this issue occurs on and setup a crash rule to perform a full dump when the next crash occurs. My problem is, when I check on the utility the next day, the rule says "complete" and is no longer actively monitoring. I usually have to keep re-activating the rule and checking on it to make sure it is still active. The next time the crash occurred, I checked for a memory dump but the rule was "complete" and not active. Therefore, I did not have a dump to analyze. I do not have extensive knowledge of capturing application crashes, but I do want to learn how to so that I can try to fix this crash.

In the advanced settings section of Debug Diagnostics, I went to the exceptions area and raised the action limit for some action types I added. This was in hopes of being able to keep the crash rule active and capture the crash. However, now I see these errors in my application event log:

Event Type: Error

Event Source: W3CTRS

Event Category: None

Event ID: 2003

Date: 7/9/2008 Time: 1:01:17 PM

User: N/A

Computer: CAICORP-H01-16

Description: It has taken too long to refresh the W3SVC counters, the stale counters are being used instead.

If you can help me understand how to keep a crash rule active in Debug Diagnostics, and also if the above error is something I should worry about, I would greatly appreciate it. I really want to figure out what is causing this application to crash with Debug Diagnostics, but I just need some help with the advanced features.

Thanks!

re: HOWTO: Understand and Diagnose an Application Pool Crash

Matthew Evans — Sat, 12 Jul 2008 10:07:09 GMT

Thanks for the article David.

We are experiencing exactly the (variety) of symptoms described. The system and event log contain a subset of the w3wp, App Pool communication errors (in various combinations) - as you describe.

I suspect some kind of subtle recursion in our latest build.

However the site is essentially unusable, we get this little beauty every 2-3 minutes:

The description for Event ID ( 5000 ) in Source ( .NET Runtime 2.0 Error Reporting ) cannot be found. The local computer may not have the necessary registry information or message DLL files to display messages from a remote computer. You may be able to use the /AUXSOURCE= flag to retrieve this description; see Help and Support for details. The following information is part of the event: clr20r3, w3wp.exe, 6.0.3790.3959, 45d691cc, mscorlib, 2.0.0.0, 461f060e, 40cf, 2d, 5s11pi0dzbt3ygdcdkat1jwohxctyitc, NIL.

Followed by a period where IIS doesn't respond to requests, after which the worker process appears to crash / recycle.

I've rolled back the build as a result.

My question is: since this is only occuring live, with realistic load, how can I obtain a debug dump outside the production environment ?

Any ideas appreciated .

re: HOWTO: Understand and Diagnose an Application Pool Crash

Idowu Ademola — Tue, 29 Jul 2008 00:59:39 GMT

Please Sir, The URL supplied is of the particular app im complaining about. Its most likely you will get the <h1>Server Unavailable</h1> error when u view it. I need help i dont know how to use the described debugger, besides My App runs with asp.net 2.0 so does this apply to me? can i resolve to isolation mode? The App has been running for 6mths prior to this problem!. Pls Any Advice will be appreciated...

Pls Kindly mail me ur reply or send me back a link to this site cos im consulting many sites for help... idmode07@gmail.com

re: HOWTO: Understand and Diagnose an Application Pool Crash

Matthew Evans — Tue, 12 Aug 2008 09:49:12 GMT

Thanks for the article David.

We are experiencing exactly the (variety) of symptoms described. The system and event log contain a subset of the w3wp, App Pool communication errors (in various combinations) - as you describe.

I suspect some kind of subtle recursion in our latest build.

However the site is essentially unusable, we get this little beauty every 2-3 minutes:

Followed by a period where IIS doesn't respond to requests, after which the worker process appears to crash / recycle.

I've rolled back the build as a result.

My question is: since this is only occuring live, with realistic load, how can I obtain a debug dump outside the production environment ?

Any ideas appreciated .

re: HOWTO: Understand and Diagnose an Application Pool Crash

David.Wang — Tue, 12 Aug 2008 13:43:06 GMT

Matthew - if you want to obtain a debug dump outside of production, then you will have to find a way to make it reproduce outside of production... otherwise, you will have to risk taking the debug dump from the production environment to get real information.

Now, I don't have a bullet-proof decision tree for you to follow. If that was possible, then it would be automatable and we wouldn't be having this discussion. What I would suggest is that you need to determine if the issue requires realistic load (i.e. repetition), a certain timing-oriented bug, or is it some logical resource leak which load can easily trip over.

Frankly, without the memory dump, one can only guess at the cause and then try to gather data to prove/disprove the hypothesis.

For example, if you think it is recursion, then you are looking for the culprit request out of all possible requests made in production. You won't need load to reproduce a recursion issue.

If the issue has to do with subtle timing issues, then it can be related to the request load quantity, sequence, etc.

If the issue has to do with resource leakage, then with a request mix you should be able to reproduce it outside of production.

//David

re: HOWTO: Understand and Diagnose an Application Pool Crash

Ian Staines — Tue, 12 Aug 2008 19:33:20 GMT

David,

I am confused about the "pageheap" option of DebugDiag. The help dialogs for DebugDiag suggest that this is a stand alone feature of DebugDiag, but when I try to enable the option I cannot figure out how to catpure exceptions generated by heap problems.

I note that most discusion on heap corruption suggest that you use "Application Verifier" in conjuction with DebugDiag to capture heap problems. If so what is the purpose of the "pageheap" option of DebugDiag?

re: HOWTO: Understand and Diagnose an Application Pool Crash

clinks63 — Tue, 30 Sep 2008 12:59:10 GMT

Hi David,

I have read the blog and some of the comments. Based on your article, it was telling that it crash might came from user code. I was just wondering why this crash happens, for I'm running a code same code for 2-3 times but in the 4th-5th time..the page is reflecting that the session has expired..and in the event viewer the log is displaying "www crashed"..we also did some investigation, and what happens is that at the code, it stops at a random line of code..

re: HOWTO: Understand and Diagnose an Application Pool Crash

David.Wang — Tue, 30 Sep 2008 15:56:13 GMT

clinks63 - Crashes do not happen on random lines of code, unless the bug is something else running inside the same process that is corrupting random memory location (and hence random lines of code seems to crash).

The most likely cause of random memory corruption? More User code. Why? Well, if IIS itself was randomly corrupting memory, that same bug would affect thousands of servers world-wide and be rapidly detected, reported, and fixed.

//David

re: HOWTO: Understand and Diagnose an Application Pool Crash

Brian Milinazzo — Tue, 14 Oct 2008 02:50:33 GMT

If anyone has been receiving this error with Cognos 8 application server, Cognos has provided a patch (hot site):

# 604680 Available report processes flipping between peak number and non peak number every 20 minutes

The detailed document for this hotfix also included a fix for a memory leak happening with AD authentication. They have said this should fix our particular issue and it appears to be resolved as we have applied this hotsite in addition to upgrading to Cognos 8.3 SP2.

Brian

re: HOWTO: Understand and Diagnose an Application Pool Crash

David.Wang — Tue, 14 Oct 2008 16:39:22 GMT

Brian - Thanks for the notice. However, your patch only fixes one specific issue for one specific product. It is possible for another product to crash, and it is possible for the same product to crash in a different way, and while both would look the same from the Event Log, neither would be fixed by the patch you mention.

Thus, I would not recommend telling people to go get patches when they see crashes. I recommend following this blog entry's advice to first diagnose the crash using automated tools, then query the support personel of the responsible module for a resolution to the diagnosed crash.

//David

re: HOWTO: Understand and Diagnose an Application Pool Crash

David — Fri, 05 Dec 2008 09:52:51 GMT

IIS Pools tool from http://www.hoststools.com/ is helpful when investigating app. pools crashes and hangs.

re: HOWTO: Understand and Diagnose an Application Pool Crash

leon — Wed, 10 Dec 2008 19:31:33 GMT

http://forums.webhostautomation.com/showthread.php?t=22049

re: HOWTO: Understand and Diagnose an Application Pool Crash

Wing-Leung — Sat, 13 Dec 2008 03:44:02 GMT

I have been troubleshooting the "The identity of application pool, '___' is invalid", event ID 1021, data field 0x8007052e for a couple of hours. I also saw the security failure audit, event ID 529, reason "Unknown user name or bad password". So, this led me to change the password of the application pool identity. I also changed the Windows user's password to match (We use a local, restricted, account for the application pool). I also stopped and restarted the application pool. However, this did not resolve the issue. However, I eventually figured out that I had to do a full IIS restart and that did the trick.

What I don't understand is why I need to do the IIS restart. When I restarted the application pool, I saw the w3wp.exe process stop so I figured a new w3wp.exe process should pick up the new credentials. For some reason it appears that IIS keeps some kind of credentials cached???

I even tested this further by getting my site working (ie: password is now correct). Then I deliberately changed the user's password to something else. However, my site still loaded and application pool was still okay even after a stop and restart of the application pool. It is not until after I did a full IIS restart that I got the identity is invalid error as expected.

I am missing something here regarding user accounts and password changes. Can one of the gurus here explain? Also, is there a way to do this without a full IIS restart (since it would cause other sites to be temporarily unavailable).

Thank you

re: HOWTO: Understand and Diagnose an Application Pool Crash

ArntK — Tue, 16 Dec 2008 03:23:54 GMT

Hi David,

We have an ongoing case with Microsoft support about this issue. We have had the System events 1011/1009 several times through the last 6 weeeks. MS has not been able to get any dump files, how is this possible? Please share any insight you have about this.

Thank you.

re: HOWTO: Understand and Diagnose an Application Pool Crash

David.Wang — Tue, 23 Dec 2008 11:21:13 GMT

Wing-Leung - Full IIS restart is required because WAS has the application pool identities "cached" for use to [re]start an Application Pool.

There is no change-notification within Windows for IIS to know whenever a user property has changed, and it is undesirable for IIS to constantly re-login the user account to start up an Application Pool (imagine if the Application Pool Identity is a domain user -- this behavior would potentially allow anonymous user to use IIS to DoS the Domain Controller under certain user configurations).

Thus, the application pool identity is cached, and it can cause issue precisely when you alter user properties. However, that is the exceptional case, not the common case - user password probably changes every 90 days to never, while recycling application pool likely happens many times during 90 days. Given the lack of change-notification from Windows for IIS to know whenever a user property has changed, the ocassional issue with exceptional case is worth the security and performance benefit of the common case.

//David

re: HOWTO: Understand and Diagnose an Application Pool Crash

Pratik — Thu, 15 Jan 2009 12:35:00 GMT

I got error in our asp.net application please give me a solution.

EventType clr20r3, P1 w3wp.exe, P2 6.0.3790.3959, P3 45d6968e, P4 mscorlib, P5 2.0.0.0, P6 471ebc5b, P7 11d9, P8 18, P9 system.outofmemoryexception, P10 NIL

re: HOWTO: Understand and Diagnose an Application Pool Crash

David.Wang — Sat, 17 Jan 2009 06:57:29 GMT

Pratik - Your application is either using too much memory, or your web server has insufficient memory for your application.

Determine how much memory your application needs. If you see the OutOfMemoryException without using that much memory, then the problem is with your ASP.Net application. Get a memory dump and walk through it using SOS.

Basically, the problem is caused by your application, and you have to fix it.

//David

re: HOWTO: Understand and Diagnose an Application Pool Crash

Henri Visser — Fri, 13 Feb 2009 14:08:39 GMT

I'm getting the same error in an application I wrote: Faulting application w3wp.exe, version 6.0.3790.3959, stamp 45d6968e, faulting module kernel32.dll, version 5.2.3790.4062, stamp 46264680, debug? 0, fault address 0x0000bee7.

Only on Windows 2003, not XP or 2000. It happens when using the transform method to transform a XML document to an HTML document:

Dim xmlout As New StringWriter()

Dim transform As New Xsl.XslCompiledTransform transform.Load(XSLTFileName)

transform.Transform(doc, args, xmlout)

Ive tried running in different app_pools, admin user account, but still no luck

Any Ideas?

re: HOWTO: Understand and Diagnose an Application Pool Crash

Henri Visser — Fri, 13 Feb 2009 17:22:16 GMT

Found the issue after using debugdiag and realizing its a memory leak:

http://social.msdn.microsoft.com/Forums/en-US/xmlandnetfx/thread/7fa6bbc4-c115-454f-af8e-0740b09d3b3a/

re: HOWTO: Understand and Diagnose an Application Pool Crash

Richard Mathis — Tue, 24 Feb 2009 21:33:57 GMT

This is a great post, but I'm having trouble figuring out how to debug IIS 7.0. Many tools have been recommended for prior versions of IIS on google, but IIS 7.0 doesn't seem to be well described. Any recommendation on the best way to debug a crashing IIS 7.0 application pool?

re: HOWTO: Understand and Diagnose an Application Pool Crash

ekreger — Thu, 26 Feb 2009 22:41:56 GMT

Hello. I have run debug diagnostics and am a little confused by its output.

[2/25/2009 2:02:23 PM] Thread created. New thread system id - 4728

[2/25/2009 2:02:23 PM] Thread created. New thread system id - 4016

[2/25/2009 2:02:23 PM] First chance exception - 0xc0000005 caused by thread with system id 2448

This is just a sample but from what I understand, First Chance Excpetions are not necessarily a problem. However.... Should I be seeing this many so often on a site that is not being used hardly at all?

re: HOWTO: Understand and Diagnose an Application Pool Crash

Mauro — Mon, 09 Mar 2009 13:58:28 GMT

Hi David,

in my application I receive sometimes this error.

Could you please have a look on it?

Faulting application w3wp.exe, version 6.0.3790.3959, stamp 45d6968e, faulting module kernel32.dll, version 5.2.3790.4062, stamp 462646fb, debug? 0, fault address 0x0000bee7.

re: HOWTO: Understand and Diagnose an Application Pool Crash

David.Wang — Tue, 10 Mar 2009 04:18:29 GMT

Richard Mathis - the same approach works for IIS7. In fact, the same troubleshooting approach works for all versions of IIS for the past 15 years.

IIS7 just comes with new features like Failed Request Tracing which allows you to troubleshoot non-crashes, but for crashes/hangs, you still apply the same described approach because it is how you debug/troubleshoot issues in general on Windows.

//David

re: HOWTO: Understand and Diagnose an Application Pool Crash

David.Wang — Tue, 10 Mar 2009 04:20:35 GMT

Mauro - You will need to use a tool like IIS State or DebugDiag to capture a stack backtrace of the crash and diagnose your issue.

It is unlikely for w3wp.exe (IIS code) or kernel32.dll (Windows code) to be the issue causing the crash, so you need to use the stack backtrace to see what other code is using the IIS/Windows code to better determine the real issue.

//David

re: HOWTO: Understand and Diagnose an Application Pool Crash

David.Wang — Tue, 10 Mar 2009 04:23:51 GMT

ekreger - It is not possible to determine correctness of exceptions without knowing the code that is being executed.

For example, First Chance Exception can result from managed Exceptions being thrown and caught. Which may be expected. Or not expected. Totally depends on the application and its architecture.

//David

re: HOWTO: Understand and Diagnose an Application Pool Crash

Marc Novak — Wed, 15 Apr 2009 13:34:42 GMT

Excellent article David.

IIS State helped me track down (precisely) what was causing our SMTP / Inetinfo service to fail.

In my case an application we had been using for many years called Policy Patrol (which hooks into the SMTP service) had issues. As detailed below. We have now removed Policy Patrol (with a view to re-installing at some point in the future)

One thing worth noting is. When you terminate IIS State it also terminates the InetInfo.exe it’s monitoring. On that basis IIS State should only be used with that in mind.

Thanks

Marc

IIS State output……….

Opened log file 'D:\iisstate\output\IISState-5628.log'

***********************

Starting new log output

IISState version 3.3.1

Wed Apr 15 10:34:41 2009

OS = Windows 2003 Server

Executable: inetinfo.exe

PID = 5628

Note: Thread times are formatted as HH:MM:SS.ms

***********************

IIS has crashed...

Beginning Analysis

DLL (!FunctionName) that failed: ntdll!RtlAllocateHeap

Thread ID: 21

System Thread ID: 1b88

Kernel Time: 0:0:0.78

User Time: 0:0:0.62

*** WARNING: Unable to verify checksum for D:\Program Files\Red Earth Software\Policy Patrol Email 4\MailProcessor\PP4_SMTPSink.DLL

*** ERROR: Symbol file could not be found. Defaulted to export symbols for D:\Program Files\Red Earth Software\Policy Patrol Email 4\MailProcessor\PP4_SMTPSink.DLL -

re: HOWTO: Understand and Diagnose an Application Pool Crash

David.Wang — Tue, 21 Apr 2009 09:51:31 GMT

Marc - failures during memory allocation (like ntdll!RtlAllocateHeap) indicate something in the process is corrupting memory that it does not own.

It *may* be possible that PP4_SMTPSink.dll is a victim, but if you remove it and the crashes go away, it is a good bet that it was the one corrupting memory, in which case you will not be able to re-install it in the future without getting a new build which fixes the memory corruption.

To help detect memory corruption, you want to use Application Verifier with full pageheap checking enabled.

http://www.microsoft.com/downloads/details.aspx?FamilyID=C4A25AB9-649D-4A1B-B4A7-C9D8B095DF18&displaylang=en

Basically, heap checks work by allocating memory on page-boundaries, so if you overstep your memory boundary, you end up touching unallocated memory and Windows immediately halts the process with an access violation -- instead of the usual packed memory allocation scheme where if you overstep your memory boundary, you end up stomping on some other memory allocation nearby and corrupt that memory.

Of course, such memory checking technique impacts available memory (allocating on page boundaries wastes lots of space in a memory page) and also hooks Application Verifier into that process for all memory allocation/deallocation, but it is also the surest and easiest way to track down memory corruption.

//David

re: HOWTO: Understand and Diagnose an Application Pool Crash

Rakesh Jain — Mon, 18 May 2009 12:10:31 GMT

Hi David

Very useful article, i appreciate your in depth knowledge on the subject.

I have a similar problem, My application get crashed and when i see the event log i see a warning message .

"W3SVC - A process serving application pool 'DefaultAppPool' suffered a fatal communication error with the World Wide Web Publishing Service. The process id was '5236'. The data field contains the error number. "

and my httperr.log says

2009-05-15 09:38:13 192.168.15.104 1339 192.168.1.49 80 HTTP/1.1 GET /MYCUBE-/Web/COOL/ENG/Online/FCISVSubscription.aspx?TXNNO=0220091350001209&MODE=VIEW&AUTHSTAT=U&LimitOrder=&LimitOrderType=&hdnTxnModified=0&MODID=OCMCFOCPT&AUTH=Y&rnd=0.5925906497712439&TITLE=Subscription - 1 Connection_Abandoned_By_AppPool DefaultAppPool

I realised from your articles that i could the problem with my application and i tried to trace the error using the Debug Diagnostic Tool v1.1.

I has generated the dump files on crash on when i did the analysis , i found nothings, it reads " there is no problem detected in dump"

I am really not sure how to approach this problem further, as log did not detect any problem but my application is getting crashed everyday.

Application : its built on asp.net 1.1 platform (it has got migrated asp code also)

db :oracle

sever: windows 2003 sp 1

Please help me out.

re: HOWTO: Understand and Diagnose an Application Pool Crash

Rakesh Jain — Wed, 20 May 2009 12:17:21 GMT

Hi David

Debug Diagnostic Tool v1.1 tools seems to found an answer, it has generated the dmp and when i did the analysis, it shows the error as

In w3wp__PID__3112__Date__05_20_2009__Time_10_12_29AM__750__First chance exception 0XC0000005.dmp the assembly instruction at kernel32!RaiseException+53 in C:\WINDOWS\system32\kernel32.dll from Microsoft Corporation has caused an access violation exception (0xC0000005) when trying to read from memory location 0x00000000 on thread 15

Not sure how to approach this problem.

Any suggestion

re: HOWTO: Understand and Diagnose an Application Pool Crash

sohbeti — Thu, 02 Jul 2009 03:28:39 GMT

kameralı sesli sohbet siteleri

re: HOWTO: Understand and Diagnose an Application Pool Crash

KrisRaam — Fri, 11 Sep 2009 17:52:34 GMT

Hi David, I have read your post which is simple and good.

I have a persisting problem on the IIS 6.0 crash. I found out i web service is causing more CPU usuage. But the same copy of the webservice in another W2k3 server where there's not any problem.

Event Type: Warning

Event Source: W3SVC

Event Category: None

Event ID: 1011

Date: 11/09/2009

Time: 13:28:54

User: N/A

Computer: KRIS

Description:

A process serving application pool 'NewAppPool' suffered a fatal communication error with the World Wide Web Publishing Service. The process id was '11280'. The data field contains the error number.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Pls help me out!!!

re: HOWTO: Understand and Diagnose an Application Pool Crash

Vikram — Thu, 12 Nov 2009 12:56:14 GMT

Hi David,

my web application (developed in ASP.NET 3.5) is not running on IIS 6.0 (OS: Windows Server 2003 SP2) it is displaying "Page Cannot be displayed" & when i checked the httperr.log file i found "Connection_Abandoned_By_AppPool DefaultAppPool".The worker process is getting crashed...

Everything was working fine until i installed some windows security updates.I guess the problem is with the updates.

But how should i fix it to work?