If broken it is, fix it you should

Using the powers of the debugger to solve the problems of the world - and a bag of chips    by Tess Ferrandez, ASP.NET Escalation Engineer (Microsoft)

What on earth caused my process to crash?

What on earth caused my process to crash?

Rate This
  • Comments 10


So you got a 1000 w3wp.exe stopped unexpectedly in the eventviewer or your process just exited in some weird undefined way and you don't know why.

When a process crashes or exits a special event will be fired called EPR (Exit PRocess) so with a debugger like windbg.exe we can attach to the process, wait for epr to be thrown and take a memory dump. When you install the debugging tools for windows you get a vbs script called adplus (http://support.microsoft.com/default.aspx?scid=kb;en-us;286350) that will do this automatically for you, and print logs for most exceptions that occurr during the process life.

Debugging tip!: When you open a dump taken in -crash mode, you will automatically be positioned on the thread that was active when the crash occurred (the most likely suspect). If you switch threads and want to get back to the faulting thread, type ~ to list all threads and the faulting thread will be marked with a dot.

If your dump only shows one active thread in the process and that thread is the main thread, the process was likely killed by something external (health monitoring, low system memory, iisreset etc.)

I'll go into more depth of some scenarios in some later posts but I'm a very top-down person so I wanted to start off with a post about some common situations when you will see a managed process exit so that you have some idea of what to look for in the dump.

In no particular order, here are some of the ones we see most in support:

Stack Overflow Exceptions

A stack overflow exception will occur when the memory allocated for the stack for a thread is used up. By default it's 1 MB so your call stack can be pretty deep, so most of the time when this happens it's because of infinite recursion, i.e.  FunctionA calls FunctionB which calls FunctionB again which calls FunctionB again... and there is no stop condition.

A bit of an obscure infinite recursion situtaion that unfortunately is pretty common is incorrect usage of the Exception Handling application block. Imagine this scenario:  Your application gets an exception, the exception handler kicks in and you have set it up to log to a file. While logging you get an exception of some kind (say access denied) and you have set the exception handler to handle this exception too.  In this case you will end up in an infinite recursive loop where you handle an exception, throw another, handle it, throw another... you get the gist.  The moral of the story?  Don't use the exception handler to handle exceptions in the exception handler:)

If you run "kb 2000" (to see the native stack) and "!clrstack" (from sos.dll to see the managed stack) you can spot the recursive pattern to track down where/why the recursion occurred.

Out Of Memory Exceptions

Most of the time when an Out Of Memory exception occurrs it is caused by a design problem, where either too much memory is stored in cache or session scope. Caching can be great for increasing performance if used in the right way, i.e. you cache the data that will be accessed the most and you don't cache it for longer than you need. In old ASP you got problems if you stored objects in session scope and believe me, it was a blessing in disguise because developers stored only the most necessary items in session scope. Storing large datasets in session scope for example is often counter-productive as you reduce the number of concurrent users your web site can handle and when memory gets high enough you may have more overhead in terms of time needed to do garbage collections and searches through cache than you would requesting the data from the database when you actually need it.

There is no one-size fits all solution here on when you should store something in session/cache and when you shouldn't. The best thing to do is at an early stage, determine how many users your application needs to be able to handle, and based on that determine how much you can allow for storage per user. And then stress-test with more than your maximum number of users to make sure you can cope.  Preferably, stress test with and without the objects in session state to see what the perf. difference is for different amounts of users.

Memory issues are very hard to fix when you are in production since they often require a lot of re-design, so a penny spent in the early stages will save a lot of pennies later.

Debugging tip!: Run !dumpheap -type System.Web.Caching.Cache to get the cache roots and then do !objsize on the addresses for these to find out how much you are storing in cache for the different applications.  (Note: InProc Session state is stored in the cache as well)

For a more indepth discussion on why you get Out of Memory exceptions see my earlier post.

Unhandled exception in COM Component

If your application calls a native COM component, you can crash if you get an unhandled exception in it.  For example if you are referencing memory that has already been freed or similar.

kb 2000 will then show the COM component on the stack so you can narrow it down from there.

Native heap corruption

This, along with GC holes are some of the nastiest issues to troubleshoot. A native heap corruption when someone writes to a location that it was not supposed to. The problem with this is that you will not see the error when the code that writes to the wrong address executes but rather when someone else is trying to access that memory address in the correct way. In other words, long after the "thief" was there.  The location that is written to can be the heap or even worse, a location where code is stored (so instructions are overwritten) or the stack so your code calls into the middle of nowhere.  Most frequently this occurrs when you write outside of the bounds of a buffer or something to that effect.

For an introduction to heap corruption read Geoff Gray's article on Heap Corruption. When you get a crash because of heap corruption, the faulting stack will often be in a heap allocation function in ntdll, and to resolve it you need to run with GFlags or PageHeap to be able to catch the thief in the act. However, the reason why it is so hard to catch is that these issues are very random and very hard to reproduce.

Managed Heap Corruption

Managed heap corruption is a heap corruption that occurrs on the managed heaps. Again, here, you get the problem long after it occurred. A managed heap corruption occurrs if someone overwrites a piece of the managed heap that it wasn't allowed to.  Normally you can't buffer overflow in managed code. If you have a byte[] and try to write outside the bounds you will get an IndexOutOfRange exception or similar.  The most common reason for a managed heap corruption is that the code called PInvoke and passed in a buffer of some sort but the buffer was too small. The PInvoked function writes to the buffer but writes beyond the bounds of the buffer and onto the next object on the managed heap.  Later on when the garbage collector is doing its work, it tries to go through the heap and things go bad so the process crashes.

If you get a crash and the top of the active stack contains a GC function, start looking for PInvokes in your code and see if you might be passing buffers that are too small.

Fatal Execution Engine Exceptions

Fatal Execution Engine Exceptions are fairly rare, but when they occurr it's normally a bug.  This means that for some reason we went into some piece of code that we were not supposed to in the CLR and the CLR has decided that in the unlikely event that someone comes in here, let's throw a Fatal Execution Engine exception and die because we can't recover from this point. In the eventlog this will be logged as Execution Engine Exception occurred and the address listed will tell exactly where in the code it occurred.  If you reach one of these and can't find a knowledge base article about it, contact support, preferably with a crash dump available since that will speed up the time to resolve the case tremendously.

GC Holes

 

These are fairly rare too. The unmanaged portion of the CLR has a pointer to managed memory and "forgot" to tell the GC about it so the GC doesn't know to keep it around (if there are no other roots) or to track movement of it, which means that if a GC happens at the "wrong time" the pointer could be pointing basically anywhere causing a lot of havoc. Yun Jin talks a little about it here http://blogs.msdn.com/yunjin/archive/2004/02/08/69906.aspx

 

To this date I have had little or no issues involving GC Holes so it wouldn’t be the first thing to look for in a "crash" dump.

No available memory left

If your process dies and at the same time you see a huge dip in the memory/available MBytes, to the point where its 0, this can cause the process to crash. Here of course the task will be to check what process stole the memory.

External process kills/Recycles the process

Countless times when I have had a customer with adplus attached in crash mode i have gotten memory dumps where the process was killed by something external.  I wanted to add this gotcha since normally when you troubleshoot a crash, this is not the actual reason for the crash you are troubleshooting (only in rare cases), but rather that someone ran iisreset or killed the process not knowing that you had a debugger attached waiting for the crash.  So if you are troubleshooting a crash and get a memory dump. Check the eventlog to make sure that noone ran an iisreset so this was the reason for your crash dump. That will save you some hours trying to find the culprit of the crash:)

Another curious one is where some kind of monitoring software is installed to monitor that the server servs pages as expected and shuts down the process if it doesn't. But usually this logs an event of the occurrance in the eventviewer as well.

Health monitoring settings

This one might be a doh!!! of course i know that we are set to recycle the process every 24 hours, or if the server has been idle for more than 20 minutes, but it's worth mentioning anyways because we get these issues pretty frequently.  Again here, normally an event will be logged in the eventlog stating why the process was recycled but the moral of the story is to have a brief look at your health monitoring settings for the application pool in iis or in machine.config to see what recycling opitons you have turned on so you don't get a surprise later.


There are more reasons for crashes than the above but they are usually pretty obscure so I hope this post has given you a little insight in where to start looking if your process suddenly dies.

Over and out...






  • I’ve written quite a few posts on memory issues because that is the type of problem we get most frequently...
  • Well, I have ASP.Net v2 crashing. So I'm trying to follow your instructions to debug it, but fail at the first hurdle. adplus says it has attached to aspnet_wp.exe, but then immediately exits and I get zero minimized command shells.

    It's always annoying when you have to debug the debugger before you can debug the platform.
  • Take a look at the log file that gets created in the hang or crash directory, somewhere near the bottom you should have an error message telling you the reason that it exited.  

    The most likely cause is that you don't have permissions to debug the process, i.e. that you are not an admin or otherwise priviledged to debug the process.  

    Another likely cause is that you are trying to debug in crash mode on a win2000 machine through terminal services.  This will not work as you are not logged in to the same windows session as the service you are debugging.  (non invasive i.e. hang mode works still because it doesnt actually break in to the process)

    If this is the case, the easiest way to deal with it if possible is to log on to the actual machine. If that is not possible you have to follow the steps in this article to run in crash mode remotely

    http://support.microsoft.com/default.aspx?scid=kb;en-us;286350#XSLTH4369121123120121120120

    Hope this takes you closer to debugging your crash...

  • "Fatal Excecution Engine" is a typo.  I really don't care, but it won't show up in Google if one searches for "Fatal Execution Engine", and this page is well worth finding.
  • Thanks for commenting on that John, I've changed it now...

    That's what you get for trying to typ faster than the brain can manage:)
  • I have taken the liberty to change this post sligthly, renaming the GC Holes section to Managed Heap Corruption which is more correct.

    Thanks to Rico (http://blogs.msdn.com/ricom) for pointing this out...
  • I get the following message when ever i try to do any thing after i load sos.dll

    "Failed to find runtime DLL (mscorwks.dll), 0x80004005

    Extension commands need mscorwks.dll in order to have something to do."

    following is what i did,

    1. Load a crash dump

    2. .load c:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\SOS.dll

    3. !threads

    "c:\dumps\" is the folder from where i loaded the crash dumps

    "C:\Program Files\Debugging Tools for Windows\" is where the debugger is installed

  • Sounds like you are either debugging an application that is not managed (.net) or that hasnt yet loaded mscorwks.dll  alternatively that it is an 1.1 app and youre trying to debug with 2.0 sos

  • addendum to last comment:)

    try lmvm mscorwks to see if mscorwks.dll is loaded

    you can also try lmvm mscor* to see what mscor dlls are loaded and what versions

  • I too get the same error

    0:025> !threadpool

    Failed to find runtime DLL (mscorwks.dll), 0x80004005

    Extension commands need mscorwks.dll in order to have something to do.

    Can you please provide a solution for this.

    BTW, I'm trying to get the minidump of ASP.NET 2.0 application.

Page 1 of 1 (10 items)
Leave a Comment
  • Please add 7 and 5 and type the answer here:
  • Post