If broken it is, fix it you should

Using the powers of the debugger to solve the problems of the world - and a bag of chips    by Tess Ferrandez, ASP.NET Escalation Engineer (Microsoft)

.NET Debugging Demos Lab 5: Crash

.NET Debugging Demos Lab 5: Crash

  • Comments 9

Last week I published a debugging challenge for Lab 5.  It was really interesting to see the results and I have to say I was really happy to see the excellent results from the poeple who commented on the debugging challenge (sounds like my work here is done:)).

Quick Poll, for the next one (a memory leak), do you want a debugging challenge or do you want the lab steps at once?

As usual it is using the Buggy Bits site, and this time we are dealing with a crash. You can read the problem description here.  

Previous demos and setup instructions

If you are new to the debugging labs, here you can find information on how to set up the labs as well as links to the previous labs in the series.

Information and setup instructions
Lab 1: Hang
Lab 1: Hang - review
Lab 2: Crash
Lab 2: Crash - review
Lab 3: Memory
Lab 3: Memory - review
Lab 4: High CPU hang
Lab 4: High CPU hang - review
Lab 5: Debugging Challenge

Reproducing the issue:

1. Recycle IIS (IISReset)

2. Browse to http://localhost/BuggyBits/CompanyInformation.aspx 

3. Type some message and click the Send button 

Note: If just-in-time debugging is enabled a message box might pop up because the w3wp.exe process crashed.  If this comes up, please ignore it.

Q: What do you see in the browser?

Explore the eventlogs

1. Open the eventlogs for system an application

Q: What events do you notice relating to the crash? (note: you may see different events on different operating systems)

Q: What does exit code '0x800703e9' mean?

Q: Based on the eventlogs, what caused the crash?

Reproduce the issue again and gather a memory dump:

1. Restart iis by running iisreset

2. Browse to http://localhost/BuggyBits/CompanyInformation.aspx again without hitting the send button

3. Open a command prompt, move to the debuggers directory and type the following command to attach the debugger and wait for the crash

adplus -crash -pn w3wp.exe -quiet

4. Enter a message on the page and click send

Q: What files get created in the dump folder (i.e. the folder named crash_mode current date and time under the debuggers directory)?

Open the dump to figure out what it is doing:

1. Open the process_shut_down dump (crash_mode with todays date and time in the debuggers directory) in windbg using file/open crash dump

2. Load up the symbols and sos

Examine the stacks:

Note: In a dump file the active thread is the thread that caused the debugger to dump the process, i.e. a process exit exception or other exception. 

One of a few things can cause the process to shut down.

  • An unhandled exception (2nd chance exception), in most cases if this is causing the shutdown, you will also get a 2nd chance exception dump, and this will be the one you want to look at to figure out what caused the crash
  • An external shutdown, i.e. an iisreset, a preemptive recycle (based on recycling options) or someone killing the process from task manager.  If this is the case, the active thread in the process shutdown dump is usually the main thread and you will see a stack similar to the following
    0:000> kL
    ChildEBP RetAddr  
    0014fbfc 7d503faf ntdll!ZwTerminateProcess+0x12
    0014fc38 7d503f5a kernel32!_ExitProcess+0x4b
    0014fc4c 79fd9e8f kernel32!ExitProcess+0x14
    0014fe74 79f7479c mscorwks!SafeExitProcess+0x157
    0014ff10 79004fab mscorwks!HandleExitProcessHelper+0x27
    0014ff10 79004fab mscoree!CorExitProcess+0x46
    0014ff20 77bcaddb mscoree!CorExitProcess+0x46
    0014ff2c 77bcaefb msvcrt!__crtExitProcess+0x29
    0014ff5c 77bcaf52 msvcrt!doexit+0x81
    0014ff70 01001a3c msvcrt!exit+0x11
    0014ffc0 7d4e7d2a w3wp!wmainCRTStartup+0x144
    0014fff0 00000000 kernel32!BaseProcessStart+0x28
    

    If this is the case you should look at the eventlog as it usually contains the recycling reason, and if you are troubleshooting a real crash you should consider disabling recyling while troubleshooting the real crash so that you won't get these type of "red herring" dumps.
  • Something in the process called process exit.  This is usually caused by some type of exception considered fatal, i.e. heap corruption, stack overflow, out of memory exceptions or fatal execution engine exceptions.  In this case the active thread in the process shutdown stack will look (and probably is) completely unrelated to the crash. In this case you will want to look at the log to see what happened just prior to the crash in order to figure out how to proceed and gather new dumps if necessary.

1. Run kb to see which of the above categories this crash/process exit falls into

Q: Which category did it fall into and why?

Examine the log file to find out what happened just prior to the crash

Q: What events occurred just prior to the crash? 

Configure adplus to gather full dumps on the exception in the log file 

1. Restart iis by running iisreset

2. Browse to http://localhost/BuggyBits/CompanyInformation.aspx again without hitting the send button

3. Copy the attached adplus configuration file to your debuggers directory  (Note: this configuration file causes adplus to take a full dump on the unknown exception)

4. Open a command prompt, move to the debuggers directory and type the following command to attach the debugger and wait for the crash

adplus -pn w3wp.exe -c Unknown.cfg

4. Enter a message on the page and click send

Open the dump to figure out what it is doing:

1. Open the 1st_chance_unknown_exception dump in windbg using file/open crash dump

2. Load up the symbols and sos

Examine the active stack:

1. Run kb 2000 and !clrstack to see what the active thread is doing (i.e. the thread that caused the stackoverflow)

Q: Can you tell what the issue is from here?

Verify your assumption in the code and modify the code to resolve the issue 

1. Open ExceptionHandler.cs and Utility.cs and device a plan for how to fix the problem

Have fun,

Tess

Attachment: unknown.cfg
  • Since I already posted a challenge for this lab earlier I didn't want to wait too long with publishing

  • I would prefer a debugging challenge

  • You've been kicked (a good thing) - Trackback from DotNetKicks.com

  • Sharepoint No VSE WSS for VS 2008 Until Summer [Via: Tariq ] WPF Podder v2 has been released! [Via:...

  • I have written a few posts about stackoverflow exceptions, here, here , here and here . The one I am

  • I downloaded and installed "Debugging Tools for Windows (x86)", but when I write "adplus -crash -pn w3wp.exe -quiet" in command prompt,I get the next error:

    !!! Error found !!!

    An output directory was not defined

    !!!ERROR-ADPlus failed to run

    Current log content

    What is missing?I use Windows 7 and iis7

  • Fee, with newer versions of adplus you need to use the -o switch and provide an output directory i.e. -o c:\mydumps for example

  • We are trying to debug a StackOverflowException in a ASP.NET applicaton (.NET 4.0) by using the method described above. Unfortunately, all managed threads show with a thread id of XXXX, and we are not able to switch to any of them to run !clrstack.

    We would appreciate any suggestions.

    Here is an example of the !threads output:

                                              PreEmptive                                                   Lock

          ID  OSID        ThreadOBJ     State GC       GC Alloc Context                  Domain           Count APT Exception

    XXXX    1  3b38 0000000000169520   1008220 Enabled  0000000000000000:0000000000000000 0000000000152230     0 Ukn (Threadpool Worker)

    XXXX    2 21960 000000000014baf0      b220 Enabled  0000000163d3c530:0000000163d3e470 0000000000152230     0 Ukn (Finalizer)

    XXXX    3  3a98 000000000628eb20   100a220 Enabled  0000000000000000:0000000000000000 0000000000152230     0 Ukn (Threadpool Worker)

    XXXX    4 1aba4 00000000062b4ad0      1220 Enabled  0000000000000000:0000000000000000 0000000000152230     0 Ukn

    .....

    XXXX   57 1bfe0 000000000ba0f550   1009220 Enabled  00000000c3ff8160:00000000c3ff8268 00000000062a0e10     1 Ukn (Threadpool Worker) System.StackOverflowException (00000000ffff01f0)

    XXXX   5f 19fa0 000000000b862c20   1009220 Enabled  0000000123c047e0:0000000123c06780 0000000000152230     0 Ukn (Threadpool Worker)

    XXXX   32  9d3c 000000000b864c60   1009220 Enabled  00000000c3419b30:00000000c341a268 00000000062a0e10     1 Ukn (Threadpool Worker)

    XXXX   67 11a8c 000000000bae3d00   1009220 Enabled  0000000000000000:0000000000000000 0000000000152230     0 Ukn (Threadpool Worker)

  • Hi SF,

    This is because you are getting in a bit too late in the process, when all the threads have died.

    You can set up a debug diag script to dump on stackoverflow (in the exceptions pane choose stackoverflow) and try and see if you can get on the threads then.

    if not you can also try to do !dumpheap -type Exception and see if you can get some more information from the stack of the exceptions or try !do 00000000ffff01f0 and see if the stack gives you something.

    Tess

Page 1 of 1 (9 items)
Leave a Comment
  • Please add 1 and 1 and type the answer here:
  • Post