Last week I published a debugging challenge for Lab 5.  It was really interesting to see the results and I have to say I was really happy to see the excellent results from the poeple who commented on the debugging challenge (sounds like my work here is done:)).

Quick Poll, for the next one (a memory leak), do you want a debugging challenge or do you want the lab steps at once?

As usual it is using the Buggy Bits site, and this time we are dealing with a crash. You can read the problem description here.  

Previous demos and setup instructions

If you are new to the debugging labs, here you can find information on how to set up the labs as well as links to the previous labs in the series.

Information and setup instructions
Lab 1: Hang
Lab 1: Hang - review
Lab 2: Crash
Lab 2: Crash - review
Lab 3: Memory
Lab 3: Memory - review
Lab 4: High CPU hang
Lab 4: High CPU hang - review
Lab 5: Debugging Challenge

Reproducing the issue:

1. Recycle IIS (IISReset)

2. Browse to http://localhost/BuggyBits/CompanyInformation.aspx 

3. Type some message and click the Send button 

Note: If just-in-time debugging is enabled a message box might pop up because the w3wp.exe process crashed.  If this comes up, please ignore it.

Q: What do you see in the browser?

Explore the eventlogs

1. Open the eventlogs for system an application

Q: What events do you notice relating to the crash? (note: you may see different events on different operating systems)

Q: What does exit code '0x800703e9' mean?

Q: Based on the eventlogs, what caused the crash?

Reproduce the issue again and gather a memory dump:

1. Restart iis by running iisreset

2. Browse to http://localhost/BuggyBits/CompanyInformation.aspx again without hitting the send button

3. Open a command prompt, move to the debuggers directory and type the following command to attach the debugger and wait for the crash

adplus -crash -pn w3wp.exe -quiet

4. Enter a message on the page and click send

Q: What files get created in the dump folder (i.e. the folder named crash_mode current date and time under the debuggers directory)?

Open the dump to figure out what it is doing:

1. Open the process_shut_down dump (crash_mode with todays date and time in the debuggers directory) in windbg using file/open crash dump

2. Load up the symbols and sos

Examine the stacks:

Note: In a dump file the active thread is the thread that caused the debugger to dump the process, i.e. a process exit exception or other exception. 

One of a few things can cause the process to shut down.

  • An unhandled exception (2nd chance exception), in most cases if this is causing the shutdown, you will also get a 2nd chance exception dump, and this will be the one you want to look at to figure out what caused the crash
  • An external shutdown, i.e. an iisreset, a preemptive recycle (based on recycling options) or someone killing the process from task manager.  If this is the case, the active thread in the process shutdown dump is usually the main thread and you will see a stack similar to the following
    0:000> kL
ChildEBP RetAddr  
0014fbfc 7d503faf ntdll!ZwTerminateProcess+0x12
0014fc38 7d503f5a kernel32!_ExitProcess+0x4b
0014fc4c 79fd9e8f kernel32!ExitProcess+0x14
0014fe74 79f7479c mscorwks!SafeExitProcess+0x157
0014ff10 79004fab mscorwks!HandleExitProcessHelper+0x27
0014ff10 79004fab mscoree!CorExitProcess+0x46
0014ff20 77bcaddb mscoree!CorExitProcess+0x46
0014ff2c 77bcaefb msvcrt!__crtExitProcess+0x29
0014ff5c 77bcaf52 msvcrt!doexit+0x81
0014ff70 01001a3c msvcrt!exit+0x11
0014ffc0 7d4e7d2a w3wp!wmainCRTStartup+0x144
0014fff0 00000000 kernel32!BaseProcessStart+0x28

    If this is the case you should look at the eventlog as it usually contains the recycling reason, and if you are troubleshooting a real crash you should consider disabling recyling while troubleshooting the real crash so that you won't get these type of "red herring" dumps.
  • Something in the process called process exit.  This is usually caused by some type of exception considered fatal, i.e. heap corruption, stack overflow, out of memory exceptions or fatal execution engine exceptions.  In this case the active thread in the process shutdown stack will look (and probably is) completely unrelated to the crash. In this case you will want to look at the log to see what happened just prior to the crash in order to figure out how to proceed and gather new dumps if necessary.

1. Run kb to see which of the above categories this crash/process exit falls into

Q: Which category did it fall into and why?

Examine the log file to find out what happened just prior to the crash

Q: What events occurred just prior to the crash? 

Configure adplus to gather full dumps on the exception in the log file 

1. Restart iis by running iisreset

2. Browse to http://localhost/BuggyBits/CompanyInformation.aspx again without hitting the send button

3. Copy the attached adplus configuration file to your debuggers directory  (Note: this configuration file causes adplus to take a full dump on the unknown exception)

4. Open a command prompt, move to the debuggers directory and type the following command to attach the debugger and wait for the crash

adplus -pn w3wp.exe -c Unknown.cfg

4. Enter a message on the page and click send

Open the dump to figure out what it is doing:

1. Open the 1st_chance_unknown_exception dump in windbg using file/open crash dump

2. Load up the symbols and sos

Examine the active stack:

1. Run kb 2000 and !clrstack to see what the active thread is doing (i.e. the thread that caused the stackoverflow)

Q: Can you tell what the issue is from here?

Verify your assumption in the code and modify the code to resolve the issue 

1. Open ExceptionHandler.cs and Utility.cs and device a plan for how to fix the problem

Have fun,

Tess