If broken it is, fix it you should

Using the powers of the debugger to solve the problems of the world - and a bag of chips    by Tess Ferrandez, ASP.NET Escalation Engineer (Microsoft)

.NET Debugging Demos Lab 2: Crash

.NET Debugging Demos Lab 2: Crash

  • Comments 31

It was nice to see that so many people downloaded the demo site already and checked out the lab instructions for the first lab, and thanks to Pedro for pointing out that the original demo site required .NET Framework 3.5... I've changed it now so the one that you can download from the setup instructions page should not require .Net Framework 3.5.  (Even though I would encourage you to download 3.5 and play around with it anyways:))

Here comes lab 2, a crash scenario on the BuggyBits site.  

Previous demos and setup instructions

Information and setup instructions
Lab 1: Hang
Lab 1: Hang - review 

Reproduce the problem

1. Browse to the reviews page http://localhost/BuggyBits/Reviews.aspx, you should see a couple of bogus reviews for BuggyBits

2. Click on the Refresh button in the reviews page. This will crash the w3wp.exe process (or aspnet_wp.exe on IIS 5) 

    Note: If you have Visual Studio installed a Just-In-Time Debugger message may pop up (just click no for the purposes of this excercise).    
    However since this message box will sit there and wait for user input in order to shut down the app you may want to
disable JIT debugging if you have visual studio 
    installed on a test system.

Examine the eventlogs

1. Open the Application and System eventlogs, the information in the eventlogs will differ based on the OS and IIS version you are running. Among other events you may
    have a System Event looking something like this...

Event Type:	Warning
Event Source:	W3SVC
Event Category:	None
Event ID:	         1009
Date:		2008-02-08
Time:		10:12:06
User:		N/A
Computer:   	MYMACHINE
Description:
A process serving application pool 'DefaultAppPool' terminated unexpectedly. The process id was '4592'. 
The process exit code was '0xe0434f4d'.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Q: What events do you see?

Q: What does the exit code 0xe0434f4d mean?

Q: Can you tell from the eventlogs what it was that caused the crash? 

Get a memory dump

1. Browse to the reviews page http://localhost/BuggyBits/Reviews.aspx again, but don't click refresh

2. Open a command prompt and move to the debuggers directory and type in "adplus -crash -pn w3wp.exe" and hit enter

Q: A new window should appear on the toolbar, what is it? 

Q: What is the debugger waiting for? Hint: Check the help files for ADplus/crash mode in windbg

3. Reproduce the issue by clicking on the refresh button in the reviews page.

Q: What files got created in the dump folder?  Note: The dump folder will be located under your debuggers directory with the name crash_mode and todays date and time 

Open the dump in windbg

1. Open the dump file labeled 2nd Chance CLR Exception in windbg (file/open crash dump).  Note that this dump got created just before the 1st chance process shutdown.

Note: if you throw an exception (.net or other) you have a chance to handle it in a try/catch block.  The first time it is thrown it becomes a 1st chance exception and is non-fatal.  If you don't handle the exception it will become a 2nd chance exception (unhandled exception) and any 2nd chance exceptions will terminate the process.

2. Set up the symbol path and load sos (see the setup instructions for more info)

 

In a crash dump, hte active thread is the one that caused the exceptions (since the dump is triggered on an exception).

Q: Which thread is active when you open the dump? Hint: check the command bar at the bottom of the windbg window.

Examine the callstacks and the exception

1. Examine the native and managed callstacks. 

kb 2000
!clrstack

Q: What type of thread is it?

Q: What is this thread doing?

2. Examine the exception thrown

!pe

Note: !pe/!PrintException will print out the current exception being thrown on this stack if no parameters are given

Q: What type of exception is it?

Note: In some cases, like this one where the exception has been rethrown, the original stacktrace may not be available in the exception.  In cases like this you may get more information if you find the original exception

3. Look at the objects on the stack to find the address of the original exception

!dso

Q: What is the address of the original exception

Hint: Look at your previous pe output to see the address of the rethrown exception.  Compare this to the addresses of the objects on the stack.  You should have multiple exceptions, a few with the rethrown exception address but one of the bottommost exceptions will be the original one (look for one with a different address).

4. Print out the original exception and look at the information and the callstack

!pe <original exception address>

Q: In what method is the exception thrown?

Q: What object is being finalized?

Note: you could actually have gotten this information by dumping out the _exceptionMethodString of the rethrown exception as well, but with !pe of the original exception you get the information in a cleaner way.

Q: Normally exceptions thrown in ASP.NET are handled with the global exception handler and an error page is shown to the user.  Why did this not occurr here?  Why did it cause a crash?

Examine the code for verification

1. Open Review.cs to find the destructor/finalizer for the Review class

Q: which line or method could have caused the exception

 

As an extra excercise you can also examine the disassembly of the function to try to pinpoint better where in the function the exception is caused

!u <IP shown in the exceptionstack>

 

Related posts

Creating dumps with Windbg and writing ADPlus Config files

ASP.NET 2.0 Crash case study: Unhandled exceptions

What on earth caused my process to crash?

.Net exceptions - Tracking down where in the code the exceptions occurred

 

Have fun debugging,

Tess

  • Hi Tess ,

    What this process exit code  '0xc0000354'  gives ?

    Sometimes we are getting the following error also

    Event Type: Warning

    Event Source: W3SVC

    Event Category: None

    Event ID:         1009

    Date: 11-11-2010

    Time: 10:44:28

    User: N/A

    Computer:   APP-01

    Description: A process serving application pool 'My App pool' terminated unexpectedly. The process id was '3452'. The process exit code was '0xc0000354'.

    Thanks

    DB

Page 3 of 3 (31 items) 123
Leave a Comment
  • Please add 4 and 1 and type the answer here:
  • Post