.NET Debugging Demos Lab 2: Crash

If broken it is, fix it you should : .NET Debugging Demos - Information and setup instructions

Fri, 08 Feb 2008 14:58:20 GMT

PingBack from http://blogs.msdn.com/tess/pages/net-debugging-demos-information-and-setup-instructions.aspx

re: .NET Debugging Demos Lab 2: Crash

Dragos — Fri, 08 Feb 2008 17:05:05 GMT

Thank you very much for the wonderful work. I was in big need of a tool like Windbg, it really helps with my work.

re: .NET Debugging Demos Lab 2: Crash

Kevin — Sat, 09 Feb 2008 06:19:57 GMT

Hi Tess,

Excellent labs, looking forward to more.

I have one question though. On a server running multiple sites (hundreds in fact) in the same app pool, how do I identify the site that caused a hang or crash having identified the root cause?

Or to put it differently, how do I match up threads to IIS sites/applications in windbg?

As a hoster we can be running up to 1000 sites on a single server with those sites divided across say 5-10 app pools.

Cheers

Kev

re: .NET Debugging Demos Lab 2: Crash

Tess — Sun, 10 Feb 2008 00:08:17 GMT

Hi Kev,

Although i probably wouldn't recommend running 200 apps per app pool because of how much memory usage there would be per process (likely OOMs just because of the dlls loaded alone) your question is very valid.

The finalizer thread is common to all apps in the process but for all other threads you can check out the threads in !threads and check which appdomain the code is running in by running !dumpdomain on the domain in the domain column.

re: .NET Debugging Demos Lab 2: Crash

GProssliner — Tue, 12 Feb 2008 13:28:05 GMT

Hello!

I have the following SOS output:

0:000> !threadpool

CPU utilization 100%

Worker Thread: Total: 2 Running: 0 Idle: 2 MaxLimit: 25 MinLimit: 2

Work Request in Queue: 0

--------------------------------------

Number of Timers: 3

--------------------------------------

Completion Port Thread:Total: 5 Free: 0 MaxFree: 4 CurrentLimit: 2 MaxLimit:

25 MinLimit: 2

It's obvious, that the 100% CPU utilization is a problem (that has been solved already). My question is if and how the Threadpool used the current CPU utilization for scheduling WorkItems or to control how and if new Workerthreads are created (the Threadpool is primary used for async Socket Operations (HttpListener) within this project).

It seems like no new threads are started, even if the current number of Threadpool Threads is below the Max-Threads (what could makes sence because there would be no resources available for the new thread).

So ... how can the values (Total, Running, Idle, MaxLimit and MinLimit) be interpreted?

Any toughts?

re: .NET Debugging Demos Lab 2: Crash

Tess — Tue, 12 Feb 2008 15:19:50 GMT

Are you saying you are getting those numbers for this lab or in some other dump? I'm just curious because you shouldnt get 100% CPU in this specific crash lab...

The 100% is for the whole system, not only the w3wp.exe process so this would also include any CPU usage by other processes.

Total = number of current worker threads started (running+idle)

Running = executing a request or work item

MaxLimit = max number of worker threads (as set in machine.config for asp.net or 1000 by default for winforms)

MinLimit = 1 per logical CPU (min number of worker threads at any given time)

The threadpool does take CPU usage into account, and currently it will not create new threads if the systems cpu usage is over 80%

re: .NET Debugging Demos Lab 2: Crash

GProssliner — Tue, 12 Feb 2008 15:33:16 GMT

Hello!

First of all: No. This is not related to this lab. I just read the Review posting where you said that questions - even not directly related to the lab - are welcome.

Thank you for sharing this information! The 80% threshold is hardcoded or configurable?

Maybe you can also explain the Completion Port Thread values (Total, Free, MaxFree, CurrentLimit, MaxLimit and MinLimit) too? I've already search over and over the web but doesn't find anything about them.

Thank you!

re: .NET Debugging Demos Lab 2: Crash

Tess — Tue, 12 Feb 2008 15:39:35 GMT

Its totally cool to ask questions not related to the lab:) just wanted to make sure that the lab didnt behave like that on your machine.

The completion port threads are pretty much the same. Completion ports are mostly used for callbacks but can be used for work items too if there are available completion port threads but no available worker threads.

The 80% is hard coded but in reality there is no use to change it since you really can't do much with new threads at that CPU level anyways.

re: .NET Debugging Demos Lab 2: Crash

Andrew Lomakin — Thu, 14 Feb 2008 13:43:12 GMT

Hello Tess!

Great post, gives very important knowledge needed for newbie .NET crash analyst :)

I have a situation where DFS management snapin is crashing due to a null-reference exception (0x80004003), and i've gone a long way to identify at what level the exception occurs (let me know if you're curious enough to look at the dump - you should have my email somewhere hopefully). Eventually i came to the clr thread stack where the exception occurs, but i'm stuck, because i want to observe what parameters are passed to each funciton in the stack, but i went through your blog posts, and Johan's, and i can't seem to find a way to do this. Can you advice a little bit please? I've been banging my head against the table about this case for weeks now.

Regards,

Andrew

re: .NET Debugging Demos Lab 2: Crash

Tess — Thu, 14 Feb 2008 14:23:14 GMT

Hi Andrew,

Yepp, I remember you:) I can't really commit to look at any dumps but I can give you some pointers.

0x80004003 is not really a clr exception, and I am not sure based on your comment if you are actually stopped at the exception or just see it on the heap. If you got it from the heap you won't be able to inspect the parameters etc. so in that case you would have to set up debug diag or an adplus config file to get a full dump on 0x80004003. Check the windbg help files for adplus config for more info on that...

If you are stopped on the exception you can either use !clrstack -p to find the parameters or if that doesnt help you can try !dso to see the objects on the stack,

Best of luck

Tess

Summary of the Week - 02/11

André Nobre — Fri, 15 Feb 2008 15:00:01 GMT

I decided to publish, every friday, some links that i judge interesting, from now. Architecture Scott

.NET Debugging Demos Lab 3: Memory

If broken it is, fix it you should — Fri, 15 Feb 2008 17:10:44 GMT

TGIF, almost time for the weekend... but before you leave, here is lab 3. Todays debugging puzzle will

re: .NET Debugging Demos Lab 2: Crash

Ed F — Fri, 22 Feb 2008 01:16:12 GMT

Thanks a lot for these labs, I'm learning a lot. The assembler stuff was really neat to learn.

.NET Debugging Demos

Paul Mooney — Wed, 27 Feb 2008 14:47:08 GMT

Lab 1: HangLab 2: CrashLab 3: MemoryLab 4: High CPU Hang

.NET Debugging Demos Lab 7: Memory Leak - Review

If broken it is, fix it you should — Thu, 03 Apr 2008 15:33:48 GMT

We have reached the end of the .NET Debugging Demos series. And we are going to end it with a review

re: .NET Debugging Demos Lab 2: Crash

baal — Sat, 05 Apr 2008 15:58:16 GMT

Note: if you throw an exception (.net or other) you have a chance to handle it in a try/catch block. The first time it is thrown it becomes a 1st chance exception and is non-fatal. If you don't handle the exception it will become a 2nd chance exception (unhandled exception) and any 2nd chance exceptions will terminate the process.

what is it mean?

does the 1st exception can become the 2nd exception?

re: .NET Debugging Demos Lab 2: Crash

Tess — Mon, 07 Apr 2008 11:04:04 GMT

an exception is considered first chance when it is first thrown, at that point you can catch it in an exception handler. If you don't handle it it becomes a 2nd chance exception (unhandled) see my post on questions about exceptions

http://blogs.msdn.com/tess/archive/2008/04/01/questions-about-net-exceptions.aspx

re: .NET Debugging Demos Lab 2: Crash

baal — Mon, 07 Apr 2008 12:20:54 GMT

public void Page_load(......)

{

UserName1 = Request.Cookies["UserName1"].Value;

UserName2 = Request.Cookies["UserName2"].Value;

}

if an Exception hannpen in the Page_Load, Yellow Screen appeard. the exception is first chance, how it become 2nd ?

re: .NET Debugging Demos Lab 2: Crash

lindsay — Fri, 06 Jun 2008 18:45:11 GMT

These labs are very interesting, cant wait to learn more!

re: .NET Debugging Demos Lab 2: Crash

J.W. — Sat, 05 Jul 2008 01:42:34 GMT

I have q question about this" any 2nd chance exceptions will terminate the process."

If the global.asax has Application_Error , then any unhandled error will be handled there. I just feel if the process is terminated, all the people on the site will lost the session ( inproc session mode), it will be too bad.

.NET 调试实例 - 信息和安装说明 (原创翻译)

Justin — Mon, 07 Jul 2008 20:42:25 GMT

.NET调试实例

这是一个系列的调式实例，目的是为了帮助你在调式.NET应用程序中最常见的挂起(Hang)、性能(performance)、内存(memory)和系统崩溃(crash)方面获得一些...

re: .NET Debugging Demos Lab 2: Crash

Tess — Mon, 04 Aug 2008 22:33:43 GMT

J.W. a 2nd chance exception is basically an unhandled exception. If it is handled by the application_error it does not get to the 2nd chance stage, so it is still a handled exception (i.e. 1st chance).

Any exceptions that occurr during requests will get handled by the application_error if they are not handled before that... however if you have an exception happening on the finalizer or on any other non-request thread, like a timer thread or similar, it will not be handled and thus be 2nd chance and crash the process...

hope that makes sense, just think of it as in 1st chance exception = exception that can still be trapped and handled, 2nd chance exception = unhandled exception

re: .NET Debugging Demos Lab 2: Crash

James — Wed, 08 Oct 2008 00:44:56 GMT

Hey Tess,

I have a web app that has a, thus far, undiagnosed crash. I was hoping to use the steps in this lab to debug it, but it doesn't seem to work.

Firstly, there was only one dump file created named: PID-2148__W3WP.EXE_-OneStopCC-__1st_chance_Process_Shut_Down__full_0f20_2008-10-07_16-05-52-828_0864.dmp

(AppPool name is OneStopCC)

What does it mean when there is a crash that only creates the 1st chance/shut down dump and why does it not seem to have an exception in it (like in the lab)?

!pe returns "The current thread is unmanaged"

re: .NET Debugging Demos Lab 2: Crash

Tess — Wed, 08 Oct 2008 11:32:38 GMT

You will get a 1st chance process shutdown when the process goes down for whatever reason, i.e. it could be that the process preemptively recycled (make sure to uncheck all recycling options), or that someone called iisreset, or that a fatal exception occurred that caused it to shutdown.

The problem is that when the process is shutting down, the reason for the shutdown may already be gone... and in a shutdown dump you will typically just see the last thread that happened to finish in the dump, it is not neccessarily the faulting one.

Check the eventlogs to make sure that it wasnt just caught due to a recycle or iisreset, and if it wasnt, then check the log that adplus outputs and follow Lab 5 to see how you can get dumps on the actual exception that came before the crash.

By default, adplus will only capture dumps on a few things like 2nd chance exceptions (unhandled exceptions), access violations, invalid handle etc. so some things, like stackoverflow exceptions aren't caught by default, but lab 5 will show you how to catch those.

re: .NET Debugging Demos Lab 2: Crash

John — Sun, 14 Dec 2008 07:17:03 GMT

Thanks for the lab- starting to understand some of this