If broken it is, fix it you should

Using the powers of the debugger to solve the problems of the world - and a bag of chips    by Tess Ferrandez, ASP.NET Escalation Engineer (Microsoft)

.NET Debugging Demos Lab 1: Hang

.NET Debugging Demos Lab 1: Hang

Rate This
  • Comments 51

 

This is the first in a series of about 10 labs on .NET debugging.  The lab will use a site called BuggyBits, and as the name suggests the bits are extremely buggy.

To get started, follow the setup instructions posted here.

I have a feeling that these hands-on-labs may generate a lot of questions and although I will try to answer any questions posted in the comments I can’t promise to answer them all so please feel free to answer other readers comments if you know the answer, and make sure that you have followed all the installation instructions.

Note: The questions in the labs (Q: … ) are only meant as an aid when troubleshooting the problem.  I will moderate any comments containing answers to these questions until I have released the lab review (about a week after the orignal lab post in order to give everyone a shot at the labs without answers)

Feel free to comment on the lab format good or bad so that I know what works well and what doesn't for future labs.

Without further a due, here comes Lab 1:

Reproduce the problem:

1. Browse to http://localhost/BuggyBits/FeaturedProducts.aspx
This should take about 5 seconds to show, you can see the start time and execution time the bottom of the page.

2. Open up 5 browsers, all browsing to this site and refresh them simultaneously

Note the execution time for each of them and make sure that the start time is pretty much the same on all (otherwise you probably didn’t run the reg file)

Q:  What are the execution times?

Q:  What is the CPU usage of the w3wp.exe process when reproing the problem? High or low CPU?

Q:  What are some potential reasons for a hang with these symptoms?

Get a memory dump:

1. Start a command window and browse to your debuggers directory.  Type the following command to prepare taking the dump but don’t hit enter quiet yet.
adplus –hang –pn w3wp.exe –quiet

2. Reproduce the problem either by browsing with 5 browsers as you did before or by stressing the site with tinyget with the following command line

tinyget -srv:localhost -uri:/BuggyBits/FeaturedProducts.aspx -threads:30 -loop:50

3. Hit enter in the adplus command window to take the memory dump while the requests are still executing.

Q: In adplus hang mode, what triggers the generation of the memory dump?


Q: What permissions do you need to take a memory dump of a process?


Q: Where are the dumps created? Hint: check the windbg help for adplus/hang mode

Open the dump in Windbg.exe

1. Open windbg and open the memory dump (.dmp file) with File/Open Crash dump.

2. Set up the symbol path (see Information and Setup Instructions for more info)

3. Load sos (see Information and Setup Instructions for more info)

Examine the stacks

1. Examine the native callstacks

~* kb 2000

2. Examine the .net callstacks

~* e !clrstack

Q:  Do you see any patterns or recognize any of the callstacks that suggests a thread is waiting for a synchronization mechanism?

Troubleshoot the hang

1. Determine the ID of the thread owning the lock
!syncblk

Q: What thread owns the lock?

Q: How many threads are waiting for the lock?
Hint: MonitorHeld = 1 for each owner and 2 for each waiter.

2. Pick one of the waiters (Hint: waiters will sit in AwareLock::Enter) and take a look at what it is doing.

~5s                          (move to thread 5, replace 5 with actual thread ID)
kb 2000                    (examine native stack)
!clrstack                    (examine .net stack)

Q: In which .net function is it waiting for the lock?

3. Determine what the owning thread is doing 

~5s                          (move to thread 5, replace 5 with actual thread ID)
kb 2000                    (examine native stack)
!clrstack                    (examine .net stack)

Q: Why is it blocking?

4. Examine the code for .NET method owning the lock to verify your theory.

Hints

The following articles may be useful when troubleshooting this hang:

Things to ignore when debugging an ASP.NET Hang - Update for .NET 2.0

A Hang Scenario, Locks and Critical Sections

.NET Hang Debugging Walkthrough

Automated .NET Hang Analysis

Have fun debugging 
/Tess

  • just catching up with the tutorial now, this is really fun and I'm learning a lot, I always wanted something like this

    thank you very much for putting this together

  • Terry, can't say I know off hand what is wrong there.  To be honest I haven't really worked much with debugging in VS2008 so I would probably not be the best person to ask.  If the guy/gal you are working with says that it is a known issue, then I would trust that.

  • Last week I published a debugging challenge for Lab 5. It was really interesting to see the results and

  • Since I already posted a challenge for this lab earlier I didn't want to wait too long with publishing

  • Links 6/Fev: ASP.NET, ASP.NET AJAX, Visual Studio, .NET, WPF

  • Hi i was tryin to implement the lab one,

    as i open the dump file in win debugger i am not able to analyse as per the stpes provided in the article above

    N e help would b appreciated.

    Regards

  • could you be a little bit more specific, what parts are failing?

  • We have reached the end of the .NET Debugging Demos series. And we are going to end it with a review

  • The purpose of my presentation was to show some common pitfalls and of course to show off windbg and

  • Collegamenti del 6 Febbraio, ASP.NET, ASP.NET AJAX, Visual Studio, .NET, WPF

  • -原文地址:http://blogs.msdn.com/tess/archive/2008/02/04/net-debugging-demos-lab-1-hang.aspx

    译者注释

    (Just...

  • First thank you for this informative and educationg blog!

    I have followed this lab but did not get any hang.  All the browsers have returned finally and when looking at the hang dump there were no deadlocks.

    Also, using the tinyget.exe to cause a hang,

    I took a hang dump during the process's run but when opening the dump, the winDBG always get stack when trying to view certain thread's stack (always the same threads).

    The cpu utilization is ~0 both for w3wp and tinyget process.

    Any ideas?

    Thank you, Michal

  • the cpu utilization should be pretty low, and the requests should eventually return (after 5 seconds) so that is normal...   you need to get the memory dumps while the requests are still executing to see why they are slow...

  • I know I'm a little late in the game, but I decided that after vacation it was finally time for me to

  • Hey Tess,

    When I run adplus -hang -pn w3wp.exe -quiet,  the w3wp.exe memory (from task manager) jumps as the dump occurs.  For instance, the memory starts at around 48Mb and within a matter of 10 seconds after running adplus it goes up to 180Mb.  Is that supposed to happen?  And if so, doesn't that mean a lot of "stuff" is being added that wasn't there before (180mb after dump - 48mb before dump = 132Mb of stuff)?

    I should add, that the Virtual Memory stays at 48Mb...

    Thanks,

    James

    btw - You're my memory debugging hero!

Page 2 of 4 (51 items) 1234
Leave a Comment
  • Please add 1 and 5 and type the answer here:
  • Post