How to debug crashes and hangs

How to debug crashes and hangs

  • Comments 38

At my job on the C# IDE QA team I've learned some useful things about debugging in Visual Studio, which I'd like to summarize in this post. Although the screenshots were made using Visual Studio 2008 SP1, this pretty much applies to other versions of VS as well.

Rich debugging support

When you develop your C# application and hit F5, the target process (your program) gets started, and then the Visual Studio process attaches the debugger to the process where your code is running. This way, you can break into the debugger and VS will provide you with all sorts of rich debugging support - current statement highlighting, call stack, watches, locals, immediate window, Edit-and-Continue and so on.

More importantly, if your application throws an exception or crashes, the debugger will intercept that and provide your with all the information about the exception.

As a side note, in Visual Studio there is a way to run your code without attaching the debugger - the shortcut is Ctrl+F5. Try throwing an exception in your code when using F5 and Ctrl+F5 to feel the difference.

throw null;

By the way, my favorite way to artificially throw exceptions is throw null; I just love the fact that it throws a NullReferenceException because it can't find the exception object and nevertheless does exactly what I want it to do :)

Crashes and the Watson dialog

What if a program crashes or throws an exception, which you don't have source code for? Moreover, you didn't start the program using F5, but the operating system launched the process. I remember that before coming to Microsoft, the only thing I could do about some application crashing was to express my disappointment about the fact (usually in Russian). Now I don't feel helpless anymore, because I've learned a couple of tricks. As an example for this we'll crash Visual Studio itself and then debug the crash.

How to crash Visual Studio?

Viacheslav Ivanov reported an interesting crashing bug in our language service recently. Save all your work and then paste this code in a C# Console Application and change 'object' to 'int':

using System;

static class Program
{
    static void Main()
    {
        ITest<object> test;
        test.Test((object /* and now change the argument type to "int" */ i) => { });
    }
}

public interface ITest<T> { }

public static class Extensions
{
    public static void Test<T, I>(this ITest<T> test, Action<ITest<I>> action) { }
}

What you will see is the Watson dialog:

image

Given this chance, I'd highly recommend everyone to click "Send Error Report" if you ever see this dialog for Visual Studio. Many people "Don't Send" and frankly I don't understand why not. There is no personal information being sent, and we don't want your personal information anyway, honest. What we want is a call-stack and minidump, if possible, so if you want us to fix the product to make it more stable in the future, you will greatly help us if you send us the Error Report. By the way, we usually fix most (if not all) crashes that come in through this dialog, so the chances that we'll fix the crash you report using the dialog are actually pretty high. For example, we've already fixed the bug mentioned above and it works just fine in current builds.

Attaching a debugger

So what can you do if an application crashes or hangs? You can attach the debugger to a running process, even if it has already crashed. The code is still being executed (the main thread of the crashed application is usually pumping messages for the error dialog). You can either choose "Debug" on the Watson dialog, or (what I usually do) is start a new instance of Visual Studio myself and attach to the process manually, without dismissing the Watson dialog.

Note: if the debuggee process crashes and you attach the debugger after the fact, you'll have to manually break into the debugger by pushing the "Pause" button. If the debugger was already attached at the moment of the crash, then it will offer you to break or continue.

You can attach the Visual Studio debugger to a running process by choosing Tools | Attach To Process (Ctrl+Alt+P):

image

You will see the Attach to process dialog:

image Interesting things to note here are the Attach to: selection. Since Visual Studio is a mixed-mode managed/native application, to get call stacks for both managed and native parts, you'd want to attach to both of them (by clicking Select...):

image

If you select both Managed and Native, you'll get richer debugging information about your callstacks - this is recommended.

Note: if you want to enable mixed-mode debugging (managed+native) for the application that you have source code for, go to project properties of the startup project, and on the Debug tag select "Enable unmanaged code debugging". Then the debugger will automatically attach using mixed-mode.

Finally, select the process from the list which you'd like to attach to (in our example, it will be devenv.exe) and click Attach. Notice that the process of the debugger itself is not shown in the list, that's why you don't see two devenv.exe in the list.

Remote Debugging

What many people don't know is that VS can be used to debug a process running on another machine on the network. To do this, you just need to start the Visual Studio Remote Debugging Monitor on the same machine with the process you want to debug:

image

The remote debugging monitor will listen to debugger connections on the other machine and you'll be able to attach the debugger using the Transport and Qualifier fields on the Attach to process dialog:

image

You can find more detailed information about Remote Debugging in MSDN and on the internet.

Set Enable Just My Code to false

One very important option in Visual Studio is "Enable Just My Code", which is set to true by default. To be able to see more information on the callstacks instead of just "Non-user code", you need to go to Tools | Options and disable this option:

image

I usually do this right after I install Visual Studio, so that I can always debug into "not my code".

Other interesting options on this page are:

  • Enable .NET Framework source stepping - in case you'd like to step into the .NET framework source code
  • Enable Source Server support
  • Step over properties and operators (Managed only) - won't step in to property getters, operators, etc. - this is new in VS 2008 SP1
  • Require source files to exactly match the original version - in case you don't have the source files for the exact .pdb symbol file, but still have a close version of the source code

Break on first chance exceptions

One very useful option in the debugger is the ability to break whenever a first-chance exception is being thrown. A first-chance exception is an exception that might be caught by the program itself later in some surrounding catch block. First-chance exceptions are usually non-fatal and handled (or swallowed) by the user, so they might not even be visible to the end user during normal execution. However, if a first-chance exception is not handled in the code and bubbles up to the CLR/OS, then it becomes a crash.

So, to break on first-chance exceptions, you can go to Debug | Exceptions to invoke the Exceptions dialog:

image

Here you can put a checkmark on Common Language Runtime Exceptions for the debugger to break every time a managed exception is thrown. This way you will see more hidden exceptions, some of them originating deep in the .NET framework class library. Sometimes there are so much first-chance exceptions that you can become overwhelmed, but they are incredibly useful to get to the root cause of the problem, because the debugger will show exactly where the exception originated preserving the original surrounding context. Another great advantage of breaking on first-chance exceptions is that the call stack is not unwound yet and the problem frame is still on the stack.

Debugging tool windows

OK, so now that we know how to attach a debugger and how to set the debugger options, let's see what happens after we've attached a debugger. For our exercise, you can open two instances of Visual Studio, attach the second instance's debugger to the first one, and then crash the first one using the lambda-expression crash code above. Instead of the Watson dialog on the debuggee process, you will see the following window in the debugger:

image

Now you have the chance to break and see where the exception happened. Continue is useful if the exception is actually a first-chance exception and you'd like to pass it on to user code to handle it. Let's hit Break and examine what tool windows are available under debugger.

Processes window

All the debugger windows are available from menu Debug | Windows. The Processes window shows the list of processes that the debugger is currently attached to. A nice trick is that you can actually attach to multiple processes at the same time. Then you can use this window to switch the "current" debuggee process and all other tool windows will update to show the content of the "current" process.

image

Note: on our team, we use this window a lot, because our tests run out-of-process. Our test process starts Visual Studio in a separate process and automates it using DTE, remoting and other technologies. Once there is a failure in the test, it's useful to attach to both the test process and the Visual Studio process under test. The most fun part is when I was debugging a crash in the debugger, and attached the debugger to the debugger process that is attached to some other process. Now if the debugger debugger crashes on you on the same bug, then you're in trouble ;) Sometimes I definitely should blog more about the fun debugging stories from my day-to-day job. But I digress.

Threads

As we all know, processes have multiple threads. If you break into a debuggee process, you will most likely end up with a list of threads that were active at the moment when you did break. Main thread is the green one - this is the UI thread of your application (in our example, Visual Studio). Main thread executes the application message loop, and is pumping the windows messages for the application's UI. If a message box or a dialog is shown, you will see this dialog's message loop on the main thread.

image

To switch threads, double-click the thread name that you're interested in. In most of the cases you'll be interested in the main thread. But if you start your own threads, give them a name so that they are easy to find in the threads list.

Call stack

Every thread has a call stack. Call stack is probably the most important thing you'd like to know about a crash - what was the sequence of function calls that lead to a crash? If the program is hanging, you'd like to know what function is it hanging in and how did it get there. Oftentimes by just glancing at a callstack you immediately know what's going on. "Is this callstack familiar?" or "Who is on the callstack?" is probably the question we ask most often during the bug triage process.

Anyway, here's the call stack window:

image

In our example we see that the C# language service module is on the stack (.dlls and .exes are called "modules" in the debugging world).

However instead of function names from cslangsvc.dll we see the addresses of the procedures in memory. This is because the symbols for the cslangsvc.dll module are not loaded. We'll look into how to load symbols in a moment.

Modules

The Modules window shows a list of .dlls and .exes loaded into the debuggee process:

image

There are multiple ways to load symbols for a given module. You can right-click on a module for a list of options:

image

Symbols for modules are stored in .pdb files and are produced with every debug build of the binary. pdb files contain mapping from compiled binaries back to the original source code, so that the debugger can display rich information (function names, source code locations, etc.) for the binary being debugged. Without symbols, debugging is only possible at the assembly level and registers window, you can't map the binaries back to the source code.

Symbols

A very useful dialog is the Tools | Options | Debugging | Symbols:

image

Here you can set paths where to look for the .pdb files. Normally, the .pdb files will be directly next to the binaries, in which case they are usually found and loaded automatically. Loading symbols takes some time, so Visual Studio supports caching symbol files to some directory. Also, if you don't want all the symbols for all the binaries loaded (it can take a while), you can check the checkbox "Search the above locations only when symbols are loaded manually". Load symbols from Microsoft symbol servers provides symbols for Microsoft products, such as Windows and Visual Studio. You can load symbols from here, or also from the Modules or Call Stack windows, by right-clicking on a module and choosing Load Symbols From. Since we're debugging into Visual Studio, the symbols are located on the Microsoft Symbols servers:

image

When we load public symbols, the following little dialog is showing:

image

After we've loaded the symbols for cslangsvc.dll we notice that the Call Stack window is now much more informative:

image

Given this call stack, anyone of our developers will now easily be able to pinpoint the problem. In our example we see that the problem happens when we try to show the Smart Tag for the Generate Method refactoring:

image

As you see, there are more modules for which the symbols haven't been loaded yet. You can load the symbols for those modules by right-clicking their stack frame and choosing Load Symbols From. Loading all the symbols is usually recommended for more detailed information.

That is why the call stack is so precious during bug reports. To save the call stack, you can Ctrl+A and Ctrl+C to copy all the stack into the clipboard. When you click Send Error Report in the Watson dialog, the call stack is included in the error report.

Also you see that the call stack is not very useful without the symbols - it's the symbols + the callstack that provide rich information about the crash.

Minidump

Another very useful information that you can provide about the crashed process is the memory dump (heap dump, minidump) - this is basically a snapshot of the memory state of the crashed process. To create a minidump, go to Debug | Save Dump As. Visual Studio will offer you to save the dump to a location on disk, and an option to just save the Minidump or Minidump with Heap:

image

Minidump with Heap will save more information than just the minidump, but will take a whole lot more disk space (for Visual Studio - possibly hundreds of megabytes - depending on the process working set size during the crash).

Those are the three components that we usually send to our developers during crash bug reporting:

  1. Call stack
  2. Symbols
  3. Minidump with heap

In most cases, they are able to understand the problem given this information. A list of repro steps is usually even better, because they can reproduce the problem themselves and enable first-chance exceptions to break early into the problem area.

Debugging hangs

If an application hangs, you can attach the debugger to it and hit break to see what thread and what call stack blocks execution. Usually looking at the call stack can give you some clues as to what is hanging the application. Debugging hangs is no different than debugging crashes.

Microsoft shares customer pain

Finally, I'd recommend everyone to watch this video:

Conclusion

In this post, I've talked about some things worth knowing for effective debugging sessions:

  • Turning off "Enable just my code"
  • Attaching the debugger using Tools | Attach To Process
  • Selecting Managed, Native for mixed-mode debugging or Enable unmanaged code debugging
  • Attaching to multiple processes
  • Selecting processes and threads
  • Breaking on first-chance exceptions using the Debug | Exceptions dialog
  • Picking the right thread
  • Loading symbols
  • Viewing the call stack
  • Saving the minidump file

Do let me know if you have feedback or corrections about this information.

kick it on DotNetKicks.com
  • I would like to know though why when debugging C++ the Quick Watch and Immediate windows many times do not execute what I type into them. For example, I have a C++ map and so to get the first item I type myMap[0] in the Immediate or the Quick Watch window. And instead of getting the first item, I get "Error: The overloaded operator is not supported" or something similar. Half and more of the times I use the Watch windows or the Immediate window to debug like this I get errors. Why is this the case and if VS can evaluate more complex expressions while debugging why can't it evaluate simple ones. The same problem I have when creating a filter for a break-point. I might want to break only if a specific variable evaluates to something. Well, many times not even the == operator is recognized making the break-point filters (you know the Break on Condition) feature useless in many cases.

  • You've been kicked (a good thing) - Trackback from DotNetKicks.com

  • Hi N.P.

    this looks pretty much like a bug - can you please file a new bug under http://connect.microsoft.com/visualstudio/feedback - someone from the C++ or Debugger team will help you out.

    Thanks,

    Kirill

  • Hello, Kirill :)

    Nice post. I shuddered for a moment when I saw your first screenshot :)

    I was wondering - which version of VS you use to do all those things described here?

    Is there any good trying to debug a crashed 3rd party (not sure if it's the right term) app that you don't have the source code or pdb for? Like windows explorer that keeps crashing on me from time to time. I tried to find the culprit shell extension (or something like that) and all I could get was call stack (not very useful since it contained calls from kernel32.dll and some unresolved calls represented by adresses) and a bunch of asm code that I'm not very good debugging at :)

  • Our offshore team managed to crash VS by creating a constant outside the scope of any member functions that got its value from a web.config file.  This crashes VS when you attempt to design the web page since there is no runtime context (i.e., no config file to read).  I do not know why one would use a function call which might fail to set a contant's value.

    Here's the way we found the problem:

    1. Load VS

    2. Load project in VS

    3. Load second copy of VS and debug by attaching to the first copy of VS.  Set debugging to break on all exceptions.

    4. Go to the first copy of VS and attempt to load the bad web page in the designer.

    5. This will throw an exception and break in the second VS's debugger.

  • Hi Ivan :)

    ha ha, I know this dialog has a horror-movie surprise effect because it brings in certain negative associations :)

    This post applies to Visual Studio 2008 Service Pack 1, but is pretty much the same for VS 2005 and VS 2008 as well.

    You can totally attach the debugger to explorer.exe and load symbols for Windows libraries such as kernel32.dll and user32.dll. They are available on the Microsoft Symbols Servers. This should give you a callstack good enough.

    Hope this helps,

    Kirill

  • Web 3 reasons why you should let Google host jQuery for you 链接A引发的思考 YUI Library, YUI Doc, and Github

  • Web3reasonswhyyoushouldletGooglehostjQueryforyou链接A引发的思考YUILibrary,YUIDo...

  • Kirill, thanks for the answer

    I remember an unsuccesful attempt to load symbols from MS servers in VS 2003. Shall try it in 2008, maybe something was changed there :)

  • Good article,

    Some extra points you could include.

    1. The "private" microsoft symbols are needed to undercover more than the function name. So if its a Microsoft dll your encountering a problem with you should probably get in contact with MS. Something like 90% of bugs submitted to Microsoft Tech Support (MTS) are caused by third parties.

    2. If people want an insight into what a dump contains - I'd recommend downloading windbg, load a dump file and issue a  !analyze

    3. For ASP.Net hangs/crashs/high CPU/OOM's DebugDiag has some great (expandable) rules using the debugging engine..

    4. There is a tool (err.exe I think) to see the description of codes like 0xC0000005 and their header file. Tip: Google the codes meaning rather than the hex code.

    5. Check out http://support.microsoft.com/kb/286350 - important to know difference between 1st and 2nd chance exceptions and using adplus to capture non VS related crashes

    6. Also check out John Robbins latest book.

    Cheers,

    Jeremy

  • I have faced lot of issues with DataGridView crashing out everyting whenever i open the form with a DataGridView  in the production environment.

    I don't face the same issue in the Development Environment.

    Not able to find the solution for it

    Your Kind Help is Appreciated

    Thanks in Advance

    Lakshmikanth

  • I was thrilled to learn about the Remote Debugging Monitor however upon looking into it further I found that it was truly dissapointing.

    The possibility of being able to debug processes running on our staging/development servers was tantalizing only to be crushed by the following requirement (from MSDN):

    To debug a process that is running under another account name: You Must Have Administrator privilege on the remote computer. If the ASP.NET worker process aspnet_wp.exe is running as SYSTEM or ASPNET, for example, you have to be an administrator on the computer where that process is running.

    How many developers actually have these kinds of permissions on a staging/dev server?

    Is there any workaround?

    Great article though, thanks!

  • Jeremy: thanks so much for these valuable comments.

    Lakshmikanth: unfortunately I cannot help people from all over the world with their debugging issues - I have my work to do. I did what I could by writing this article. You're on your own from here :)

    whatispunk: unfortunately I'm not aware of any workaround. You might ask around on the forums though...

  • Oh I got the Watson <a href="http://www.notionsolutions.com">window</a> alright...after pasting that code that is..this has been a fun and educating exercise for me..and I gotta say that after reading and going through what you did I just realized that I knew very little about debugging...

  • I recently found a new trick from 101 visual studio tricks site.

    I have enabled this setting and it throws the exact location of exception.

    goto Debug-->exceptions and

    check common language runtime exceptions.

    that is it wherever u have exceptions unhandled ur ide will show you while executing the application.

    Happy coding.

Page 1 of 3 (38 items) 123
Leave a Comment
  • Please add 2 and 6 and type the answer here:
  • Post