.NET Crash Dump and Live Process Inspection

.NET Crash Dump and Live Process Inspection

Rate This
  • Comments 39

Analyzing crash dumps can be complicated. Although Visual Studio supports viewing managed crash dumps, you often have to resort to more specialized tools like the SOS debugging extensions or WinDbg. In today’s post, Lee Culver, software developer on the .NET Runtime team, will introduce you to a new managed library that allows you to automate inspection tasks and access even more debugging information. --Immo

Today are we excited to announce the beta release of the Microsoft.Diagnostics.Runtime component (called ClrMD for short) through the NuGet Package Manager.

ClrMD is a set of advanced APIs for programmatically inspecting a crash dump of a .NET program much in the same way as the SOS Debugging Extensions (SOS). It allows you to write automated crash analysis for your applications and automate many common debugger tasks.

We understand that this API won’t be for everyone -- hopefully debugging .NET crash dumps is a rare thing for you. However, our .NET Runtime team has had so much success automating complex diagnostics tasks with this API that we wanted to release it publicly.

One last, quick note, before we get started: The ClrMD managed library is a wrapper around CLR internal-only debugging APIs. Although those internal-only APIs are very useful for diagnostics, we do not support them as a public, documented release because they are incredibly difficult to use and tightly coupled with other implementation details of the CLR. ClrMD addresses this problem by providing an easy-to-use managed wrapper around these low-level debugging APIs.

Getting Started

Let's dive right into an example of what can be done with ClrMD. The API was designed to be as discoverable as possible, so IntelliSense will be your primary guide. As an initial example, we will show you how to collect a set of heap statistics (objects, sizes, and counts) similar to what SOS reports when you run the command !dumpheap –stat.

The “root” object of ClrMD to start with is the DataTarget class. A DataTarget represents either a crash dump or a live .NET process. In this example, we will attach to a live process that has the name “HelloWorld.exe” with a timeout of 5 seconds to attempt to attach:

        int pid = Process.GetProcessesByName("HelloWorld")[0].Id;
        using (DataTarget dataTarget = DataTarget.AttachToProcess(pid, 5000))
        {
            string dacLocation = dataTarget.ClrVersions[0].TryGetDacLocation();
            ClrRuntime runtime = dataTarget.CreateRuntime(dacLocation);

            // ...
        }    

You may wonder what the TryGetDacLocation method does. The CLR is a managed runtime, which means that it provides additional abstractions, such as garbage collection and JIT compilation, over what the operating system provides. The bookkeeping for those abstractions is done via internal data structures that live within the process. Those data structures are specific to the CPU architecture and the CLR version. In order to decouple debuggers from the internal data structures, the CLR provides a data access component (DAC), implemented in mscordacwks.dll. The DAC has a standardized interface and is used by the debugger to obtain information about the state of those abstractions, for example, the managed heap. It is essential to use the DAC that matches the CLR version and the architecture of the process or crash dump you want to inspect. For a given CLR version, the TryGetDacLocation method tries to find a matching DAC on the same machine. If you need to inspect a process for which you do not have a matching CLR installed, you have another option: you can copy the DAC from a machine that has that version of the CLR installed. In that case, you provide the path to the alternate mscordacwks.dll to the CreateRuntime method manually. You can read more about the DAC on MSDN.

Note that the DAC is a native DLL and must be loaded into the program that uses ClrMD. If the dump or the live process is 32-bit, you must use the 32-bit version of the DAC, which, in turn, means that your inspection program needs to be 32-bit as well. The same is true for 64-bit processes. Make sure that your program’s platform matches what you are debugging.

Analyzing the Heap

Once you have attached to the process, you can use the runtime object to inspect the contents of the GC heap:

        ClrHeap heap = runtime.GetHeap();
        foreach (ulong obj in heap.EnumerateObjects())
        {
            ClrType type = heap.GetObjectType(obj);
            ulong size = type.GetSize(obj);
            Console.WriteLine("{0,12:X} {1,8:n0} {2}", obj, size, type.Name);
        }
    

This produces output similar to the following:

         23B1D30       36 System.Security.PermissionSet
         23B1D54       20 Microsoft.Win32.SafeHandles.SafePEFileHandle
         23B1D68       32 System.Security.Policy.PEFileEvidenceFactory
         23B1D88       40 System.Security.Policy.Evidence
    

However, the original goal was to output a set of heap statistics. Using the data above, you can use a LINQ query to group the heap by type and sort by total object size:

        var stats = from o in heap.EnumerateObjects()
                    let t = heap.GetObjectType(o)
                    group o by t into g
                    let size = g.Sum(o => (uint)g.Key.GetSize(o))
                    orderby size
                    select new
                    {
                        Name = g.Key.Name,
                        Size = size,
                        Count = g.Count()
                    };

        foreach (var item in stats)
            Console.WriteLine("{0,12:n0} {1,12:n0} {2}", item.Size, item.Count, item.Name);
    

This will output data like the following -- a collection of statistics about what objects are taking up the most space on the GC heap for your process:

           564           11 System.Int32[]
           616            2 System.Globalization.CultureData
           680           18 System.String[]
           728           26 System.RuntimeType
           790            7 System.Char[]
         5,788          165 System.String
        17,252            6 System.Object[]
    

ClrMD Features and Functionality

Of course, there’s a lot more to this API than simply printing out heap statistics. You can also walk every managed thread in a process or crash dump and print out a managed callstack. For example, this code prints the managed stack trace for each thread, similar to what the SOS !clrstack command would report (and similar to the output in the Visual Studio stack trace window):

        foreach (ClrThread thread in runtime.Threads)
        {
            Console.WriteLine("ThreadID: {0:X}", thread.OSThreadId);
            Console.WriteLine("Callstack:");

            foreach (ClrStackFrame frame in thread.StackTrace)
                Console.WriteLine("{0,12:X} {1,12:X} {2}", frame.InstructionPointer, frame.StackPointer, frame.DisplayString);

            Console.WriteLine();
        }
    

This produces output similar to the following:

        ThreadID: 2D90
        Callstack:
                   0       90F168 HelperMethodFrame
            660E3365       90F1DC System.Threading.Thread.Sleep(Int32)
              C70089       90F1E0 HelloWorld.Program.Main(System.String[])
                   0       90F36C GCFrame
    

Each ClrThread object also contains a CurrentException property, which may be null, but if not, contains the last thrown exception on this thread. This exception object contains the full stack trace, message, and type of the exception thrown.

ClrMD also provides the following features:

  • Gets general information about the GC heap:
    • Whether the GC is workstation or server
    • The number of logical GC heaps in the process
    • Data about the bounds of GC segments
  • Walks the CLR’s handle table (similar to !gchandles in SOS).
  • Walks the application domains in the process and identifies which modules are loaded into them.
  • Enumerates threads, callstacks of those threads, the last thrown exception on threads, etc.
  • Enumerates the object roots of the process (as the GC sees them for our mark-and-sweep algorithm).
  • Walks the fields of objects.
  • Gets data about the various heaps that the .NET runtime uses to see where memory is going in the process (see ClrRuntime.EnumerateMemoryRegions in the ClrMD package).

All of this functionality can generally be found on the ClrRuntime or the ClrHeap objects, as seen above. IntelliSense can help you explore the various properties and functions when you install the ClrMD package. In addition, you can also use the attached sample code.

Please use the comments under this post to let us know if you have any feedback!

Attachment: ClrMDSample.cs
Leave a Comment
  • Please add 2 and 3 and type the answer here:
  • Post
  • @kevinlo2: It's on our list but we don't have a timeline yet.

  • This is great! I can see this going pretty big in just overall debugging.

    I still seem to be getting the "unable to attached to pid" error, though. Of course, I've only tried on the calc, notepad, and issexpress processes.

    Keep up the awesome work!

  • Finally got past an issue attaching to a running process! I'm not seeing any stack traces in the managed threads though.

  • This is awesome stuff - I can't remember the last time I installed a framework and had so many "are you serious???" moments.

    Very cool - keep up the good work!

  • This is awesome! Looking forward for further samples.

    I was looking to automate IIS app pool process memory dump / analysis and make it as self-service tool on our shared web farm. This sounds very promising for that.

  • Very cool stuff - looking forward to reading more about this.

    I'd be interested to see a way of grabbing more information about the objects or even the whole objects themselves.

  • I'm seeing odd behavior when trying to use retrieve native call stacks. When I try to use IDebugControl::GetStackTrace, it appears to not return all the frames in the stack.

    For example, if I retrieve the stack for thread zero in a sample dump file via IDebugControl::GetStackTrace in ClrMD, I get the following frames:

       ############ Frames for thread 0 [B128] ############

       [0]: 7C82845C ntdll!KiFastSystemCallRet

       [1]: 77E61C8D kernel32!WaitForSingleObject

       [2]: 5A364662 w3dt!IPM_MESSAGE_PIPE::operator=

       [3]: 0100187C w3wp

       [4]: 01001A27 w3wp

       [5]: 77E6F23B kernel32!ProcessIdToSessionId

    If I look at the stack in Visual Studio or WinDbg, or if I retrieve it using IDebugControl::GetStackTrace in a WinDbg extension, I get the following:

       ############ Frames for thread 0 [B128] ############

       [0]: 7C82845C ntdll!KiFastSystemCallRet

       [1]: 7C827B79 ntdll!ZwWaitForSingleObject        <<<< Skipped by ClrMD

       [2]: 77E61D1E kernel32!WaitForSingleObjectEx   <<<< Skipped by ClrMD

       [3]: 77E61C8D kernel32!WaitForSingleObject

       [4]: 5A364662 w3dt!WP_CONTEXT::RunMainThreadLoop

       [5]: 5A366E3F w3dt!UlAtqStartListen                    <<<< Skipped by ClrMD

       [6]: 5A3AF42D w3core!W3_SERVER::StartListen    <<<< Skipped by ClrMD

       [7]: 5A3BC335 w3core!UlW3Start                          <<<< Skipped by ClrMD

       [8]: 0100187C w3wp!wmain

       [9]: 01001A27 w3wp!wmainCRTStartup

       [10]: 77E6F23B kernel32!BaseProcessStart

    Note that all the frames listed by ClrMD exist in the true call stack, but it has skipped the indicated frames in between. Have you seen this behavior before?

    The code I'm using looks like this:

       DataTarget dataTarget = DataTarget.LoadCrashDump(@"c:\scratch\mydump.dmp");

       // Not actually using the ClrRuntime in this snippet, but my actual code is doing this,

       // so I'm including it in case it matters.

       ClrInfo clrInfo = dataTarget.ClrVersions[0];

       string dacLocation = clrInfo.TryGetDacLocation();

       ClrRuntime clrRuntime = dataTarget.CreateRuntime(dacLocation) ;

       // Retrieve the required debugging interfaces

       IDebugControl4 control = (IDebugControl4) dataTarget;

       IDebugSystemObjects3 sysObjs = (IDebugSystemObjects3) dataTarget;

       sysObjs.SetCurrentThreadId(0);

       DEBUG_STACK_FRAME[] frames = new DEBUG_STACK_FRAME[100];

       uint frameCount = 0;

       control.GetStackTrace(0, 0, 0, frames, 100, out frameCount);

       // Note: after the call, frameCount is set to 6, instead of 11, like it should be

    As you may have noticed from my output above, I'm also having some issues with symbol resolution via IDebugSymbols::GetNameByOffset where occasionally I'm getting incorrect or incomplete names; but, I'm hoping that this is just something I have wrong in the code that sets up the symbol path.

  • Slight typo in the above code sample. These two lines:

      IDebugControl4 control = (IDebugControl4) dataTarget;

      IDebugSystemObjects3 sysObjs = (IDebugSystemObjects3) dataTarget;

    should have read

      IDebugControl4 control = (IDebugControl4) dataTarget.DebuggerInterface;

      IDebugSystemObjects3 sysObjs = (IDebugSystemObjects3) dataTarget.DebuggerInterface;

  • Great... Love this.  One question.  I have a simple program that allocates a List<> and adds instances of a class called PayLoad (see code below).  After the first time through the while loop, one Payload instance is allocated and added to the List<PayLoad>.  When I dump all the heap allocated objects from my test program's namespace I see:

    Name:                                            Total Size:                         Total Number:

    TestProgram.PayLoad[]                    96                                     2

    TestProgram.PayLoad                      24                                     1

    Does anybody have any idea where the TestProgram.PayLoad[] instances come from?  I'm trying to measure memory usage of my namespace object and this seems to skew it a bit.  Thanks.

    class Program

       {

           static void Main(string[] args)

           {

               List<PayLoad> payloadList = new List<PayLoad>();

               while (true)

               {

                   Console.WriteLine("Adding new payload");

                   payloadList.Add(new PayLoad());

                   Console.ReadLine();

               }

           }

       }

       class PayLoad

       {

           public int a;

           public int b;

       }

  • I tried to run the sample to analyze .dmp file which was taken from a program running on the same machine as the sample.

    but i keep getting the following exception when trying to create the runtime object:

      Message: Failure loading DAC: CreateDacInstance failed 0x80131c30

      at Microsoft.Diagnostics.Runtime.Desktop.DacLibrary.Init(String dll)

      at Microsoft.Diagnostics.Runtime.Desktop.DacLibrary..ctor(DbgEngTarget dataTarget, String dll)

      at Microsoft.Diagnostics.Runtime.DbgEngTarget.CreateRuntime(String dacFilename)

      at DumpFetch.App..ctor()

      at DumpFetch.App.Main()

      at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)

      at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)

      at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()

      at System.Threading.ThreadHelper.ThreadStart_Context(Object state)

      at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean ignoreSyncCtx)

      at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)

      at System.Threading.ThreadHelper.ThreadStart()

    Any ideas?

  • I'm analyzing 11 GB dmp file and top size type is "Free". What does "Free" means?

  • >> I'm analyzing 11 GB dmp file and top size type is "Free". What does "Free" means?

    Free is a pseudo object that represents a free space (a hole) in the GC heap.   They exist when the GC happens but decides not to compact.

    Having large amounts of fee space is not necessarily bad (especially if they are a few big chunks), since these to get reused.     When these free areas get too numerous/small, the GC will compact.  Large objects (> 85K), are treated differently, and placed in their own region of memory and currently are not ever compacted (however in V4.5.1 we have added the ability to do this explicitly)

  • Is any way to find using ClrMD that some object is reachable from roots or not reachable from roots?

  • > Is any way to find using ClrMD that some object is reachable from roots or not reachable from roots?

    Yes.  In fact, this is how PerfView uses ClrMD, but you have to calculate the object graph manually to find that information.  There's not a simple function call to do this.

    The functions which you use to do most of the work are:

    - ClrHeap.EnumerateRoots - To get the roots.

    - ClrHeap.GetObjectType - To get the type of the object in the root.

    - ClrType.EnumerateRefsOfObject - To enumerate what objects the current object references.

    With these functions you build the full heap graph...and any object not in the graph is considered dead.  Any object you do reach is considered live.

    (There are false positives and negatives from this approach, but they are rare.  We unfortunately aren't 100% accurate in the root reporting for all versions of the runtime.)

  • Thank you, Lee Culver!

    It helped me find cause of memory leak.

Page 2 of 3 (39 items) 123