If broken it is, fix it you should

Using the powers of the debugger to solve the problems of the world - and a bag of chips    by Tess Ferrandez, ASP.NET Escalation Engineer (Microsoft)

.NET Finalizer Memory Leak: Debugging with sos.dll in Visual Studio

.NET Finalizer Memory Leak: Debugging with sos.dll in Visual Studio

Rate This
  • Comments 10

Normally I write about issues that only manifest themselves in production environment, issues that you can't really reproduce in a controlled dev environment every time you perform a certain action.  In those cases you need to use tools like windbg to gather dumps and do post-mortem debugging.

Windbg works really well for those types of issues, but it has its shortcommings since it is not really a managed debugger so it is much harder to set breakpoints in .NET code or step through code, or even inspect objects in a visual way like you can in a managed debugger like Visual Studio.

Visual Studio on the other hand doesn't allow you to do post-mortem debugging the same way windbg does, and there is no easy way to view information about the domains loaded in the process or to view information about the objects on the .net heaps.  (For some links on post-mortem debugging with Visual Studio, see Peters comment to this post)

My colleague had an issue that was pretty easily reproducible but he needed both worlds, i.e. stepping through code to a specific point, looking at the objects on the stack visually and at the same time he needed to view the contents on the heap, so he resorted to having two debuggers attached, visual studio debugging managed, and windbg debugging native with sos to view the managed heap.  That is pretty nasty, and there is a much easier way to combine these two worlds... debugging with sos in Visual Studio.

To illustrate how it works i'm using a sample from Ingo Rammer.

If you want to follow along you can download the sample code here under the link Hardcore Production Debugging.   The specific sample I am using is FinalizerProblem which is basically the winforms equivalent of my post on Unblock my Finalizer.

Problem description:

When I click on the button "Do Work" it creates a number of instances of the class MyBusinessObject.  Even though I know that the objects should no longer be referenced after that they don't appear to go away even if I invoke a GC.Collect() with GC.WaitForPendingFinalizers().  Why are my objects not released?

Debugging the issue:

In this case we could easily attach windbg and use sos.dll per my post above to figure out that the reason these objects are sticking around is because of a blocked finalizer, but I will use Visual Studio in order to show you how to load up sos in it and run sos commands.

Step 1: Enable Native Debugging for the project

In order to load an extension like SOS.dll you have to be debugging in native mode, so before starting the debugger go into Project/Properties/Debug on the context menu for the project and check the box for Enable unmanaged code debugging.

Step 2: Debug and Break

Debug the problem as you normally would in Visual Studio until you have reproduced the issue (i.e. in this case click on "Do Work" to instanciate the objects, followed by "Run GC" to perform the garbage collection.

Break into the process (Debug menu/Break All)

Step 3: Load sos

In order to load sos.dll you have to open up the Immediate Window (Debug/Windows/Immediate or Ctrl+D, I) and type

.load C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\sos.dll

This should yield the response

extension C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\sos.dll loaded

Step 4: Debug with sos

Now we are ready to debug with sos.dll.

There are a few rules here...

1. You can not run native commands like kb, dc etc.

2. You can not run ~* e which means you cant run ~* e !clrstack to see the stacks on all threads

You can however run all the commands in sos.dll like !dumpdomain, !dumpheap etc.

To switch the thread context for thread specific commands like !clrstack and !dumpstackobjects you can open the threads window (debug/windows/threads) and doubleclick the thread you want to switch to.  I will show an example of that later...

 

The first thing we want to do is find our objects on the heap

!dumpheap -type MyBusinessObject
PDB symbol for mscorwks.dll not loaded
 Address       MT     Size
027437e4 01d6683c       12     
02743830 01d6683c       12     
0274387c 01d6683c       12     
...
02747d6c 01d6683c       12     
02747db8 01d6683c       12     
02747e04 01d6683c       12     
02747e50 01d6683c       12     
02747e9c 01d6683c       12     
02747ee8 01d6683c       12     
total 30 objects
Statistics:
      MT    Count    TotalSize Class Name
01d6683c       30          360 FinalizerProblem.MyBusinessObject
Total 30 objects

Then we can grab one of those objects and run !gcroot on it to find out why it is still around

!gcroot 02747d6c
Note: Roots found on stacks may be false positives. Run "!help gcroot" for
more info.
Error during command: warning! Extension is using a feature which Visual Studio does not implement.

Scan Thread 7092 OSTHread 1bb4
Scan Thread 6864 OSTHread 1ad0
Finalizer queue:Root:02747d6c(FinalizerProblem.MyBusinessObject)

In this case it is rooted in the finalizer queue which means it is waiting to be finalized and if we look at the finalizequeue we can see that we have 69 objects that are waiting to be finalized so the question that remains is why we aren't finalizing them...

!finalizequeue
SyncBlocks to be cleaned up: 0
MTA Interfaces to be released: 0
STA Interfaces to be released: 0
----------------------------------
generation 0 has 0 finalizable objects (002906d0->002906d0)
generation 1 has 36 finalizable objects (00290640->002906d0)
generation 2 has 0 finalizable objects (00290640->00290640)
Ready for finalization 69 objects (002906d0->002907e4)
Statistics:
      MT    Count    TotalSize Class Name
7b47f8f8        1           20 System.Windows.Forms.ApplicationContext
...
7910b694       10          160 System.WeakReference
7b47ff4c        4          224 System.Windows.Forms.Control+ControlNativeWindow
01d6683c       22          264 FinalizerProblem.MyBusinessObject
01d65a54        1          332 FinalizerProblem.Form1
7b4827e8        2          336 System.Windows.Forms.Button
7ae78e7c        8          352 System.Drawing.BufferedGraphics
...
Total 105 objects

If we were debugging in windbg, the next natural step would have been to run !threads, figure out which one was the finalizer and look at what is is running.

Since we are in Visual Studio instead we can open the threads window, navigate to the thread that is performing Finalization and look at what it is doing.  If you can't determine that by the function that is currently on the top of the user-code part of the callstack you can still identify it with !threads.

The cool thing here is that we jump straight into the code, where it is blocking and can use everything we are used to in Visual Studio like the watch and locals window or stepping in code for example, but without loading up sos.dll we would not have been able to track down why our MyBusinessObject instances were still around.

Final words:

I know many people are a bit adversed to starting with windbg since it means you need to learn a whole new toolset and perfectly honestly windbg is not as "beginner friendly" as for example visual studio is, then again, its meant to be used mostly for post-mortem debugging which is not exactly an everyday task for most people.

Hopefully though with the info above you can still begin to use sos.dll in the cozy and familiar visual studio debugging environment.

Speaking of making things a bit more user-friendly and visual,  Ingo Rammer who wrote the demo has developed a tool called SOSAssist which is kind of a GUI interface to sos.dll.

 

Until next time,

Tess





  • Maybe I misundestood the context when you said "Visual Studio on the other hand doesn't allow you to do post-mortem debugging".  The following are a sample couple of links that describe post-mortem (crash dump) debugging with Visual Studio:

    http://msdn2.microsoft.com/en-us/library/d5zhxt22.aspx

    http://www.codeproject.com/debug/postmortemdebug_standalone1.asp

  • Reading your posts is like passing by a horrible car wreck.  You know you don't want anything to do with what's going on in the crash scene, but goddamn if you don't have to take a peek.  Also, you never want to be in the middle of one of them... but you might.

  • NoVa Code Camp 2007.02 [Via: Jeff Schoolcraft ] EarthClassMail.com - Moving from LAMP to .NET 3.5 [Via:...

  • Thanks mcgurk,

    That would have to be the weirdest compliment I've ever gotten:)

  • Hi Peter,

    No you didnt missunderstand, I just phrased it wrong...

    You can do limited dump analysis in visual studio, and to be honest it works pretty good, but given how hard it is to get to the native parts and run commands like ~*  etc. it's almost a can't.  

    I'll edit the post to reflect this... thanks for the comment...

  • SOSAssist saved my life.

    Thanks for sharing it!

  • Just one question:

    Why can we 'load' some internal windbg from VS so that we can execute some useful windbg commands ?

  • I still have to figure out a way to host the video from the break-out session but until then, here are

  • This article saved my life (job)! Thank you.

  • I get this in VS 2010 :

    .load \\devsrv\symstore\framework\v4.0.30319.544\sos.dll

    extension \\devsrv\symstore\framework\v4.0.30319.544\sos.dll loaded

    !threads

    PDB symbol for clr.dll not loaded

    Yet in module list shows the pdbs for a CLR.DLL have been loaded. Is it a version issue?

    All subsequent commands echo a blank line in the immediate window.

    The clr version seems to be correct as when I load the same dump file in WinDbg it shows this:

    0:008> !eeversion

    4.0.30319.544 retail

    Workstation mode

    SOS Version: 4.0.0.4 retail build

    Any ideas?

Page 1 of 1 (10 items)
Leave a Comment
  • Please add 4 and 1 and type the answer here:
  • Post