If broken it is, fix it you should

Using the powers of the debugger to solve the problems of the world - and a bag of chips    by Tess Ferrandez, ASP.NET Escalation Engineer (Microsoft)

I have a memory leak!!! What do i do? (defining the "where")

I have a memory leak!!! What do i do? (defining the "where")

  • Comments 17


Not that it matters a tremendous lot but just because it is a big pet-peeve of mine I want to differentiate between a real memory-leak and high memory usage.

A memory leak is when you used some memory and lost the pointer to the allocation so you can no longer de-allocate that memory.  If you still have a pointer to it, you have high-memory usage, which might be just as bad in terms of what happens to the process but still differentJ Just to make things simple I’ll use the term memory leak for both of these from now on…

I like to think of solving performance and memory leak issues like peeling an onion. You take away one layer at a time, starting with the most obvious one, and then you define a limit where you determine that the problem is resolved.

There are two types of memory leaks, the gradual memory leak (where memory continuously grows at approximately the same rate) and the sudden memory jump.  You troubleshoot them approximately the same way, except for in the latter case you also try to figure out if something out of the ordinary happened at the time of the jump, such as extreme load on the server etc.

So where are you leaking? As I mentioned in my previous post the process’ memory space contains a few different types of “objects” such as threads, manged object heaps, managed loader heap, native heaps, dll’s, and virtual allocations, so a good place to start is by running a performance monitor log with the following counters.

   Process/Virtual Bytes
   Process/Private Bytes
   .net CLR Memory/# Bytes in all Heaps
   .net CLR Memory/% Time in GC
   .net CLR Memory/Large Object Heap size
   .net CLR Loading/Bytes in Loader Heap
   .net CLR Loading/Current Assemblies

The main thing to look for is if private bytes grow at approximately the same rate as virtual bytes and if # Bytes in all Heaps seem to follow the same curve.

If the private bytes keep increasing but # Bytes in all Heaps do not, you’re likely looking at a native memory leak (i.e. you are leaking in a COM component or similar) but if # Bytes in all heaps increase at the same rate as private bytes your leak is likely in managed code.

Much the same way, if you see a steady increase of virtual bytes but your private bytes stay pretty steady, your application probably has a problem where it is reserving a lot of virtual memory that it’s not using.  

Bytes in Loader Heap and Current Assemblies should stay fairly constant once the process has started up and all app domains are loaded. If this keeps continuously increasing it is very probable that you have an assembly leak. I will write more about this later but for now, if it is an asp.net application, check that debug=false in all web.configs and that you are not running into one of the following issues.

Memory usage is high when you create several XmlSerializer objects
   or
Assembly leak because of script blocks in XSLT's

So now you know approximately where you are leaking, the next step is to find out why and I will talk more about this in future posts but knowing the “where” is a big step.





  • did we ever get to the future posts ... "why is memory leaking" .. i am interested in native memory leak

    Thanks,
    Maur
  • There are various case studys in here showing why memory is leaking, no native memory leak yet though... I haven't really decided on a good one to show native debugging techniques with, but you can start by downloading the IIS Diagnostic Toolkit  and run DebugDiag (a "self diagnosing" tool), which works for other processes than IIS as well

    http://www.microsoft.com/downloads/details.aspx?FamilyID=9BFA49BC-376B-4A54-95AA-73C9156706E7&displaylang=en

    /Tess
  • The following are based on problems I see people having every day in production ASP.NET systems.
    1....
  • I am particulary interested in investigating native memory leaks. We have one, and would like some pointers on how to g about pinpointing the source. Is there a posting/blog that deals with memory leaks in unmanaged code?
  • I am particulary interested in investigating native memory leaks. We have one, and would like some pointers on how to g about pinpointing the source. Is there a posting/blog that deals with memory leaks in unmanaged code?
  • I will try to post something on that shortly, in the meantime take a look at the iis diagnostic toolkit  (available at http://www.microsoft.com/windowsserver2003/iis/diagnostictools/default.mspx)  specifically the debug diagnostics tool,  that is what I normally use for un-managed leaks.

  • From the article:

    "Much the same way, if you see a steady increase of virtual bytes but your private bytes stay pretty steady, your application probably has a problem where it is reserving a lot of virtual memory that it’s not using."

    Can you give an example of how an application reserves memory, but does not use it?  

    We have this issue (VM increasing , private bytes steady). We create large arrays, but how is "creating" differentiated from "using"?

  • If someone calls virtualalloc without comitting the memory, that would do the trick

    http://msdn2.microsoft.com/en-us/library/aa366887.aspx

    In general it would be one of a few things

    1. A native component calling virtualalloc and leaving most of it uncommitted

    2. if you see it in the start of the process it might be that you have many processors and thus create a lot of managed heaps, but you havent allocated too much .net memory yet

    it could be other things too but those are probably the most common.  I would use DebugDiag and enable leaktracking to see who is allocating it.

  • The following are based on problems I see people having every day in production ASP.NET systems. 1 .

  • W poniższym artykule wykorzystano materiały znajdujące się w następujących artykułach Tess: http://blogs.msdn.com/tess/archive/2005/11/25/i-have-a-memory-leak-what-do-i-do-defining-the-where.aspx

  • Hi,

    I have encountered similar problem of increasing private bytes. So I have taken a dump and analyzed using windbg.

    In order to fing the root of particular object I ran !gcroot . I got following output

    0:000> !gcroot 185b0438

    Scan Thread 0 (10a0)

    Scan Thread 2 (16cc)

    Scan Thread 3 (13fc)

    Scan Thread 7 (e80)

    Scan Thread 4 (113c)

    Scan Thread 30 (1220)

    Scan Thread 32 (159c)

    Scan Thread 31 (ac0)

    Scan Thread 33 (118)

    Scan Thread 43 (17e4)

    Scan HandleTable 41b0048

    Scan HandleTable 2022a0

    HANDLE(Strong):43911c0:Root:054b2000(System.Object[])->044c0d8c(ABC.FeatureCollection)->185b27e8(System.Collections.ArrayList)->185b2800(System.Object[])->00000000()

    Does it mean that system.object is pointing to null value and reference to the object is not disposed.

  • Hi Tess,

    You have posted information about debugging native and unmanaged code. Can you post the links for debugging techniques for Managed code?

  • Sunny,  basically this whole blog is about debugging managed code,  for memory issues, check out the post index for the articles under "memory issues"

  • Sakar, that's some seriously strange output, I am guessing it is some kind of problem with sos...  i wouldnt trust it and it is irrelevant for you as the object you are gcrooting is not showing up here

  • About memory leaks, for diagnosing a problem I'd like to get memory address value of a handle.

    Do you know how to get that? Obliviously it's not something one should rely on to remain constant over time, but for debugging/logging it could be useful (for example: log it to see if two threads are dealing with the same object)

    Since the debugger can retrieve this value, there is probably an interface (ICorDebug?) for it :)

    Object ^ anObject; as the debugger shows it (like 0x01223AA) it is probably the memory address of the GCHandle of anObject.

    it is not toString() which equals "System::Object"

    it is not getHashCode()

    it is not lAddress in (which probably makes sense, since it's probably the address of the newly allocated handle)

       GCHandle lHandle = GCHandle::Alloc(aObject);

       int lAddress = GCHandle::ToIntPtr(lHandle).ToInt32();

       lHandle.Free();

    anObject cannot be casted to System::Int32 using dynamic_cast (exception)

Page 1 of 2 (17 items) 12
Leave a Comment
  • Please add 8 and 8 and type the answer here:
  • Post