Spat's WebLog (Steve Patrick)

When things go wrong...

Memory leaks...

Memory leaks...

  • Comments 7

I was all set to write up a great post on memory leaks, how to track them down etc.. blah blah blah.. but then I found this little gem http://support.microsoft.com/kb/268343

Heck I didnt even know we did KB's with video - just shows how much I know.

Anyway, this article and video does a great job of explaining how to use the stack trace report and the UMDH tool itself.

We use this a lot here and wanted to share the love a bit.

Just to add a few more items that are helpful in tracking mem leaks.

1. Always get a perfmon log . Are you sure it is a memory leak? Are there other items which follow it? LIke threads? Or handles?  -- perfmon is a nifty tool for confirming things like this.

2. Do NOT restart the process you are tracking - I had one fellow give me a bunch of umdh logs and each one had a different PID - like this:

// UMDH: version 6.0.5072.0: Logtime 2005-11-02 10:13 - Machine=BadMachine - PID=968
//

// OS version 5.2 Service Pack 1
// Umdh OS version 6.0
// Debug privilege has been enabled.
// Connecting to process 968 ...
// Process 968 opened (handle=2000) ...
// Debug options set: 00000007
// Debug library initialized ...
// Module enumeration completed.
// Reading target process trace database ...
// Database size 01000000
// Trace database read.
// Stack trace data base @ 01010000
// # traces in the data base 31039

 Then

// UMDH: version 6.0.5072.0: Logtime 2005-11-02 10:13 - Machine=BadMachine - PID=845

// UMDH: version 6.0.5072.0: Logtime 2005-11-02 10:13 - Machine=BadMachine - PID=912

Don't do this :) - kinda hard to track a leak when you keep restarting the app..

3. If you can, get a userdump of the process as well when you are ready to stop collecting UMDH logs and ready to analyze. Why? The raw stack data is in there and there are times when it is useful to have.

spat

Leave a Comment
  • Please add 7 and 5 and type the answer here:
  • Post
  • Thanks, this kb article is great. This is the first tine I saw video in a kb article as well, I hope they do more.

  • Say you have a leak on a customer machine.  Your company's .pdb files are accessible via VPN, but if your IT guys won't let a customer machine in, is there a way to get 'raw' stack traces for the allocations, bring them back into your network, and then let UMDH interpret them?

  • UMDH cant do this - you basically end up with something like:

    Allocations for trace BackTrace8502420:

    009A0210  

    +   52326 (  52326 -      0)      1 allocs BackTrace8509272

    +       1 (      1 -      0) BackTrace8509272 allocations

           76FEDAB4

           76799B12

           767A30B3

           767A2FBC

           767A2F0C

           767A2319

           767A1C58

           767A1BC0

           76799A97

           76799A61

           76799D04

           770D140F

           770D159E

           770D1488

           7710BEB3

           7710BE45

           77109D68

           7710BC31

           7710BEDC

           7710C156

           74608CFB

           74608EDB

           74609F98

           74600563

           7465188D

           77111A10

           77111AE8

           77111C03

           77113656

           76FE0E6E

           7712D25C

           77101460

    You can do a few things - you can use SymFromAddr - http://msdn2.microsoft.com/en-us/library/ms681323.aspx

    Or you can use the built in debugger functionality - when you get back to corp net - load up the process under the debugger, find some free mem to edit and place the addresses in there. Then do a DPS on it and it will resolve them all for you.

    Cant think of an easier way - except, since it sounds like you are on site bring the PDB's with you  :)

  • Ah, I see.  So if I don't have a CD with my symbols at the customer site (I'm not physically there), and I'm forbidden to connect from there to my network using VPN, I could get a minidump to go with my UMDH files.

    I tried this out.  I built this to get leaky.exe and a matching leaky.pdb file,

    1 #include <stdio.h>

    2 #include <stdlib.h>

    3

    4 void leak(void)

    5 {

    6  malloc (0x999);

    7 }

    8

    9 int main (int argc, char **argv)

    10 {

    11  char line[80];

    12  while (fgets (line, sizeof(line), stdin))

    13   leak();

    14  return 0;

    15 }

    I hid the .pdb file away.  Now I can pretend I'm at a customer site.  No symbols for leaky.exe.  I get some UMDH snapshots and a minidump, just enough of the process so I can resolve addresses later.  Leaky.exe continues to run.

    - gflags -i Leaky.exe +ust

    - (start leaky, say it's pid 9999)

    - umdh -p:9999 -f:001.txt

    - (hit return to make leaky leak)

    - umdh -p:9999 -f:002.txt

    - (hit return to make leaky leak)

    - umdh -p:9999 -f:003.txt

    - Attach WinDbg to PID 9999 noninvasive,

    - .dump C:\Leaky.mdmp,

    - quit

    OK, now time to pretend I've gone home to my symbols.  The files 001.txt, 002.txt, 003.txt all contain hex addresses that UMDH couldn't interpret (no symbols).  Leaky.mdmp is a minidump I'll use below.  Umdh -v 001.txt 003.txt shows a leaky stack:

     ntdll!RtlDebugAllocateHeap+000000E1

     ntdll!RtlAllocateHeapSlowly+00000044

     ntdll!RtlAllocateHeap+00000E64

     MSVCRT!_heap_alloc+000000E0

     MSVCRT!_nh_malloc+00000013

     MSVCRT!malloc+00000027

     leaky!???+00000000 : 0040102E

     leaky!???+00000000 : 00401057

     leaky!???+00000000 : 0040115D

     kernel32!BaseProcessStart+00000023

    Let's resolve those addresses:

    - Open WinDbg

    - set symbol path to point to my symbols

    - Open crash dump, Leaky.mdmp

    - ln (List Nearest Symbols) for each address in the leaky stack trace:

    0:000> ln 0040115D

    (0040107a)   leaky!mainCRTStartup+0xe3   |  (0040118a)   leaky!exit

    0:000> ln 00401057

    leaky.c(13)+0x5

    (00401033)   leaky!main+0x24   |  (0040106e)   leaky!malloc

    0:000> ln 0040102E

    leaky.c(6)+0xb

    (00401020)   leaky!leak+0xe   |  (00401033)   leaky!main

    So now I have:

     ntdll!RtlDebugAllocateHeap+000000E1

     ntdll!RtlAllocateHeapSlowly+00000044

     ntdll!RtlAllocateHeap+00000E64

     MSVCRT!_heap_alloc+000000E0

     MSVCRT!_nh_malloc+00000013

     MSVCRT!malloc+00000027

     leaky!leak+0xe    leaky.c(6)+0xb

     leaky!main+0x24   leaky.c(13)+0x5

     leaky!mainCRTStartup+0xe3  

     kernel32!BaseProcessStart+00000023

    And here's what UMDH would have given me if I'd had symbols from the start:

     ntdll!RtlDebugAllocateHeap+000000E1

     ntdll!RtlAllocateHeapSlowly+00000044

     ntdll!RtlAllocateHeap+00000E64

     MSVCRT!_heap_alloc+000000E0

     MSVCRT!_nh_malloc+00000013

     MSVCRT!malloc+00000027

     leaky!leak+0000000E

     leaky!main+00000024

     leaky!mainCRTStartup+000000E3

     kernel32!BaseProcessStart+00000023

    I wouldn't have hit on this without your help.  Thanks.

    -Robert

  • We are testing our Windows Server based product on Longhorn. During stress testing, we see what appears to be a fairly significant memory leak (over >48 hrs). Look at the first stack trace. There is only one function call there - ntdll!RtlAllocateHeap+0000021D, so it does not tell you what triggered the leak. Is this because UMDH could not walk the stack? (due to a some loaded module that uses FPO?). How can you get around this?

    + 36686968 ( 40772037 - 4085069)  13723 allocs BackTrace162528

    +    4185 (  13723 -   9538) BackTrace162528 allocations

    ntdll!RtlAllocateHeap+0000021D

    +  811504 ( 811504 -      0)      1 allocs BackTraceD4354F4

    +       1 (      1 -      0) BackTraceD4354F4 allocations

    ntdll!RtlAllocateHeap+0000021D

    pdh!PdhiHeapAlloc+00000029

  • To be clear: I am talking about BackTrace162528

    and not BackTraceD4354F4 (which has the full call stack, just not including it in the comment).

  • Ya this is a problem with UMDH  - you can try leakdiag here : ftp://ftp.microsoft.com/PSS/Tools/Developer%20Support%20Tools/LeakDiag

    It's a litle different but pretty straightforward in usage.

    steve

Page 1 of 1 (7 items)