Hello debuggers, the debug ninja is back again. Sometimes we have a scenario where a process is using a lot of memory, and the only data we are able to get at the moment is a user dump. Ordinarily data from tools such as umdh or xperf would be preferable because they provide memory usage data over a period of time and can include call stack information. However, umdh requires restarting the process (which loses the state of high memory usage), and xperf requires the installation of the Windows Performance Toolkit which may not always be an immediate option.
When we have such a dump we may not be able to specifically identify what piece of code is generating the high memory usage, but we may be able to narrow the scope of troubleshooting to a specific dll.
The first thing we need to do is identify what type of memory is using most of the address space. The debugger command !address –summary allows us to do this:
0:000> !address -summary
--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free 489 7fe`6ff5a000 ( 7.994 Tb) 99.92%
Heap 9094 1`75ed1000 ( 5.843 Gb) 93.47% 0.07%
<unknown> 275 0`12e41000 ( 302.254 Mb) 4.72% 0.00%
Image 937 0`05a6a000 ( 90.414 Mb) 1.41% 0.00%
Stack 138 0`01700000 ( 23.000 Mb) 0.36% 0.00%
Other 14 0`001bd000 ( 1.738 Mb) 0.03% 0.00%
TEB 46 0`0005c000 ( 368.000 kb) 0.01% 0.00%
PEB 1 0`00001000 ( 4.000 kb) 0.00% 0.00%
From this example we can see that most of the memory is used by heap. A process will usually have multiple heaps, each created by a call to HeapCreate. We can examine the size of each of these heaps with !heap –s:
0:000> !heap -s
LFH Key : 0x0000006c1104d280
Termination on corruption : ENABLED
Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast
(k) (k) (k) (k) length blocks cont. heap
0000000000100000 00000002 16384 12824 16384 1180 254 5 0 3 LFH
0000000000010000 00008000 64 4 64 1 1 1 0 0
00000000003d0000 00001002 1088 708 1088 121 20 2 0 0 LFH
0000000003080000 00001002 1536 700 1536 4 4 2 0 0 LFH
00000000033a0000 00001002 5229696 1377584 5229696 414244 4039 3059 0 2c LFH
External fragmentation 30 % (4039 free blocks)
Virtual address fragmentation 73 % (3059 uncommited ranges)
0000000003380000 00001002 64 8 64 3 1 1 0 0
0000000003600000 00001002 512 56 512 3 1 1 0 0
0000000003c20000 00001002 512 8 512 3 1 1 0 0
0000000003220000 00001002 512 8 512 3 1 1 0 0
0000000003e50000 00001002 512 8 512 3 1 1 0 0
0000000003d00000 00001002 512 148 512 5 3 1 0 0 LFH
From the above output we can see that most of the memory is being used by heap 00000000033a0000.
At this point we need to try to identify what this heap is used for. A brute force method to do this is to search memory with the ‘s’ command.
0:000> s -q 0 l?7fffffffffffffff 00000000033a0000
000007fe`f21810a0 00000000`033a0000 00000000`00000001
The output of the ‘s’ command may be verbose. You will need to manually examine the addresses where ‘s’ finds hits. Most of these addresses will probably be in heap memory, we are looking for an address that matches a module. I snipped the above output to just show the relevant hit, an address inside of a loaded module.
The search of memory revealed that the heap 00000000033a0000 is used by the module useheap.dll, specifically it is part of the global ‘Blob’ class.
0:000> ln 000007fe`f21810a0
At this point we don’t know specifically what code in useheap.dll has allocated a lot of heap, however we have significantly narrowed the scope of the problem. We can now determine if there is a known issue with heap usage in useheap.dll that is addressed in a later version. We may also know from experience that this module uses a lot of memory under specific circumstances, such as a high volume of work sent to this service.
I hope this example helps the next time you have high memory usage and only have a user dump to troubleshoot with. Good luck!
Every Wednesday (usually) we post a debug tip to our twitter page at https://twitter.com/#!/ntdebugging. This blog is an archive of these tips to allow our readers to find this information easily. Periodically we post an updated blog with the current archive. Follow us on twitter if you want to see the new tips as we post them.
The goal of these tips is to share debug commands, and forms of commands (parameters, flags, etc) that we in Platforms Global Escalation Services find useful. I hope you can add these commands to your toolkit and they will help you debug more efficiently.
!thread/!process [address] e - on x64 will not show you the meaningless Args to Child information.
.frame /c [FrameNumber] - sets context to specificied stack frame. On x64 provides more reliable register information than .trap.
kn - Dumps call stack with frame numbers, easier than counting stacks for .frame.
.frame /r [FrameNumber] - same as .frame /c, but shows registers without changing context.
Note: With .frame /c or /r you can only trust the nonvolatile registers. See http://bit.ly/dik4OR for vol/nonvol regs.
k=rbp rip FrameCount - Dumps call stack starting at rbp/rip on x64. Useful when the stack is corrupt.
.process/.thread /p /r [address] - sets new process context, sets .cache forcedecodeuser, and reloads user symbols.
!process [address] 17 - Sets the context for this command, avoids the need for .process to see user stacks. Try !process 0 17
~~[ThreadID]s - Changes threads in user mode. Use Thread ID number from output such as !locks. Ex: ~~[1bd4]s
runas /netonly /u:<account> windbg.exe - Launch windbg with domain account. Use when dbg computer isn't in domain and symbol server is.
!heap -p -a <address> - Shows information about the heap block containing <address>, even if you aren't using pageheap.
ub - Unassembles starting at a location prior to your address. Accepts l<number> to specify how many instructions to go back. ub . l20
!stacks 2 [FilterString] - Finds kernel mode call stacks that contain the FilterString in a symbol.
!thread [address] 17 (or 1e on x64) - Sets context for this command, avoids the need for .thread/.process for user stacks.
.hh [Text] - Opens the debugger help. [Text] is the topic to lookup in the index. Example: .hh !pte
?? can dump structs using C++ style expressions. Ex: ??((nt!_KTHREAD*)(0xfffffa800ea43bb0))->ApcState
bp /t EThread - Sets a kernel mode breakpoint that only triggers when hit in the context of this thread.
bp /p EProcess - Sets a kernel mode breakpoint that only triggers when hit in the context of this process.
gc - If you run 'p' and hit a breakpoint, gc takes you where p would have gone if you had not hit the bp.
gu - Go until the current function returns. Effectively this unwinds one stack frame. #windbg
pc - Steps through until the next 'call' instruction. Combine with other commands to find who returned your error> pc;p;r eax
pt - Steps through until the next 'ret' instruction. Similar to gu, but pt stops on the ret and gu stops after the ret.
.ignore_missing_pages 1 - supresses the error: "Page 2a49 not present in the dump file. Type ".hh dbgerr004" for details"
.exr -1 shows the most recent exception. Useful in user dumps of crashes, especially for no execute crashes (NX/DEP).
wt - Trace calls until they return to the current address. More useful with -or to get return values. Use -l for depth.
.thread /w - Changes to the WOW64 32-bit context from 64-bit kernel mode. Wow64exts doesn't work in kernel mode.
??sizeof(structure) - Gets the size of a structure, it's easier than counting.
sxe ld:module.dll - Enables an exception which will break into the debugger when module.dll is loaded.
vertarget - Shows OS version of the debug target. Also shows machine name, uptime, and session time (when the dump was taken).
!vm 1 - In a kernel debugger, shows basic information about memory usage. Available, committed, pagefile, pool, sysptes, etc.
.time - Shows session time (when dump was taken) and system uptime. In user mode shows process uptime, kernel/user time.
ba w size [address] - Break on write access only. Replace size with the num bytes you want to watch. Ex: ba w 4 005d5f10
.process -i <address> - Make the process active and break into the debugger. Use in live kernel debugs to get into process context.
.reload /f /o - Overwrites cached files in your downstream symbol store. Useful when your store has corrupt pdbs.
->* - Use with dt to dump pointers. Example: dt _EPROCESS [Address] ObjectTable->*
!for_each_module s -a @#Base @#End "PTag" - Find the drivers using pool tag "PTag".
.unload [DllName] - Unloads the debug extension you didn't intend to load. Omit DllName to unload the last dll loaded.
!exqueue dumps the executive worker queues. Use flags 7f to dump the worker threads and the queues.
lmvm <module> - Dumps information about the module. Remember to use <module> and not <module.dll>.
!thread -t TID - Dump a thread using thread ID rather than thread address. Useful when working with a critical section.
!list - Walks a linked list and displays informatino for each element in a list. See blog later today for an example.
.time -h # - Shows the debug session time using the timezone offset of #. Ex: .time -h 0 shows when a dump was taken in UTC.
!session - Lists all of the user session IDs. A quick way to list the active sessions from a dump of a terminal server.
!session -s SessionID - Changes the current session context to SessionID. Useful when looking at GDI, or other per session data.
| ProcNum s - Switches to process number ProcNum. Use when debugging multiple dumps, or processes, in one windbg.
!! - Launches a shell process and redirects its output to the debugger. The same as .shell, but "bang bang" sounds cooler.
uf Function - Dumps the assembly for Function (name or address). Useful for optimized code that is not contiguous in memory.
uf /c Function - Shows all of the calls made by Function (can be function name or address).
!wow64exts.sw - switches between x64 and x86 contexts. Often used to reverse .thread /w context switch.