Troubleshooting Pool Leaks Part 6 – Driver Verifier

Troubleshooting Pool Leaks Part 6 – Driver Verifier

Rate This
  • Comments 0

In part 5 we used poolhittag to get call stacks of pool being allocated and freed.  This information is often essential to identifying the cause of a memory leak; however it is not always feasible to configure a live kernel debug to obtain this information.  Fortunately there are alternative methods to get such call stacks.

 

Driver verifier has an option to enable pool tracking for a specific driver, or for multiple drivers.  This functionality was first introduced in Windows Vista and Windows Server 2008.  This information is also captured when driver verifier is used to enable special pool, however for the purposes of this article we will focus on using pool tracking.

 

The data stored by driver verifier requires a debugger to view.  Any method of debugging can be used for this.  You can use a live kernel debug as we described in part 4, you can get a memory dump (kernel or complete, a small dump is insufficient), or you can use livekd.

 

If you have used the steps from Part 1, Part 2, or Part 3, you likely have an idea which drivers are likely involved in creating the pool leak.  In this example we are generating the leak using notmyfault, the same tool we have been using in prior examples.  As seen in Part 2, the relevant driver is myfault.sys.

 

Although driver verifier has GUI, the easiest way to enable this functionality is with the below command from an elevated command prompt:

 

Verifier /flags 8 /driver myfault.sys

 

The above command will provide the following output, allowing you to confirm that the expected settings are enabled:

 

New verifier settings:

 

Special pool: Disabled

Pool tracking: Enabled

Force IRQL checking: Disabled

I/O verification: Disabled

Deadlock detection: Disabled

DMA checking: Disabled

Security checks: Disabled

Force pending I/O requests: Disabled

Low resources simulation: Disabled

IRP Logging: Disabled

Miscellaneous checks: Disabled

 

Verified drivers:

 

myfault.sys

 

 

You must restart this computer for the changes to take effect.

 

After rebooting the system, reproduce the memory leak and attach a debugger or generate a memory dump after the memory has been leaked.

 

Break in with the debugger (Ctrl+Break or Ctrl+C) or load the dump in windbg (File – Open Crash Dump).

 

Set the symbol path and reload symbols.

 

1: kd> .symfix c:\symbols

1: kd> .reload

Loading Kernel Symbols

...............................................................

 

The !verifier command has various options to view information about driver verifier.  To view the pool allocations which have been tracked by verifier for notmyfault.sys, use the following:

 

0: kd> !verifier 3 myfault.sys

 

Verify Level 8 ... enabled options are:

      All pool allocations checked on unload

 

Summary of All Verifier Statistics

 

RaiseIrqls                             0x0

AcquireSpinLocks                       0x0

Synch Executions                       0x0

Trims                                  0x0

 

Pool Allocations Attempted             0xb

Pool Allocations Succeeded             0xb

Pool Allocations Succeeded SpecialPool 0xa

Pool Allocations With NO TAG           0x1

Pool Allocations Failed                0x0

Resource Allocations Failed Deliberately   0x0

 

Current paged pool allocations         0x0 for 00000000 bytes

Peak paged pool allocations            0x1 for 00000080 bytes

Current nonpaged pool allocations      0xa for 009CE000 bytes

Peak nonpaged pool allocations         0xa for 009CE000 bytes

 

Driver Verification List

 

Entry     State           NonPagedPool   PagedPool   Module

 

fffffa80031b5830 Loaded           009ce000       00000000    myfault.sys

 

Current Pool Allocations  0000000a    00000000

Current Pool Bytes        009ce000    00000000

Peak Pool Allocations     0000000a    00000001

Peak Pool Bytes           009ce000    00000080

 

PoolAddress  SizeInBytes    Tag       CallersAddress

fffffa8005400000     0x000fb000     Leak      fffff8800447d634

fffffa80052fb000     0x000fb000     Leak      fffff8800447d634

fffffa8005200000     0x000fb000     Leak      fffff8800447d634

fffffa80050fb000     0x000fb000     Leak      fffff8800447d634

fffffa8005000000     0x000fb000     Leak      fffff8800447d634

fffffa8004efb000     0x000fb000     Leak      fffff8800447d634

fffffa8004e00000     0x000fb000     Leak      fffff8800447d634

fffffa8004cfb000     0x000fb000     Leak      fffff8800447d634

fffffa8004c00000     0x000fb000     Leak      fffff8800447d634

fffffa8004a66000     0x000fb000     Leak      fffff8800447d634

 

At the bottom of the above output is the list of allocations made by notmyfault.sys.  For our purposes we are going to assume that these allocations have been leaked, as opposed to just being normal allocations that were not yet freed when the debugger was attached.

 

The !verifier command has an option to view call stacks for one of the tracked allocations.  Keep in mind that the size of the database is limited and only more recent allocations will be kept in the database.

 

0: kd> !verifier 80 fffffa8005400000

 

Log of recent kernel pool Allocate and Free operations:

 

There are up to 0x10000 entries in the log.

 

Parsing 0x0000000000010000 log entries, searching for address 0xfffffa8005400000.

 

 

======================================================================

Pool block fffffa8005400000, Size 00000000000fa000, Thread fffffa80044ceb60

fffff80001927cc6 nt!VeAllocatePoolWithTagPriority+0x2b6

fffff80001927d3d nt!VerifierExAllocatePoolEx+0x1d

fffff8800447d634 myfault!MyfaultDeviceControl+0x358

fffff8800447d727 myfault!MyfaultDispatch+0xb7

fffff80001932750 nt!IovCallDriver+0xa0

fffff800017a7a97 nt!IopXxxControlFile+0x607

fffff800017a82f6 nt!NtDeviceIoControlFile+0x56

fffff8000148bed3 nt!KiSystemServiceCopyEnd+0x13

Parsed entry 0000000000010000/0000000000010000...

Finished parsing all pool tracking information.

 

The above output shows the call stack leading to the pool allocation.  This is the same information we had seen in Part 5, however we are able to obtain this information using a dump or livekd, whereas the steps from Part 5 required an invasive debug and extended system downtime.

 

When you have completed troubleshooting, disable driver verifier with the following command and reboot:

verifier /reset

Leave a Comment
  • Please add 5 and 8 and type the answer here:
  • Post