MmCm – A Non Paged Pool Accounting Adventure

MmCm – A Non Paged Pool Accounting Adventure

  • Comments 1

Here's one from the Rube Goldberg debug collection!

The dripping sarcasm is because I'm about to show the reeeeally long way to figure out what's eating MmCm, skip down to the end if you are in a time crunch.

 

Otherwise, do resist the temptation to skip ahead as some of the techniques can be used in other debugs.

 

-Tate

 

 

What is this Pool Tag?

 

MmCm is a Non Paged Pool (NPP) tag allocated via a kernel mode call (a driver or the OS itself…applications cannot allocate this memory directly) to either MmAllocateContiguousMemory  or MmAllocateContiguousMemorySpecifyCache.

 

Great comments from msdn…

 

MmAllocateContiguousMemory can be called to allocate a contiguous block of physical memory for a long-term internal buffer, usually from the DriverEntry routine…

 

A device driver that must use contiguous memory should allocate only what it needs during driver initialization because physical memory is likely to become fragmented as the operating system runs.

 

Warning  If you use the MmAllocateContiguousMemory routine on computers with large amounts of memory, the operating system's performance might severely degrade when the system tries to create a contiguous chunk of memory. This degradation is reduced for Windows Server 2008. Memory that MmAllocateContiguousMemory allocates is uninitialized...

 

 

First, how much is normal?

 

As the comments indicate it is contingent mostly on the driver's needs, done at initialization time.  We have seen consumption in the single MB range up to tens of MB consumed depending on what drivers are loaded.

 

 

What drivers are the most likely consumers of this memory and why would the amount vary?

 

The most likely consumers on any machine would be the following types of drivers. Note that each will likely allocate a multiple of some amount per instance for each physical adapter exposed to Windows. (i.e. A single NIC driver used for 3 NICs in a system will likely allocate 3 times the MmCm vs. a single NIC.)

 

Storage Adapters for your SAN...SCSI Controller, Fibre Channel Controller, etc.

Video Adapters

Network Adapters

 

These drivers need to pre-allocate memory to service transfers and as mentioned need to do this right at boot time since memory can become fragmented later.  In other words, this allocation pattern attempts to "bake-in" a known set of scratch space if you will for the adapter to function properly given the features you wish to use.

 

 

So what's the problem?

 

Not everyone plays nice and the machine can run out of memory!  This is downright painful on x86 Exchange servers and their limited NPP ceiling.

 

Recall from an earlier post and your likely experience, that running out of such memory can cause hangs or general system instability as the machine runs out of NPP.  The scary part is that this usually happens unexpectedly under incrementally higher than normal user demand. i.e when you most need the server it fails.  Not coincidently most often we see older  x86 machines, especially /3GB configured Exchange servers, have this memory consumption problem. I often wonder just how many machines are out there just waiting for a liiiiitle more user load and are hovering at the brink of NPP exhaustion…

 

<Ok, queue the sad music for the heart breaking x86 Exchange server example>

 

This server has the standard 128MB maximum for NPP because of /3GB

 

0: kd> !vm

 

*** Virtual Memory Usage ***

Physical Memory:      851418 (   3405672 Kb)

Page File: \??\C:\pagefile.sys

  Current:   4193280 Kb  Free Space:   4038364 Kb

  Minimum:   4193280 Kb  Maximum:      4193280 Kb

Available Pages:      639032 (   2556128 Kb)

ResAvail Pages:       796753 (   3187012 Kb)

Locked IO Pages:         272 (      1088 Kb)

Free System PTEs:      27083 (    108332 Kb)

Free NP PTEs:           5778 (     23112 Kb)

Free Special NP:           0 (         0 Kb)

Modified Pages:         4138 (     16552 Kb)

Modified PF Pages:      4116 (     16464 Kb)

NonPagedPool Usage:    25151 (    100604 Kb)

NonPagedPool Max:      32026 (    128104 Kb)

PagedPool 0 Usage:     11497 (     45988 Kb)

PagedPool 1 Usage:      1645 (      6580 Kb)

PagedPool 2 Usage:      1667 (      6668 Kb)

PagedPool 3 Usage:      1662 (      6648 Kb)

PagedPool 4 Usage:      1679 (      6716 Kb)

PagedPool Usage:       18150 (     72600 Kb)

PagedPool Maximum:     63488 (    253952 Kb)

Session Commit:          586 (      2344 Kb)

Shared Commit:          4720 (     18880 Kb)

Special Pool:              0 (         0 Kb)

Shared Process:         7517 (     30068 Kb)

PagedPool Commit:      18214 (     72856 Kb)

Driver Commit:          8779 (     35116 Kb)

Committed pages:      237424 (    949696 Kb)

Commit limit:        1867524 (   7470096 Kb)

 

Given 100Megs used, 28MB of that is MmCm

 

0: kd> !poolused /t 10 2

   Sorting by  NonPaged Pool Consumed

 

  Pool Used:

            NonPaged            Paged

Tag    Allocs     Used    Allocs     Used

MmCm     3210 28779488         0        0        Calls made to MmAllocateContiguousMemory , Binary: nt!mm

NDpp     1013  4076960         0        0        packet pool , Binary: ndis.sys

MPIO   121985  3457752         0        0        UNKNOWN pooltag 'MPIO', please update pooltag.txt

elxs        6  3299344         0        0        UNKNOWN pooltag 'elxs', please update pooltag.txt

BCM0       24  3057232         0        0        UNKNOWN pooltag 'BCM0', please update pooltag.txt

LSwi        1  2654208         0        0        initial work context

 RaME        3  2572288         0        0        RiAllocateMiniportDeviceExtension , Binary: storport.sys

ElxA        5  2360208         3      160        UNKNOWN pooltag 'ElxA', please update pooltag.txt

Io        223  2319712       123     5136        general IO allocations , Binary: nt!io

TPLA      512  2097152         0        0        UNKNOWN pooltag 'TPLA', please update pooltag.txt

TCPt       40  1662448         0        0        TCP/IP network protocol , Binary: TCP

Mdl      7225  1435160         0        0        Io, Mdls

 Pool        3  1134592         0        0        Pool tables, etc.

 Devi      602  1124400         0        0        Device objects

 RcpI        1  1048576         0        0        Internal memory mgr initial heap block , Binary: sacdrv.sys

brcm       38  1032528         0        0        UNKNOWN pooltag 'brcm', please update pooltag.txt

TOTAL      219941 75555240     70157 56589336

 

That's not a lot right?  Well, it may be more than you like.  Depending on load cycles on this server it could be too much, you'd have to know a bit more history of the maximum amount of NPP demanded at peak usage, etc. to know how close to the edge this server may be.  Let's go with "what's using the ~28MB?" as the question to answer...

 

 

Who uses this memory?

 

Here's where it gets interesting.  A useful heuristic here is to group the sizes of the allocations in Excel.

You get this data from a !poolfind MmCm

 

0: kd> !poolfind MmCm

 

Scanning large pool allocation table for Tag: MmCm (f9a67000 : f9b67000)

 

*f7fc7000 :free large page allocation, Tag was MmCm, size was 0x2000 bytes

*f8298000 :free large page allocation, Tag was MmCm, size was 0x2000 bytes

*faeae000 :large page allocation, Tag  is MmCm, size  is 0x4000 bytes

*faeb2000 :large page allocation, Tag  is MmCm, size  is 0xd000 bytes

*fae84000 :large page allocation, Tag  is MmCm, size  is 0x11000 bytes

*fae95000 :large page allocation, Tag  is MmCm, size  is 0x11000 bytes

*fae23000 :large page allocation, Tag  is MmCm, size  is 0x1000 bytes

*fae24000 :large page allocation, Tag  is MmCm, size  is 0x3000 bytes

*fae2d000 :large page allocation, Tag  is MmCm, size  is 0x1000 bytes

.

.

.

fdf2e000 size:  f18 previous size:    0  (Allocated) MmCm

fdf2f000 size:  f18 previous size:    0  (Allocated) MmCm

fdf30000 size:  f18 previous size:    0  (Allocated) MmCm

fdf31000 size:  f18 previous size:    0  (Allocated) MmCm

fdf32000 size:  f18 previous size:    0  (Allocated) MmCm

fdf33000 size:  f18 previous size:    0  (Allocated) MmCm

fdf34000 size:  f18 previous size:    0  (Allocated) MmCm

fdf35000 size:  f18 previous size:    0  (Allocated) MmCm

fdf36000 size:  f18 previous size:    0  (Allocated) MmCm

fdf37000 size:  f18 previous size:    0  (Allocated) MmCm

fdf38000 size:  f18 previous size:    0  (Allocated) MmCm

fdf39000 size:  f18 previous size:    0  (Allocated) MmCm

fdf3a000 size:  f18 previous size:    0  (Allocated) MmCm

fdf3b000 size:  f18 previous size:    0  (Allocated) MmCm

fdf3c000 size:  f18 previous size:    0  (Allocated) MmCm

 

Searching NonPaged pool (fe000000 : ffb7e000) for Tag: MmCm

 

 

When I have to do this, usually some variation of following works.

 

.logopen

!poolfind

.logclose

 

Open the txt file up in notepad, clean it up a bit (remove the frees and extra lines)

Open the txt in Excel and use the old standby Text Import Wizard via file open

I usually also save the large pool allocs to one txt file and the regular to another and then cut paste to combine them in Excel.

 

There's probably an easier way to do this import, but this works well enough.

image

You can really go crazy getting the data all pretty but don't bother as even something as ugly as this is useful because a quick sort on Column B shows the allocation size pattern.  Note the Text Import Wizard allows you to skip columns here, so you can import only the address column and the size column.  i.e. faeae000’s column and the 0x4000 column in the above.

 

image

Via Excel I just generated another table on the fly to keep track of our totals here.  Note, I’m just counting the number of each size of alloc by visual inspection.  i.e. There are 9 allocations of 0x1000 size in my list.

 

Allocation Sizes in Bytes (h)  

Number of Allocations

Total MmCm in Bytes

0xf18 bytes 

2014

7,782,096

0x1000 bytes

9

36,864

0x2000 bytes

1017

8,331,264

0xf000 bytes

23

1,413,120

0x11000 bytes

92

6,406,144

0x156000 bytes

2

2,801,664

 

Ahhh…..In total we've got 26 of 28 Megs sampled here, well representing the MmCm usage minus some insignificant allocs and rounding, great.

 

(I love the new Win7 calc.exe)

clip_image003

So how does this help me again?  Here's the crucial step.  Look at samples of these allocations (which is why I kept the address of the alloc in the Excel import).  There are very often telltale tags or strings inside the alloc that give you a strong indication if not proof positive who allocated this memory.

 

Here's a common allocation example:

 

Dump several samples of the f18 sized allocations…

 

0: kd> dc fdf18000 fdf18000+f18

fdf18000  0be30000 6d436d4d fd1df008 00000000  ....MmCm........<--the pooltag, okay…knew that...

fdf18010  1f2e3d4c 00000000 00000000 00000000  L=..............<-------hum, this 1f2e3d4c seems to be in every one of these puppies...

fdf18020  00000000 00000000 00000000 00000000  ................

fdf18030  00000000 00000000 00000000 00000000  ................

fdf18040  00000000 00000000 00000000 00000000  ................

 

Remember we learned that these are predominantly allocated on boot?  Turns out that Storport actually allocates 1000 of these(per adapter) based on the extension size of the adapter and guess what it does, it puts a handy little tag in here defined as 1f2e3d4c.    Here's the proof.

 

storport!RaInitializeRaidResources+0x47:

f5d6aeb3 8bd7            mov     edx,edi

f5d6aeb5 8bce            mov     ecx,esi

f5d6aeb7 ff1510d1d6f5    call    dword ptr [storport!_imp_InterlockedPushEntrySList (f5d6d110)]

f5d6aebd ff4508          inc     dword ptr [ebp+8]

f5d6aec0 8b4508          mov     eax,dword ptr [ebp+8]

f5d6aec3 3b450c          cmp     eax,dword ptr [ebp+0Ch]

f5d6aec6 c747084c3d2e1f  mov     dword ptr [edi+8],1F2E3D4Ch<------------------ah ha!

f5d6aecd 72be            jb      storport!RaInitializeRaidResources+0x21 (f5d6ae8d)

 

 

Cool! So now I've accounted for 7.8 Megs of the 28, and I know at this point that since I've got ~2000 allocs I've got at least two storage adapters on this machine.

Actually, If you are suspicious about the roughly 1000 0x2000 sized allocations and you think it's another adapter with a larger extension size, you'd be right!

 

0: kd> dc fcc20000

fcc20000  fcc22000 00000000 1f2e3d4c 00000000  . ......L=......<-------again!

fcc20010  f7b7e000 00000000 00000000 00000000  ................

fcc20020  00000000 00000000 00000000 00000000  ................

fcc20030  00000000 00000000 00000000 00000000  ................

fcc20040  00000000 00000000 00000000 00000000  ................

fcc20050  00000000 00000000 00000000 00000000  ................

fcc20060  00000000 00000000 00000000 00000000  ................

fcc20070  00000000 00000000 00000000 00000000  ................

 

 

Now, I've accounted for likely all my storage adapters (7,782,096 + 8,331,264  =  16,113,360) which are between the three of them consuming 16 of 28 Megs!

(I say three because of the ~3000 allocations in total, with 1000 per adapter as stated.  So two adapters of size 0xf18 and one of size 0x2000)

Sure enough checking msinfo32 on this machine shows three adapters.

 

So what about the rest of the odd ball sized allocations and 12 Megs?  Especially the 92 0x11000 and the two whopping 0x156000 bytes ones?

 

First , the two whopper 0x156000 byte allocs.  Here's where you are limited usually only by your own creativity and patience...( this particular sample is courtesy of a fellow persistent debugger,  Pushkar)

 

//Dump the beginning of one…

 

0: kd> dc fdb24000

fdb24000  00036c40 fdb24080 fdc43100 08b24080  @l...@...1...@..

fdb24010  00000000 08c43100 00000000 00000001  .....1..........

fdb24020  00000000 00000000 00000000 00000000  ................

fdb24030  00000000 00000000 00000000 00000000  ................

fdb24040  00000000 00000000 00000000 00000000  ................

fdb24050  00000000 00000000 00000000 00000000  ................

fdb24060  00000000 00000000 00000000 00000000  ................

fdb24070  00000000 00000000 00000000 00000000  ................

 

On the line below with the highlight we are looking for interesting pointers and !pool'ing them to figure out who's associated with this alloc.

 

0: kd> dc

fdb24080  00000003 00000940 08b24100 00000000  ....@....A......

fdb24090  fdef0080 00000000 08b241d8 00000000  .........A......<-----This is the line

fdb240a0  00000080 08b25080 00000000 00000080  .....P..........

fdb240b0  08b35080 00000000 00000080 08b29080  .P..............

fdb240c0  00000000 00000080 08b37080 00000000  .........p......

fdb240d0  00000080 08b2d080 00000000 00000080  ................

fdb240e0  08b39080 00000000 00000080 08b31080  ................

fdb240f0  00000000 00000080 08b3b080 00000000  ................

 

//Oh, it happens to be a Device object...

 

0: kd> !pool fdef0080

Pool page fdef0080 region is Nonpaged pool

*fdec1000 : large page allocation, Tag is Dev., size is 0x33000 bytes

Owning component : Unknown (update pooltag.txt)

 

//Dump it out via devobj, adding the object offset and poolheader length…this one looks like it's going to belong to my Emulex adapters

 

0: kd> !devobj fdec1000+38

Device object (fdec1038) is for:

ElxPlus*** ERROR: Module load completed but symbols could not be loaded for elxplus.sys

\Driver\elxplus DriverObject fdef41c0

Current Irp 00000000 RefCount 1 Type 0000002a Flags 00000048

DevExt fdec10f0 DevObjExt fdef3ca0

ExtensionFlags (0000000000) 

AttachedTo (Lower) fdf68ba0 \Driver\PnpManager

Device queue is not busy.

 

On to the 0x11000 sized, first pass is to just dc out the memory like before and look for interesting tags or strings…At the end of Network card adapter driver allocations via their calls into the Ndis layer for instance, you can flag them by the END of the allocation may have an ND** string.  I don't see any of those here but it "looks like" the 0x11000 sized allocations have network related data and strings as a common factor throughout, so I'm guessing these are associated with one of the four network adapters on this machine.  Turns out if you sample more of these they may have pointers back to other interesting pool allocations that can flag the network miniport adapter, etc.

 

You may be thinking, hum…I get the storport one but these last two sound like mere correlation?  Absolutely.  However, given the lack of randomness sampling multiple allocations it's often strong enough to at least inform your hypothesis and test, quickly.

 

 

Yes, there is an easier way!

 

If this memory is allocated at driver initialization and most drivers initialize at system boot time, it stands to reason that most of this memory will be consumed on boot.  Cool!  Then it also stands to reason we can use our friend Poolmon.exe to quickly check how much is being used shortly after boot and do some quick testing by excluding some of the usual suspect adapters.  First, you could just trust me and check for ancillary storage, NIC, and enhanced video adapters and disable/remove these first as a low risk test.

 

In any case, the testing sequence is:

 

1.      Promptly record the amount of MmCm after a reboot via Poolmon.exe

(Sort by Non Paged Pool (toggle through by hitting 'P') and then descending by bytes (same, but 'B')

2.      Disable the ancillary test adapter(s)

3.      Reboot, check poolmon.exe again

4.      What's the decrease?

 

5.      Perhaps repeat testing with updated drivers or disabled features if you must have the additional hardware in place (hoping they use less MmCm).

 

I hope this post saved you time by being wary of configuring more adapters than absolutely necessary, especially /3GB x86 Exchange servers and if you have to reactively triage this a quick and dirty way to determine the primary consumers of this memory.

 

Enjoy!

 

-Tate

 

P.S. If you can't scale down the number or features of let's say one of the suspect adapters, do check with your driver vendor to see if there is a way to throttle back the consumption.  For instance, when we shipped the Scalable Networking Pack feature, we often saw higher consumption in the NIC drivers supporting this functionality, later driver releases from hardware vendors reportedly used much less.

 

 

 

 

Bonus:

 

Here's another handy debug trick to isolate who can call these two Memory manager functions and narrow down the driver population...

 

//Find the address of MmAllocateContiguousMemorySpecifyCache

0: kd> x nt!MmAllocateContiguousMemorySpecifyCache

e080efac nt!MmAllocateContiguousMemorySpecifyCache = <no type information>

 

//Use the handy !for_each_module to scan each module for the address (e080efac )

0: kd> !for_each_module ".echo ${@#ModuleName} ;s -d ${@#ModuleName} L?${@#Size} e080efac"

 

//Hits will look like this, with the matching address listed…

ATMFD

nt

e0a81044  e080efac e0806d12 e082e237 e082ddc7  .....m..7.......

hal

e0a81044  e080efac e0806d12 e082e237 e082ddc7  .....m..7.......

RDPWD

.

.

.

ati2mtag

f5575234  e080efac e083b69c e08454c9 e082f651  .........T..Q...

f5836084  e080efac e08329f9 e0839c13 e08e2d78  .....)......x-..

.

.

.

ql2300

f5d6d174  e080efac e0806d12 e08412f9 e08eaeb1  .....m..........

f5e170cc  e080efac e08d15a3 e08e8543 e0815d48  ........C...H]..

storport

f5d6d174  e080efac e0806d12 e08412f9 e08eaeb1  .....m..........

f5e170cc  e080efac e08d15a3 e08e8543 e0815d48  ........C...H]..

elxstor

f5e170cc  e080efac e08d15a3 e08e8543 e0815d48  ........C...H]..

SCSIPORT

f5e170cc  e080efac e08d15a3 e08e8543 e0815d48  ........C...H]..

 

 

//Example match in the import table for ati2mtag...

0: kd> dps f5575234

f5575234  e080efac nt!MmAllocateContiguousMemorySpecifyCache

f5575238  e083b69c nt!ZwQueryInformationProcess

f557523c  e08454c9 nt!PsGetCurrentThreadProcessId

 

 

//Repeat for the other call...

0: kd> x nt!MmAllocateContiguousMemory

e080be42 nt!MmAllocateContiguousMemory = <no type information>

 

 

 

 

Share this post :

 

 

 

 

Leave a Comment
  • Please add 3 and 3 and type the answer here:
  • Post
  • Very informative. It realy help me to understand this Tag

Page 1 of 1 (1 items)