Understanding Pool Consumption and Event ID: 2020 or 2019

re: Understanding Pool Consumption

Richard — Wed, 20 Dec 2006 00:22:36 GMT

Hi Tate.

What a great blog entry. Nice detailed and good explanation on what you are doing and why.

2-3 years ago, I started doing memory dump analysis, but first after attending Mark and David's 5 day session on trouble shoting windows, I was able to do a good memory dump analysis. This blog entry is at the same level (for me, being an sysadmin and not a dev guy).

This entry is a sure keeper, and I just subscribed to the blog.

Hoping to see more entries with the same level of information.

Richard

re: Understanding Pool Consumption and Event ID: 2020 or 2019

shubhankar — Fri, 22 Dec 2006 16:35:46 GMT

Great explanation. Superb!

I have done memory dump analysis for customers just on the basis of debugging help file but this entry really helps in understanding pool consumption.

Thanks

re: Understanding Pool Consumption and Event ID: 2020 or 2019

SpodBoy — Thu, 04 Jan 2007 17:51:48 GMT

Thanks for this. A great explanation that helped troubleshoot my issue in next to no time.

Thanks

SpodBoy

re: Understanding Pool Consumption and Event ID: 2020 or 2019

Greg Kreis — Sun, 07 Jan 2007 22:22:07 GMT

Thanks to your excellent article, with the paged and non-paged memory columns enabled on Taskman, I found that WZCSLDR2.exe was eating up 105k of non-paged memory and growing by 4-6k every second! No other app was anywhere near that much in non-paged memory usage. I researched it and many said it is related to the Dlink wireless card. I decided to disable it from startup and the wireless card still works and no more non-paged hogging! Whew.... THANKS!!!

re: Understanding Pool Consumption and Event ID: 2020 or 2019

Santosh — Tue, 20 Feb 2007 08:39:03 GMT

It was of great help.

Cheers Mate!

Server Hangs with Event ID: 2021 and 2022

Microsoft Advanced Windows Debugging and Troubleshooting — Tue, 19 Jun 2007 14:21:00 GMT

Hi again! This is Tate from the CPR team and I’m going to show you how to debug a Server Service hang

re: Understanding Pool Consumption and Event ID: 2020 or 2019

Mfartura — Wed, 20 Jun 2007 09:14:46 GMT

Hi Tate, great post! Thank you.

Just one thing I would like to verify: As far as I know Windows XP doesn't have the pool tagging option enabled by default. You would need to run the gflags, enable it and boot the system in order to have to be able to use a tool like poolmon. Windows 2003 will do by default though.

Am I wrong?

re: Understanding Pool Consumption and Event ID: 2020 or 2019

ntdebug — Wed, 20 Jun 2007 16:30:43 GMT

Yes, Windows XP will require you enable pooltagging with gflags before using tools like poolmon.

Thank you, Jeff-

re: Understanding Pool Consumption and Event ID: 2020 or 2019

Mike — Mon, 09 Jul 2007 16:31:11 GMT

Great post!

Thanks a lot. I expirienced unavailability of PDC. After reading this, I discovered TrendMicro NtListener using over 113k handles.

Thanks for great help!

re: Understanding Pool Consumption and Event ID: 2020 or 2019

Joerg Fischer — Fri, 24 Aug 2007 10:56:19 GMT

with the help of Your clear explanation and hints about pool consumption and event id 2019 i found statusclient.exe from the HP Toolbox with more than 500 000 handles to be the cause of our server trouble after migration from Windows adv. server 2000 to Windows enterprise server 2003 R2. Every 3-5 days a restart was necessary because the server stopped responding and delivered permanently ID 2019.

Thanks for this helpful contribution !

re: Understanding Pool Consumption and Event ID: 2020 or 2019

Martin Necas — Fri, 14 Sep 2007 16:39:17 GMT

Are there any chance to determine, which process(es) is using (and how much) memory allocation through "MmAllocateContiguousMemory".

I have an high consumption of nonpaged memory pool and only thing i see in PoolMon is MmCm flag with 40 MB in size.

I am running almost out of nonpaged memory, because I am running with /3GB (Exchange).

Thanks

Home

[Great question. I will likely follow up later with the debugging method that you can use, but for now, most of the recent issues we have seen on this revolve around the new Scalable Networking Pack release (KB912222) which most customers are getting as a result of installing Windows 2003 SP2. Here is the article to turn off the new functionality that is consuming the contiguous memory (most likely) KB936594 and avoid any potential incompatibility from your NIC drivers. Three major components usually are the primary consumers by the way of this memory, Video Card Driver, Storage Driver, and Network Stack. For the Video, you could also swich to a standard vga driver and that will not only get back the memory but more System PTEs as well. For the storage stack, all you can do here likely is get on the latest recommended driver version from your storage vendor, this memory consumed is usually the lowest of the three. Again, most of these cases we have seen have been because of the SNP and the need to use KB936594 to disable the functionality especially on resource constrained /3GB machines. -Tate]

re: Understanding Pool Consumption and Event ID: 2020 or 2019

Martin Necas — Fri, 14 Sep 2007 17:41:07 GMT

Tate, Thank You for soon reply and explanation!

I am looking forward to see the debugging method you mentioned.

I tried to apply the patch from KB936594 to the other server (IBM x3655), where the MmCm was at 28 MB. After applying and restarting, the MmCm tag is at the same level 28 MB in size :-(

Just to make all clear, I am using the latest possible BIOSes, firwares, drivers, ... on both servers ...

<DIV class=commentowner>[Sorry Martin, should have specified that it is the registry changes in that article that are most relevant and not the actual binaries mentioned therein...to disable the SNP features essentially is the goal]</DIV>

Understanding handle leaks and how to use !htrace to find them

Noticias externas — Fri, 14 Sep 2007 22:34:08 GMT

Hello, my name is Jeff Dailey, I’m an E scalation E ngineer for the Global Escalation Services P latforms

TalkBackVideo Understanding handle leaks and How to use !htrace to find them

Noticias externas — Fri, 14 Sep 2007 23:51:17 GMT

Hello, my name is Jeff Dailey, I’m an E scalation E ngineer for the Global Escalation Services P latforms

re: Understanding Pool Consumption and Event ID: 2020 or 2019

Martin Necas — Tue, 18 Sep 2007 11:40:03 GMT

Jeff, I've read your article about !htrace, however I think I'm not in the "handle issue". When I take a look at the processes and show the "handles" columnt, I see: store.exe - ~ 13600 handles, followed by System - ~ 11 120 handles. All other are bellow 4000. Should I search the cause here or not ?

[So handle count over 10,000 is only an issue if the count continues to grow or you are currently getting 2019 or 2020s. If your machine is running ok with no resource related errors you can most likely ignore an elevated handle count. If however you are seeing low on pool conditons, you should investigate these processes first. Thank you Jeff- ]

re: Understanding Pool Consumption and Event ID: 2020 or 2019

Preshan Naidoo — Tue, 25 Sep 2007 10:18:12 GMT

Thanks for a great article. Helps like this is rare and explanations thats given is better. Well done for making me understand.

re: Understanding Pool Consumption and Event ID: 2020 or 2019

Andy K — Mon, 01 Oct 2007 18:04:47 GMT

Umm, help. Spent way too many hours on this problem. Have a ton more knowledge on the subject but no solution.

Basically, Ddk continues to grow until the nonpaged pool drops below 20 meg and IIS starts to refuse connections. It used to take 3 weeks for this to happen. But after upgrading Trend Micro to 8.0 in an effort to stop the leak, the process of loosing memory has accellerate.

Sygate, Trend Micro as well as other system drivers use Ddk. I'm at a loss I can find no webpage that will give me my smoking gun saying this version of this will cause a memory leak.

Anybody have any thoughts?

[Yes, this problem is unfortunately all to common (use of the Ddk tag instead of some meaningful one to associate with the driver in question. In this case Driver Verifier should be able to help as it will track pool usage by driver not tag, see the "Using Driver Verifier" section in the post for this.- Tate]

re: Understanding Pool Consumption and Event ID: 2020 or 2019

Andy K. — Thu, 04 Oct 2007 15:24:46 GMT

Thanks for the advice. Driver Verifier was able to prove Trend Micro was the cause of the leak on our dev server. OfficeScan 8.0 has a known memory loss and has a patch. We were able to plug that leak.

Unfortunately the production server has a different memory leak(it had the patched version of OfficeScan on it). That leak is more ocasionally and of larger amounts of memory than the dev server leak. The dev server leak was every couple of seconds and of only 4k at a time. The Production leak is of random time(ocasional) for about 2 megs a time.

Very reluctant to turn on driver verifier on the production server. On the test server we could not monitor(just pool tracking) the sygate drivers as the server would not boot if driver verifier was monitoring them. Had to boot to safe mode and clear the settings.

I think we are going to try to eliminate possibilities by shutting off certain services and see if the problem goes away. Let me know if you have any other suggestions.

Thanks, Andy

re: Understanding Pool Consumption and Event ID: 2020 or 2019

Karan — Sun, 13 Jan 2008 20:49:54 GMT

An amazing blog entry. it is very helpfull and helped me a lot. keep up the good work!!

re: Understanding Pool Consumption and Event ID: 2020 or 2019

Matto — Thu, 27 Mar 2008 01:11:15 GMT

[Moderator's note: Our policy is to avoid naming specific 3rd party applications that behave badly. So the following comment has been edited.]

Superb! Used the handle count; method 1 and result was an immediate smoking gun for me pointing at a process called abcdefg.exe which is currently using 98000 odd handles and climbing consistently at 8 per minute. Thank You.

High pool memory usage on TS based SoftGrid clients under heavy load

Rod Trent at myITforum.com — Mon, 12 May 2008 21:26:45 GMT

Source: The SoftGrid Team Blog I have been on site with a few customers lately and a common issue I have

re: Understanding Pool Consumption and Event ID: 2020 or 2019

Ian — Fri, 23 May 2008 10:59:52 GMT

Great and real easy for a novice like me to understand.

re: Understanding Pool Consumption and Event ID: 2020 or 2019

Vijay — Mon, 02 Jun 2008 17:26:30 GMT

Nice to know about Non paged Pool issues,

Good article,

Thanks,

Vijay

re: Understanding Pool Consumption and Event ID: 2020 or 2019

skp — Tue, 03 Jun 2008 16:29:25 GMT

Is there are maximum handle count?

[The practical limit is the respective Pool which will be consumed by the objects in question referenced by said handles and if not that then the Pool consumed by the handle table itself. Usually a few thousand handles borders on the abnormal considering this practical limit especially on x86 machines.]

re: Understanding Pool Consumption and Event ID: 2020 or 2019

Floyd — Wed, 02 Jul 2008 01:42:43 GMT

[Moderator's note: Our policy is to avoid naming specific 3rd party applications that behave badly. So the following comment has been edited.]

Briliant. Found my problem was with [app] from [company]. Just keeps on eating away at the pool. 115 thousand handles in 2 days... [The company] has no cure, so I turned it off.

re: Understanding Pool Consumption and Event ID: 2020 or 2019

Rehan — Tue, 08 Jul 2008 10:54:47 GMT

Hi Tate, great article. I was wondering if you had any advice in troubleshooting MmSt paged pool utilization. I've got a server on 2003 sp1 with MmSt using over 65mb in paged pool, with the total hitting 325MB. I've implemented the ms registry change in increasing to the maximum paged pool size with trimming at 60% which seems to be buying me time for now.

However what I would really like to know is which processes are behind the utilization. From what I've been able to find out is that MmSt (beyond the brief definition in the pooltag.txt file) holds the system primitives which are responsible for tracking memory mapped files in the system. (the handle count on this machine doesn't seem to indicate the issue - highest totals are 4000 file handles and 7000 event handles in the system as a whole)

I can find the memory mapped files for each process on the machine via process explorer for example, but I'm not sure if the MmSt usage related to the size of the memory mapped files or their number.. or both?

Any advice would be greatly appreciated so I can get back to the appropriate software vendor.

Pool 소모 event ID 2020, 2019

!analyze -v — Thu, 04 Sep 2008 05:37:26 GMT

"이 문서는 http://blogs.msdn.com/ntdebugging blog 의 번역이며 원래의 자료가 통보 없이 변경될 수 있습니다. 이 자료는 법률적 보증이 없으며

re: Understanding Pool Consumption and Event ID: 2020 or 2019

Werner Schröter — Tue, 14 Oct 2008 19:21:21 GMT

one of the best articles ever!

re: Understanding Pool Consumption and Event ID: 2020 or 2019

Moloy — Wed, 03 Dec 2008 20:41:22 GMT

Indeed, an excellent post. Continue sharing the knowledge :)

re: Understanding Pool Consumption and Event ID: 2020 or 2019

Greg Harrington — Tue, 23 Dec 2008 00:51:40 GMT

Any thoughts on how to diagnose the MmCm tag utilizing 80+MB when the SNP features are disabled? Would driver verifier or debug be useful or a waste of time? Is there any way to pinpoint the root cause?

[Hi Greg – to discover root cause with the debugger you can use the PoolHitTag method mentioned at http://msdn.microsoft.com/en-us/library/cc267847.aspx and watch for allocations of the same size as you observe populating pool with your !poolfind MmCm when the machine has consumed the 80MB. However, a typically far easier and faster method since this allocation usually happens at boot is to disable the NIC and Storage adapters on the machine to determine if they are indeed allocating this memory at boot and if so reenable only the minimum number of adapters afterwards.]

re: Understanding Pool Consumption and Event ID: 2020 or 2019

william huber — Fri, 10 Apr 2009 21:14:58 GMT

absolutely phenomenal. This resolved the issue on my forest root after it stopped servicing requests. I restarted the service that was responsible for the executable that had way too many handles and it cleared up.

Thanks!

Long Distance Router

Long Distance Router — Fri, 19 Jun 2009 22:29:04 GMT

Dlink and Linksys are really facing off to see who can get the longest distance routers lately its funyn how far they will go

re: Understanding Pool Consumption and Event ID: 2020 or 2019

Don K — Tue, 23 Jun 2009 23:37:53 GMT

As most of the folks who read this have said, it's an excellent walk through of what I consider to be very complex issue with memory leaks and hangs.

Much apprecated Tate!

Thanks