The Memory Shell Game

The Memory Shell Game

Rate This
  • Comments 12

Hello, this is Somak.  Today I’d like to drop some Memory Manager info on the blog that I’ve used to communicate in brief (believe it or not) how the system deals with memory.  If you are ever faced with checking how much Available Memory you have(or don’t have), poor system performance, questions about page faults, or having a Working Set Trimming performance issue, or just want a primer into how Windows manages memory on the system, be not afraid and read on!

How this memory stuff works

The fastest and most expensive memory is built into the CPU.  The next fastest and less expensive is physical RAM.  Next we have the hard drive, followed by remote storage or backup.  Each step down the ladder adds significantly to access time.  For example, physical RAM can be almost one million times faster than a hard disk.  Since it is volatile and cost considerably more than hard disk space, we are limited at how much we can put in the system.  This limit is far less than the much slower and non-volatile hard disk.  Due to these design constraints we have a tiered memory structure in the computer.  To achieve the fastest response times and best overall performance, an operating system must efficiently use this tiered memory structure.  It must do its best to reduce the need to retrieve data from a slower storage media like the hard disk.  It has to do this while juggling the memory and I/O demand from all running processes.  The following paragraphs are an overview of how the Memory Manager achieves this in Windows.  A more detailed description can be found in Chapter 7 of Microsoft Windows Internals, Fourth Edition (ISBN: 0-7356-1917-4).  This book is the great source for how stuff works in Windows.

First let’s lay out some definitions.  They will be useful later on when I talk about the interactions.  These definitions are high level to maintain brevity.

Virtual Memory – This is a memory that an operating system can address.  Regardless of the amount of physical RAM or hard drive space, this number is limited by your processor architecture.  On a 32 bit processor you are limited to 4 GB of addressable virtual memory (2^32).  With a default installation on a 32 bit box (not using /3GB) the kernel reserves 2GB for itself.  Applications are left with 2GB of addressable virtual memory.  When applications execute they are only presented with this 2GB of addressable memory.  Each application gets its own 2GB virtual memory to play with.  If you have 50 processes running, you’ll have 50 independent 2GB Virtual Memory address spaces and one 2GB Virtual Address space for kernel.  This is possible because Virtual Memory always exists, but doesn't really exist (hence the term virtual).  We basically lie to the application and say, here is 2GB of memory address space for you to use.  This memory isn’t allocated until the application explicitly uses it.  Once the application uses the page, it becomes committed.  A virtual memory page is then translated to a physical memory page.  From this translation, the virtual page can reside in physical RAM or on the hard disk.

Physical Memory –  This is the physical storage media.  It can be physical RAM, the hard disk, optical disks, tape backups, etc.  This is anything that can store data.  Most times when people talk about Physical Memory, they refer to physical RAM (the memory sticks on your motherboard), but with virtual page translation, physical memory can also be on the hard drive (in your paging file).  Physical RAM is limited by your processor architecture.

Committed Memory –  When an application touches a virtual memory page (reads/write/programmatically commits) the page becomes a committed page.  It is now backed by a physical memory page.  This will usually be a physical RAM page, but could eventually be a page in the page file on the hard disk, or it could be a page in a memory mapped file on the hard disk.  The memory manager handles the translations from the virtual memory page to the physical page.   A virtual page could be in located in physical RAM, while the page next to it could be on the hard drive in the page file.

Commit Limit –  This is the maximum amount of memory that all your applications and the OS can commit.  If you had 50 applications fully allocate their 2 GB of virtual address space, you would need 100GB of commit limit (ignore kernel memory usage to keep the numbers simple).  So 100GB of committed pages can be backed by physical RAM or the hard drive.  If you have 100 GB of RAM, you could handle this memory load.   In most cases, 100 GB of RAM isn't economically feasible so the Commit Limit is comprised of physical RAM and the page file.  If you have 2 GB of physical RAM and 98 GB of page file, then your commit limit would be 100 GB.

Page file –   This is the storage area for virtual memory that has been committed.  It is located on the hard drive.  Since hard drive space is cheaper than physical RAM, it is an inexpensive way to increase the commit limit.

Working Set – This is a set of virtual memory pages (that are committed) for a process and are located in physical RAM.  These pages fully belong to the process.  A working set is like a "currently/recently working on these pages" list.

Modified pages - Once a virtual memory page leaves the process's working set, it is moved to another list.  If the page has been modified, it is placed on the modified page list.  This page is in physical RAM.  A thread will then write the page to the page file and move it to the standby list.

Standby pages - This is a page that has left the process' working set.  This page is in physical RAM.  A standby page is like a cache for virtual memory pages.  It is still associated with the process, but not in its working set.  If the process touches the page, it is quickly faulted back into the working set.  That page also has one foot out the door.  If another process or cache needs more memory, the process association is broken and it is moved to the free page list.  Most of the pages in available memory are actually standby pages.  This makes sense when you realize that these pages can be quickly given to another process (hence available), but you should also understand that they are page caches for working sets and can be quickly given back if the process touches the page again.  The vast majority of available memory is not wasted or empty memory.

Free pages - When a page is taken off of the standby page list, it is moved to the Free page list.  This page is in physical RAM.  These pages are not associated with any process.   When a process exits, all of its pages are then dumped onto this list. Typically, there is a very small to no amount of free pages hanging around physical RAM. 

Zeroed pages - When a free page is zeroed out, it is placed on the Zero page list.  This page is in physical RAM.  These are the pages that are given to processes that are making memory allocations.  Due to C2 security requirements, all pages must be scrubbed before handed to a new process.  When the system is idle, a thread will scrub free pages and put them on this list.  Only a small amount of zero pages are required to handle the typical small memory allocations of processes.  Once this list is depleted, and if there is demand for more pages, we pull pages off of the Free page list and scrub them on the fly.  If the Free page list is depleted, then we pull pages off of the standby list, scrub them on the fly and hand them to the new process.

What is Task Manager telling me?

Prior to Windows Vista, Task Manager reports memory usage using accounting methods that you probably are not expecting.  It is because of these accounting practices, that we rarely use Task Manager to gauge system performance and memory usage.  We typically use it for a quick overview or to kill processes.  I highly recommend using Performance Monitor (perfmon.msc) for investigating performance issues.  Here's the breakdown of the numbers on the Performance tab:

Physical Memory

Total - The is the total physical RAM installed in the system.

Available - This is the total of the Standby, Free and Zeroed list.  Free and Zeroed makes sense, but Standby seems odd at first.  Standby pages were added to this number because they are available for a quick scrub and given to a new process.  So they are technically available (with minimal effort).

System Cache- This is the total of the Standby list and the size of the system working set (which includes the file cache).  Standby pages are added to this list because they are cached pages for working sets.

PF Usage - This is the total number of committed pages on the system.  It does not tell you how many are actually written to the page file.  It only tells you how much of the page file would be used if all committed pages had to be written out to the page file at the same time.

Commit Charge

Total - This is the total virtual memory that has been committed.  This includes all committed memory for all processes and the kernel.

Limit - This is the maximum amount of committed memory this system can handle.  This is a combination of physical RAM and the page file.

Peak – This is the highest amount of memory committed thus far on this system, since boot.

How does this work?

The memory manager optimizes physical RAM usage across the entire system.  Since physical RAM is a finite resource, it has to balance sharing this critical resource amongst all process, the kernel and file I/O.   It tries to keep disk I/O to a minimum, which results in a more responsive system.  It does this by moving pages around to meet the demand of the system.

Typically, large sections of physical RAM are used for file cache.  This is cache is necessary to improve disk performance.  Without it, disk I/O would make the system crawl along at an nearly unusable pace.  The file system cache is just like a working set for a process.  Pages removed from the file cache are moved to the standby or modified page list.  Many of the standby pages in RAM are probably file cache pages that were removed from its working set.  For example, on a file server, if you see 8 GB of available memory, most of these pages are probably standby pages for the file cache.  The file cache's working set could be 500 MB, but the 8 GB of standby pages should also be considered part of the file cache.

Now let's take a look at how the memory manager handles processes.  While an application is working with its virtual memory pages, the memory manager keeps the pages in the process' working set.  Since the vast majority of application do not use all of its memory all the time, some pages will age.  Old pages are removed from the working set.  If they are modified, they are moved to the modified list.  The page is saved to the page file and moved to the standby list.  If the page hasn't been modified, it is moved directly to the standby list.  These pages will remain on the standby page list until there is a demand for it.

If the application touches the page again, it is soft faulted back into the process' working set.  If the process doesn't use the page for a very long time, or if the demand for the page is greater elsewhere, the page is moved off of the standby list.  It is disassociated with the process and moved to a the free page list.  From the free page list, the page is scrub on demand or lazily and placed on the zero page list.  It is from the zero page list that other processes or the kernel or the file cache will get a new page.

If after a very long time the application once again needs a page that is not in its working set, the memory manager will handle the memory fault.  If the page is on the standby list, it is quickly put back into the process' working set.  If the page is no longer in the standby list, a hard fault occurs.  The memory manager issues I/O to the hard disk to read the page(s) from the page file.  Once the I/O complete, the page is placed back into the process' work set.

All of this is done to keep physical RAM highly utilized and disk I/O to a minimum.  We don't want to allow process to horde physical RAM for pages that are rarely used.  The physical RAM must be shared with other processes, the kernel and the file cache.  If you see lots of available memory on your system, rest assured that it is not going to waste.  The vast majority is on standby lists for processes and the file cache.

Also note that page file usage isn't that bad.  The page file allows the Memory Manager to save modified pages before placing the page on the standby list.  The page is still in physical RAM and can be quickly faulted back into the process.  This method gives the process a chance to reclaim an old page and it allows the page to be quickly used if there is demand elsewhere.

The best way to see the totals of these lists is to use a kernel debugger (live or postmortem).  Use the !memusage command and you'll get an output like this:

0: kd> !memusage

loading PFN database

loading (100% complete)

Compiling memory usage data (99% Complete).

Zeroed:    414 (  1656 kb)

               Free:      2 (     8 kb)

Standby: 864091 (3456364 kb)

Modified:    560 (  2240 kb)

    ModifiedNoWrite:     30 (   120 kb)

Active/Valid: 182954 (731816 kb)

         Transition:      2 (     8 kb)

                Bad:      0 (     0 kb)

            Unknown:      0 (     0 kb)

TOTAL: 1048053 (4192212 kb)

Of the 4GB of physical RAM, only 1.6 MB are on Zeroed or free pages.  731 MB is in process, system and file cache working sets.  2 MB are on the modified page list.  The vast majority, 3.4 GB, is on the standby list.  On this server, most people will see 3.4 GB of wasted physical RAM, but you will know better.

What should I be worried about?

Typically you shouldn't be worried about these things, until you have a performance problem.  If your system is sluggish or slow to respond or you are getting errors about out of memory, then you need to rely on this information.  You will need to collect a performance monitor log of the problem time.  If the counter list is daunting, then use the Performance Monitor Wizard to configure the performance monitor log.

Once the log is collected, you'll need to analyze several counters.  I'm not going into detail about how to review performance monitor logs this time.  I'll save that lengthy topic for another time.  For now I'll focus on the counters relevant to the Memory Manager.

One of the biggest reasons for slow performance and sluggish system responsiveness is disk bottleneck.  Look at Physical Disk\% Idle Time, Avg. Disk sec/Read and Avg. Disk sec/Write counters.  If your system drive is under 50% idle or your disk response times are way above your drive specifications, then you need to investigate further.  Look at the Memory\Available Mbytes.  You should have a couple hundred Mbytes of Available Memory.  This is one of the most important performance monitor counters.  If this number drops too low, your standby lists, process working sets and cache will be greatly reduced.  You'll need to find out if a process is consuming physical RAM.  Check for large process working sets or for large file cache.

You will also need to see if paging is really affecting system performance.  Take a look at Memory\Pages Input/sec and correlate that to Physical Disk\Avg. Disk sec/Read.  Pages Input/sec is the number of pages being read in from the page file.  These are the hard faults (when the page wasn't on the standby list).  If your Avg. Disk sec/Read is close to your drive's specification and the drive's idle time is high, than paging really isn't a problem.  Small amounts of hard faults are expected as applications will every once in a while re-touch an old page.  As long as this I/O is not consistent or the disk can't keep up, you probably will not notice this impact.

You can also look at Memory\Pages Output/sec and Physical Disk\Avg. Disk sec/Write.  These are the page commits to the page file when a modified page is removed from a process' working set.  As long as the disk can keep up with the writes, this shouldn't be a problem.  Remember that once the page is saved to the page file, it is placed on the standby list.  If there isn't a great demand for new pages, it can remain on the standby list for a very long time.  When the process touches the old virtual page again, it can be soft faulted back into the working set.  If there is great demand for memory, you'll see process working sets aggressively being trimmed.  Unless there is memory pressure, this is all done with lazy I/O, so you should not see much of an impact from this activity.

The Memory Manager works to meet current demand and prepares for future demand when possible.  You need to look at a performance monitor log to see if there is memory pressure on the system.  You'll see this in low Available Mbytes and reductions in process working sets.  You'll be able to correlate this to increase disk I/O to the page file.  If you have established that there is memory pressure on the box, you need to figure where that demand is coming from.  Check for working set increases from processes, file cache or bottlenecked disk I/O to data drives.

Leave a Comment
  • Please add 1 and 4 and type the answer here:
  • Post
  • PingBack from http://www.artofbam.com/wordpress/?p=6897

  • Very good article.

    Very good reference to Windows Internals as well.

    Thank you for writing.

    It might be a rather naive question but I would like to understand how it is implemented the hierarchical access time for those pools.

    I mean, I suppose accessing the standby page´s pool 'takes lesser' then acessing free page´s pool or zeroed page´s pool and that acessing all the three pools 'takes longer' than accessing the pages on the working set.

    Is it only because of the calls and context switching(efforts) that would take a process to request the memory manager to grab those pages(which are not on its working set space) or there are other reasons like the design of those particular pools(and here I am thinking of pools as queues and the different approaches one might implement for acessing queues and serializing/parallelizing access strategies).

    See, I dont quite get it what makes the standy pages more quickly acessible in opposite to free pages or zeroed pages, for instance.

    Since they are all on the same physical memory and no concepts such as NUMA are appliable.

    Thank you once again.

    [The access time for a page in the process’ working set will be the fastest. If a page leaves the process’ working set, it is placed on to the standby list. If a thread accesses the page when it is on the standby list, the thread will encounter a trivial delay. A page fault exception occurs and is handled by the memory manager. No context switch occurs for this operation. This soft fault will require some CPU cycles and a few access requests to physical RAM. The performance hit for this soft fault is so small; you will probably never notice it. I have only heard of one high performance application being affected by this, but I never saw any performance data to validate this claim. The application probably accessed its pages poorly and perhaps could have benefited from locking some of its critical pages into physical memory.

    When a process requests a new page, transitioning from the zero page list will be the fastest. After that is transition from the free list to the zero page list. This transition requires a page scrub. This is bound by the speed and bandwidth of your physical RAM. There is no context switch here as well. Once again, these are very small performance hits. Chances are that you’ll never notice the hit from this operation. I hope this answers your questions.]
  • Indeed, a very good article. Such a complex topic written in a very lucid elegant way...thanks so much.

  • If the the os gets hibernated, the standby pages are written to the hiberfil.sys?

  • Cache is used to reduce the performance impact when accessing data that resides on slower storage media. 

  • Cache is used to reduce the performance impact when accessing data that resides on slower storage media

  • It is my understanding that executables (exe and dll files) are mapped into virtual storage but not all of them are always read into physical memory.

    It is not clear to me whether the executables are read into memory and then written to the pagefile, but that seems inconsistent with my understanding.

    So I don't understand enough of this issue to ask a good question, but I suspect it is something that needs clarification.

    Great questions! If multiple process load the same DLL (like ntdll.dll), it isn’t efficient to load the file multiple times. Modern day operating systems load executable modules (exe or dll) in shared memory. The virtual page in each process where the module is loaded will point to the same shared section for the module. This way only one instance of the module is loaded. Each process will maintain its own private data (for globals and such) and use the shared pages for static and unmodified pages in the module. When the module’s pages are removed from working sets, the page is not committed to the page file. These pages are backed by the file on disk, so it doesn’t make sense to save a copy to the page file as well. Basically we remove the page from physical memory. A hard fault to one of these pages will be resolved from the file on disk. If a process modified module globals or data structures, those pages would be converted to private committed and would be backed by the page file.
  • This article truly is not clear about the relevance of the Commit Charge to performance. It is my understanding that for a specific combination of processes and such in a system, the Commit Charge will be the same regardlous of the amount of physical memory. Some people seem to think that the Commit Charge indicates whether pysical memory is adequate (or correspondingly inadequate) and if it does then I don't understand how Commit Charge is relevant.

  • Some errors in your definitions:

    Committed Memory – "When an application touches a virtual memory page the page becomes a committed page.  It is now backed by a physical memory page"

    Not quite: a page becomes a "committed page" when an application commits virtual memory.  At that time, the Memory Manager deducts from the system commit limit and charges it to the Commit Charge Total.  If and when it is actually touched, then the memory manager backs it with a physical page (that part you had correctly).

    Commit Limit - "If you had 50 applications fully allocate their 2 GB of virtual address space, you would need 100GB of commit limit (ignore kernel memory usage to keep the numbers simple)"

    Not quite: if you had 50 applications fully allocate their 2GB with PRIVATE committed address space, then what you said is true - but if you had 50 apps with 2GB of committed READ ONLY memory (e.g. shared memory or memory mapped files), you would need little or no pagefile space (or RAM) to support this.  Again, only private nonshareable process virtual memory counts against the commit limit (not shareable stuff like code in EXEs or DLLs).

    Standby list: "It is still associated with the process, but not in its working set."

    Not quite: it may not be associated with a process if all references to the file have been closed; however, it is still available to a future process that opens and references that cached file data.

    PF Usage is wrong - it is (in Task Manager on XP) *identical* to Commit Charge Total.  Your definition (# number of pages that have at one time been pulled out of a process' working set and committed to the page file) is completely off.

    Also, my new instructor Alex Ionescu has a new tool (for Vista and Server 2008 - doesn't run on XP/2003) that shows a superset of system memory information than what the kernel debugger !meminfo shows - and unlike local kernel debugging on Vista which requires booting in Debugging Mode, this tool doesn't require that... see http://www.alex-ionescu.com/tools.zip (he will be posting a blog post on his site to explain these in the future).

    David, Thanks for your feedback. Your clarifications are absolutely correct. The posting was written as a primer for memory manager concepts. It is hard to generalize this complex system. I had to keep the examples simple and brief to easily illustrate the basic concepts. Many things above have some purposefully omitted detail that can be explored further in your book (linked in the articled). Also, thanks for catching the PF Usage error. I’ve updated the definition above.
  • Excellent article, thanks! So good, that I have some questions on it:

    "If the process doesn't use the page for a very long time, or if the demand for the page is greater elsewhere, the page is moved off of the standby list."

    1. What does that "very long time" actually mean: 10 seconds, 10 minutes, an hour or a week?

    2. What is the reason to remove standby pages from RAM if there is no need in additional RAM?

    Of course, there might be a demand for more available RAM for other processes in future, but likewise in future these standby pages might be required by the process they belong to.

    [Here are the answers to your questions:

    1. Good observation. The working set management algorithm is largely demand based. We try to balance current demand with possible future demand. If there aren't enough free or zeroed pages to meet the system's need, the working set manager will start trimming the least recently used pages. After the zeroed and free pages, we pull from the standby list. If there will never be any future demand, or there are always enough zeroed pages, then the standby list could remain as is. This scenario; however, will most likely never occur. There will always be demand for new pages. Dependent on your system's rate of demand for new pages, older pages on your standby list will be removed. There isn't a fixed time of when your standby page will be scrubbed and passed on, it is just tied system activities like file cache usage, process startup and end and current process memory usages. Given enough time and enough demand for new pages and if the process doesn't soft fault back in the standby page it will be recycled.

    2. If there is no more demand for new pages then we don't remove pages from the standby list. This; however, usually is never the case. You system is almost always doing something with memory. Some processes are requesting new pages, others releasing some. Some processes may be working with cached I/O, other processes could be started or ending. There is always constant churn on the memory pages. Over time, typically you'll see older standby pages removed from the standby list and given to some thread that requested a new page. As long at this page continues to be used, it will remain active for the process. Eventually it may be freed or not used in a long time. If it sits long enough on the standby list, eventually it will most likely be recycled and given to someone else who needs it.]
  • Thanks for the great post.

    If the Total Physical Memory is way more than the Peak Commit Charge, would there ever be the case where the memory manager will ever need to convert any of the standby page list to zero page list or free page list?

    Also, is there a way to monitor the value of the standby page list without having to install Windows debugger?

    Thanks.

    [Yes. The Memory Manager many need to scavenge old pages from the standby list for a variety of reasons. Here are just two examples: 1) Processes starting and stopping - When a process starts, it will get pages from the zero, then the free page list. If there aren’t enough pages, the Memory Manager will pull pages off of the standby list. When a process terminates, its pages will be placed on the free page list. If there is more demand from staring processes than stopping ones, standby pages will be converted. 2) Memory demand from different processes. If some processes have a spike in memory consumption and that demand cannot be met from the free or zero page lists, then the Memory Manager will pull pages from the standby list. Some page churn can be expected as pages change ownership between processes and the system file cache. Typically you won’t notice this as the more used pages tend to remain in working sets or stay long enough on the standby page list to be soft faulted back in.]
  • Thanks again for a super post. It sure has generated a good debate on this subject, which I think needs more of, These posts only serve to give a greater understanding of how memory is broken down and how its interaction works within a system. It lends itself to taking a bit of the mystery out of this complex topic Cheers!

Page 1 of 1 (12 items)