In the last session, focusing on virtual memory, it was noted that there was almost no connection between virtual and physical memory. The only connection is that the system commit limit is the sum of physical memory and the size of the paging file(s). This session focuses on the physical memory aspects of the memory management architecture in Windows.
The working set is the physical memory that Windows gives to your process, given to you when you demand it by touching the virtual address space. A process starts with an empty working set – as it references pages that aren’t in the working set, it incurs a page fault. Many such page faults are resolved simply from memory (for example, if you load an image that is already in the file cache).
Each process has a working set default minimum and maximum. Windows doesn’t pay any attention to the maximum working set; Windows uses the minimum to commit that amount of memory, as well as to lock memory.
At some point, the memory manager decides the process is large enough; it then gives up pages to make room for new pages. If your process is ballooning, the memory manager will tend to favor it because it is obviously memory-hungry; but at some stage, it will start to pull pages out of the working set that are least recently used before it will allocate new memory to that process.
Unlike other OS, it is a local page replacement policy, meaning that it will pull the pages from the requesting process.
The working set consists of shareable and private pages; there are performance counters that measure this. Shareable working set is effectively counted as “private” if it is not shared. Note that the working set does not include trimmed memory that is still cached.
Windows Task Manager only offers the private working set figure; Process Explorer gives you a more detailed view. Remember – it’s only when you touch the memory that is committed that it becomes part of the private working set. Comparing testlimit with –m, –r and –d switches demonstrates the differences here.
The system keeps unassigned physical pages on one of several page lists in the Page Frame Number (PFN) database, each maintained as FIFO list:
Note that meminfo gives you a way to investigate these page lists: Alex Ionescu provides further helpful commentary on the tool on his blog.)
When Windows starts, all memory is in the free page list. As a process demands memory, it is faulted in from the zero page list (however, the free page list can be used if the memory is to be used for a mapped file, since the data will be overwritten before the process sees it). If a page is pulled from a process, it may be moved to the modified or the standby page, depending on whether it contains private data that needs to be flushed or not. When a process exists, its pages are moved to the free page list.
There are two threads that are specifically responsible for moving threads from one list to another. Firstly, the zero page thread runs at the lowest priority and is responsible for zeroing out free pages before moving them to the zeroed page list. Secondly the modified page writer is responsible for moving pages from the modified page list to the standby page list, first writing out any data.
What happens if the zero and free pages are exhausted? The memory manager starts using the standby page list; if that is exhausted, then the modified page list can be used (once the modified page writer has flushed the content and moved it to the standby page list). If all these pages are exhausted, the last resource is to remove memory from the working set.
With this information, we can more helpfully interpret the data provided by Task Manager.
Rammap is another useful tool, showing the most detailed breakdown of what’s going on in RAM at the time it takes the snapshot. It breaks down all the memory in the system by type and page list. For processes, it shows the private working set for each process as well as memory for that process on the standby or modify page list as well as kernel memory in the page table.
As you can see, having a low “free memory” value in Windows is normal – and a good thing. In fact, a technology like SuperFetch has the goal of reducing free memory (moving pages from the free to the standby list as it caches files into memory).
SuperFetch is used to proactively repopulate RAM with what it believes is the most useful data: it takes into account the frequency of page usage, and the usage of the page in the context of other pages in memory. Interestingly, the choices it makes are also influenced by a time-based heuristic – it understands that users launch different applications at the weekend than during the weekday, and at the evening than the daytime. SuperFetch is disabled in Windows 7 if the operating system is booted from an SSD drive, because the primary benefit is to reduce random read seeks on a platter-based hard drive.
It’s worth noting that there are actually eight standby page lists in practice on Windows Vista and above; this is used to ensure that the least important standby pages are scavenged first. (Rammap can show you how much memory has been repurposed from each of the standby priority lists.)
A common question: do you have enough RAM in your system? There’s actually no sure-fire way to tell if you have enough memory. If available memory is low for a considerable length of time, then you probably could use more. You can check Process Monitor to see if there are an excessive number of reads from the paging file (there’s no performance counter for paging file reads).
Mark closed with some examples of how certain memory usage can be hard to or inspect or detect: a shared memory leak, large quantities of reserved but uncommitted memory, locked memory, driver locked pages and AWE pages.
[Session CD02 | presented by Mark Russinovich]
Windows has both physical and virtual memory. Memory is managed in pages, with processes demanding it as necessary. Memory pages are 4KB in size (both for physical and virtual memory); but you can also allocate memory in large (2-4MB, depending on architecture) pages for efficiency. In general, there are very few things in common between physical and virtual memory, as we’ll see.
On 32-bit (x86) architectures, the total addressable memory is 4GB (232 bytes), divided equally into user space and system space. Pages in system space can only be accessed from kernel mode; user-mode processes (application code) can only access data that is appropriately marked in user mode. There is one single 2GB system space that is mapped into the address space for all processes; each process also has its own 2GB user space. (You can adjust the split to 3GB user / 1GB system on PAE-enabled systems, but the 4GB overall address space limit remains.)
With a 64-bit architecture, the total address space is theoretically 264 = 16EB, but for a variety of software and hardware architectural reasons, 64-bit Windows only supports 244 = 16TB today, split equally between user and system space. Mark: “I think I’m pretty safe in saying that we’ll never need more than 16TB, but people smarter than me have been proven wrong with statements like that, so I’m not sure if I should make that claim or not!”
Within a process, virtual memory is broken into three categories: (i) private virtual memory – that which is not shared – e.g. the process heap; (ii) shareable – memory mapped files or space that you’ve explicitly chosen to share; (iii) free – memory with an as yet undefined use. Private and shareable memory can also be flagged in two ways: reserved (a thread has plans to use this range but it isn’t available yet), and committed (available to use). Why should you reserve memory? It allows an application to lazily commit contiguous memory. For example, a stack: the system only needs to commit a small amount in advance, and can grow as appropriate by committing previously-reserved memory.
Task Manager only lets you see the private bytes (“commit size”); indeed, most virtual memory problems are due to a process leaking private committed memory, since it backs the various heaps in the system. But this doesn’t offer a full view of the address space – it doesn’t account for shareable memory that isn’t shared (e.g. a DLL only loaded by this process). Fragmentation can also be an issue, causing the address space to be exhausted prematurely. Process Explorer lets you split out the overall virtual memory used by a process from the private bytes; another useful tool is VMMap (with a new v3.0 released for PDC10), which provides a really deep level of inspection into the virtual memory used by a given process.
In the example above, the committed memory is about 54MB, but only a tiny amount is private. The middle section shows how the memory is allocated – images (DLLs and executables), mapped files, shared sections, heaps (unmanaged and managed), the stack region, private data (memory that is private and doesn’t fall into one of the other categories, e.g. if you call VirtualAlloc()), and page tables. If you click on one of these rows, you’ll filter the details section at the bottom of the window.
Fragmentation can be an issue, as mentioned earlier. Mark demonstrated creating a large number of threads with testlimit –t. The number of threads he was able to create on a 32-bit machine was limited to ~2900 even though in theory he should have been able to create more, because the address space was so fragmented that there were no more contiguous spaces of 1MB or more (1MB being the default size).
Mark then demonstrated how mapped files are used heavily by Explorer. Files are mapped regularly so that it can get access to the resources and metadata from within them. You can trace file mappings with Process Monitor – so if you don’t understand why a file is mapped, you can see how the DLL got loaded by capturing a snapshot filtered to “Load Image” operations and examining the stack at the point the image was loaded.
You can also use VMMap to compare the difference between two snapshots of a process. In the given example of loading a file into Notepad, the committed memory grew from the 54MB above to over 100MB – with most of it coming from image loading.
New in VMMap 3.0 is tracing, which enables you to launch a process with profiling and track virtual and heap activity over a period of time.
Certain kinds of process committed memory are charged against the system commit limit. System committed memory is that which is backed either by the paging file or physical memory. Not all memory in the address space is system committed memory – for example, the notepad.exe image or its DLLs are backed by disk, so if the system tossed out of memory and Notepad wanted to access it again, it could simply retrieve it from notepad.exe on disk.
When Windows needs to reallocate private memory, it has to store it in the paging file (otherwise it would be missing when it needed to retrieve it). When that limit is reached, you’ve run out of virtual memory. You can increase the system commit limit by adding RAM or increasing the paging file size: if paging files are configured to expand, that means that the system commit limit may not actually be the maximum limit.
So how is the paging file size determined? By default, it is automatically managed by Windows based on the RAM available: on systems with >1GB RAM, the minimum is equal to the size of the RAM and the maximum is 3x RAM or 4GB (whichever is larger). You can manually adjust this, however.
What’s ironic about this algorithm is that in the dialog, the recommended value is often a completely different number (on my 8GB machine, it’s 1.4x RAM), which as Mark notes fails to inspire confidence in the algorithm.
You can view system commit charge and limit from Task Manager or Process Explorer; you can test exhausting the system commit limit by running testlimit –m. (Be careful – this will stress your machine to its limit!)
So how should you size your paging file? Many people will give you advice based in multiples of the RAM, but that isn’t good advice. You should look at the system commit peak for the most extreme workload and if you want to apply a multiplication factor (1.5x, 2x), apply it to that value instead. Note that it’s important to ensure that the paging file is big enough to hold a kernel crash dump. Although you can turn off the paging file altogether, it is useful – the system can page out unused, modified private pages, providing more RAM for real workloads.
[Session CD01 | presented by Mark Russinovich]
Now that the keynote is over, my formal duties for the event are mostly complete. Lots going on still, and I hope to meet a number of you in person – but I thought I’d take a little downtime and watch a few sessions. Rather than just greedily hoarding all the knowledge I learned for myself (incidentally, did you know that Apple require you to sign a non-disclosure agreement to attend their breakout sessions?), I’d prefer to share my notes with you. So over the next few posts you’ll hopefully get some useful distillations of several client-related sessions from the event.
It goes without saying – if you don’t like my choices, you can watch it all live yourself online at http://player.microsoftpdc.com/session. Let’s see how this goes!
Yesterday I shared the story of the Steve Ballmer keynote demo that was breaking and the urgent call I got to help figure it out.
I left you hanging as to the solution; a few of you posted interesting ideas of what might have gone wrong. But Richard Cooper was the first to figure it out – congratulations!
The answer can be found in the defaultnew-msh.js file. Here’s the relevant function:
(function(){
if(typeof window.external.msIsSiteMode!="undefined"){
function d(){
var o=new Date;
if(o.getHours()==8||o.getHours()==9||o.getHours()==10||
o.getHours()==11||o.getHours()==12||o.getHours()==13){
window.external.msSiteModeClearJumpList();
window.external.msSiteModeCreateJumpList("MyPlate Reminder");
window.external.msSiteModeAddJumpListItem("Track Your Breakfast",
"/myplate/","/media/images/ico/bell.ico");
window.external.msSiteModeSetIconOverlay("/media/images/ico/bell.ico",
"Reminders");
window.external.msSiteModeShowJumplist()
}
if(o.getHours()== 14||o.getHours()==15||o.getHours()==16||
o.getHours()==17||o.getHours()==18||o.getHours()==19){
window.external.msSiteModeAddJumpListItem("Track Your Lunch",
if(o.getHours()==20||o.getHours()==21||o.getHours()==22||
o.getHours()==23){
window.external.msSiteModeAddJumpListItem("Track Your Dinner",
window.setTimeout(function(){
d()
,3E5)
})();
Now you can hopefully see that the notifications are set up based on the time: the site authors had designed around certain time windows when users would be tracking breakfast / lunch / dinner. But (by design) there’s no notification between midnight and 8am.
I was tempted to tell the demo team that the demo would magically start working in about half an hour or so, but I decided that was a bit unfair to their blood pressure. They set the clock forward an hour, and the demo ran flawlessly.
It's a good example of how demos can be unpredictable - despite your best efforts to rehearse and prepare, there are so many variables in the system for a keynote demo that you can expect something unexpected.
I’m sat in the PDC keynote room this morning for the final run-through – we finished rehearsing last night at 1am and we started this morning at 5am. Behind the scenes, there’s a whole conference room full of equipment: video streaming equipment to broadcast the event around the world, a bank of about forty demo machines (primary and backup for each demo), video and lighting control, a slide editing suite and site production services.
There are a whole host of challenges that are only exposed in this scenario: as an example, we need to remote a USB cable from a Windows Phone to a PC backstage (and provide a second one as an emergency backup); we need to provide consistent and reliable networking for myriad devices even when the wifi and cellular networks are saturated; we need to be able to switch between all forty machines from a single demo podium. Last night, we saw a Windows 7 error message that I’d never seen before – we’d exceeded the USB specification by chaining five hubs together in a certain configuration. Given all this, you can probably imagine why demos can fail: ironically, there comes a point where redundancy introduces so much complexity that it actually increases the risk of failure.
Crossing fingers for a demo failure-free morning! Be sure to tune in, won’t you? We’re broadcasting live over the internet.
Tomorrow is the start of the PDC, and while I have a quiet moment before the final keynote rehearsals start, I thought I’d share this little story with you.
I got an urgent email very early yesterday morning as I was just waking up from a colleague who was about to go on stage to do a demo in a Steve Ballmer keynote. The demo showed how IE9 enables web apps to feel more like Windows desktop apps, using features like pinned site mode, jump lists, notification icons and so on. The issue was that the notification icons just weren’t popping up on cue. They’d worked perfectly well the night before during rehearsals, but all of a sudden they weren’t appearing.
Everyone was racking their brains to try and figure out what was going wrong. Had someone tinkered with the demo machine? Had the website changed? I’ve been there before, and you can probably imagine how stressful it is to hit a demo issue so close to a CEO keynote.
From the relaxed atmosphere of my bedroom, I decided to go have a quick look at the code to see why it was failing. I was amused to discover the source of the issue. Rather than jump straight to the answer, I’ll give you the chance to figure it out for yourself – see how long it takes you, and go ahead and post in the comments section when you’ve figured it out.
The site in question is LiveStrong – co-founded by the Lance Armstrong Foundation. (Hit F12 when you’ve opened the link below to open the IE Developer Tools.)