Still more misinformation about virtual memory and paging files

Never believe anything you read on /. -- it's a shadow of its former self; mostly populated by a bunch of trolls and ne'er do wells, the editors are so far up themselves they're in danger of becoming Klein bottles, and it's nothing more than a mouthpiece for press releases these days.

Even Linux has the idea of a working set, although the Linux algorithm (http://www.linux-tutorial.info/modules.php?name=Tutorial&pageid=311) appears to be based on the number of free pages in the system as a whole, unlike NT which IIRC treats each individual working set (each process plus the system working set) separately.

My source as always is "Windows Internals, 4th Edition" by Mark Russinovich and David Solomon.

It's worth being clear that the system working set ('Memory: Cache Bytes' performance counter or 'System Cache' in Task Manager) does not just cover the file system cache, but all kernel-mode pageable code and data, including paged pool. Some memory is double-counted in 'Available' and in 'System Cache' in Task Manager - I've actually seen the sum of the two exceed the actual amount of memory available on my 1GB system at work.

Yes; why don't we bitch about his guide... But wait, where is *your* optimization guide?

PS - You need to write to him and inform him that a course in MS Dos 4.0 is a prerequisite for publishing articles on the internet.

(I’m sorry if this comes off a little trollish but so do your comments on his article)

Larry,

As your comments above seem reasonable and you seem to know a thing or two perhaps you can help out. I have been looking for a way to make WinXP limit the size of memory it will use for Disk Caching functions. Win9x was configurable through the system.ini with:
[vcache]
MinFileCache=4096
MaxFileCache=32768

This was a wonderful adjustment given that Windows will try to use all "free" memory that is available for disk cache when anything over what is now a small amount of memory has very negligable, if any gains.

It's a pretty sorry statement when you have three programs running and one of them is writing a lot of data to disk (Photoshop), and windows decides to swap your browser or mail program out to disk because 700 MB just isn't enough disk cache.

Any ideas? Or contacts that could solve this problem?

Shouldn't one always read EVERYTHING that the writer has to say before giving out comments?

Plus, at least he tried and write the article. Not accurate? SO what, help the author correct it to be so and help save the world for further condemnation with wrong information!

I just keep seeing bashing saying I'm right and you're wrong but no further actions to help right the wrong.... is that smart? I think not.

If its any consolation, the article was written in 1999. And throughout the article claims are made but not backed up with empirical evidence. We can see how a pagefile or swap file or whatever it is you wish to call it can be taken from across the disk to some edge from pretty pictures, and we can see from a graph that the outer edges of a disk will run faster, but from those two graphics the author infers that moving the swap file has a 16+ percent improvement. Sure, there's some theory on seeking and reading added, but there's also latent assumptions about the workload strewn in there.

The biggest problem with the article is that it needs a big "In theory," prepended to every statement.

Actually the date on the post is December 2004, not 1999 - that's only 4 months old.

And I tried to read the article. But 5 minutes/page load (mostly due to a dead lan connection that was massively impeding my ability to get real work done) kinda took my wind out of it.

I gave up after page 4.

Manip, my comments were 100% trollish. And I HAVE written about memory management before:
http://blogs.msdn.com/larryosterman/archive/2004/03/18/92010.aspx

And the correction article I wrote to call out the screw ups in the first one:
http://blogs.msdn.com/larryosterman/archive/2004/05/05/126532.aspx

I'm QUITE sensitive to this particular issue, having had my hand slapped quite hard on it (the dev manager in charge of memory management, in an email to Brian Valentine, used my article as an example of the unbelivably bad information about memory management that's being put out on the web).

So when I see popular forums promoting this kind of drivel, I get annoyed.

If y'all would like, I'd be more than happy to do a sentence-by-sentence set of corrections, but it'd just come off as being even more trollish - the article's got that many errors.

Having now finally finished the article (from home, where the net works :)), once he gets off his discussion of how virtual memory works and starts discussing how to relocate a paging file and where to put it, his article isn't that bad. Essentially once he gets beyond his discussions about permanent and semi-permanent paging files, the final parts of his discussion (where he discusses the various issues related to spindles etc) doesn't appear to be that incorrect (although I'm not sure about the RAID stuff).

Alan, check the two articles I linked above - the reality is that Windows will use your paging file as much as you're asking it to use, and no more. Windows, in general, won't ever page out an application unless some other application is using more memory than is physically available on the machine - the only time pages will be discarded from the standby list is if someone else (photoshop, in your case) needs them.

And Windows tries REALLY hard to ensure that it doesn't throw pages out that it doesn't have to.

Memory management is a difficult subject.

A certain large company keeps its details secret.

As raymond demonstrated in "how much memory can your app access" series (a per process view of memory), most comments were about people's machines.

How about you go get "dev manager in charge of memory management" to answer technically common questions.

In the absence of "official" data all I can say to "how do I disable paging to make my paid for app run faster" is "the processor is designed to page, the chipset is designed to page, Windows is designed to suit the hardware, so is also designed to page". This is not really an answer so I normally avoid debates.

Theory is supposed to be a guiding thought for action. So how does one apply an unknown implemtation to guide ones actions in common user senarios.

Larry, I haven't read the linked article but I don't understand your criticism regarding the paragraphs you quoted.

As for the 4 MB in a DOS computer thing, are you unaware of an old technology called DOS extenders? EMS/XMS drivers certainly allowed DOS to access 15 MB, if in a rather inconvenient way (page swapping). I know I had and used more than 1 MB RAM in my 286/386 DOS systems... hard disk caches were a popular candidate for XMS storage.

As for the second paragraph, it sounds certainly "good enough" as a high-level description on what virtual memory does. Or did the author claim to deliver a detailed description of the NT kernel? Or what exactly is your criticism?

I'm having problems with paging right now. The latest WinFX CTP shipped as a 460mb MSI file. It takes an incredible amount of time to install, during which the PC thrashes like crazy.

I have 512mb of RAM, with normally 300 or so free. When installing the CTP first explorer, then two msiexec processes in turn allocated the about 400mb each to process the install script. It took well over an hour just to extract the MSI, which only contained a 455mb ISO image, a readme and an EULA. My peak commit rose to 824mb.
I think I would far rather have been able to download the ISO image seperately and saved all the hastle.

I don't think the Windows installer system was designed to support such large files. The wrong tool for the job.

I think what the author said about the OS gradually paging apps out to disk as data is loaded is correct, but not for the stated reason.

Try typing this in an NT command prompt with a dir that has a lot of big files:

for /r %x in (*) do @copy "%x" nul

CMD itself uses a neglegible amount of memory but you'll find that most of your other apps are paged out by the time it finishes. My understanding is that the NT memory manager treats the cache manager like any other process, so when it sees the cache manager memory mapping a lot of data, it gradually enlarges its working set of the cache, paging other app memory out in the process. As Alan noted above, this happens in Windows 95/98 as well, even though the VCACHE architecture is different.

In other words, I don't think your assertion that Windows only throws out pages when programs need them is quite correct -- when the disk cache is involved, Windows will also page when it thinks the app *might* need more memory for caching. This behavior was very useful in the days of Windows 95 when memory was very tight and paging memory to increase the disk cache from 1MB to 4MB helped considerably, but nowadays doing it to enlarge the cache from 500MB to 800MB isn't as helpful.

Edward:
What OS (and service pack) were you using? Running anything else at the same time or was this solely because of the install? How 'bout your hardware: recent-ish CPU and enough defragged room on your HDD that those couldn't have been bottlenecks? And this hour was install time alone, not including download time?

I can't publicly promise you many answers about installation performance, but I might be able to get the installer folks to look at it. Or the team that shipped the CTP. An hour to install their package seems way too long.

Setup lag (especially w/ large files) makes me nervous because my team's code is in their critical path. We (cough) map a view of an entire .exe or .dll to hash it. I doubt that's a bottleneck that would make your installation take so long, but it could very well explain why your peak memory usage was so high.

That was an implementation detail (which may change in future releases) but it ought to be so incredibly obvious to anyone who runs WinVerifyTrust calls under a debugger or even verifies signatures on big PEs that I'm assuming I don't need to talk to lawyers, my bosses, or HR tomorrow morning. :-)

Phaeron,
In that case, some program needed to use the memory. Now it only needed to use it to copy a file, but it DID need to use the memory.

Having said that, there IS an issue that I purposely ignored. If you have a boatload of low frequency threads running (like a bunch of tray icons that wake up to check the internet to see if a new version of software's available), then the system needs to page those apps in.

And that may, in turn cause the pages that hold your application to be paged out. Even though you, the user weren't using those pages, someone wanted to use them.

This particular issue's been one of great concern to the memory management people in Windows - they know that this leads to highly negative user experiences and they're working really hard to fix it.

Sorry for the length of this post. One thing led to another and...

Drew wrote:
<i>We (cough) map a view of an entire .exe or .dll to hash it. I doubt that's a bottleneck that would make your installation take so long,</i>

I don't. If the MSI file is in this case considered to be an exe and you map it, obviously the memory manager will happily swap in new pages into your process working set when you touch them sequentially. Since the previous page(s) in this file are likely most recently touched in the system, this will basically uselessly grow, and grow, and grow the physical memory used - even if you in reality need nothing more than a sliding window of, let's say, 64KB - and wreak havoc on any system that doesn't have the amount of RAM free at least as large as the mapped file (and then some).

The way I think it's best solved is by just not map such insanely large files, but instead CreateFile non-cached. Unfortunately one can't say "not really cached, but still do a little read-ahead (by setting the sequential flag)", why it has to be non-cached.

To optimize, one could allocate two buffers (say 1MB each) and switch between the two when using asynch I/O - fill one buffer while performing calculations on the other.

It will make the code more complex - once, and once only. The end-user benefits should stand out like the sun burns in your eyes after a long night of hacking (which, conincidentally, is way more time than is required to implement this). :-)

One could probably even do this by using two mapped sections, 64KB each, and alternate between them when trying to access the data by catching the memory exceptions and remap the other buffer and... The sky^H^H^HWin32 API is the limit. :-)

Furthermore, since an installation program knows (or at least should know) it will only read the source files once, and then they are likely not to be used again in foreseeable time, it also knows those files should be opened non-cached.

Having been hurt by NT5.0 fc.exe while comparing large files, I realized Microsoft hadn't at all considered the issue of mapping (the whole of) files way larger than available (or even free) RAM. At least I hope it's an oversight, that would be hilarous (as in "how the heck could you miss THAT?" with an evil grin) if it wasn't hurting so much (in lost time).


Larry, I don't agree that copying files to nul needs to cache the files (thereby messing up the whole memory behaviour on the system for a while). Why can't it open them non-cached? I don't see any read-ahead issue here, since the process hasn't got anything to do but waiting for I/O anyway. OK, a user might use this to behaviour to explicitly put files in the cache. If that's a concern, could perhaps a new flag ("/nocache"?) be added to e.g. fc.exe, xcopy.exe and cmd.exe's copy command, just to name a few?


I might also add that I think for copy operations, the application (or library) itself is often in a way better position to judge how caching should be done. Consider copying files (especially large files) between two volumes, residing on the same disk vs. on different disks.

Copying files around on the same physical disk could greatly benefit from the application/library doing the copying doing its own buffering (f.ex. CopyFile(), xcopy.exe, cmd.exe's own copy, ...) , filling up a buffer a bit larger than 64 or 256KB before writing that buffer to another place on the same disk. This I know, since I have written copy functions several orders of magnitude (!) faster than what many, incl. MS itself uses in many places (installers are historically horribly bad in this area - installers originally intended for packages of "reasonable" sizes now still treat files several gigabytes large the same as a 2KB file).

When copying files between two physical devices, buffering becomes less of an issue and streaming is what more counts. I believe there are even MSDN samples of how to do this. :->

Either way, bogging down the cache (and therefore the whole system) when copying, or even just reading files so large they can't ever fit the memory (or even the cache) of a system, especially for sequentially read/write operations like when copying a file or calculating a checksum, is just dumb. Turn off caching for those file operations.


When copying in general (a number of) files only a fraction of the size of cache-granularity (assuming the still hardcoded 256KB virtual address space granularity I read about somewhere), especially if done by an installer that should know the source files will not be used again, I think it should do it's own caching of them.


(while on the subject, I'd like a new FSCTL for NTFS - to tell it to use more memory when relocating a file during defrag operations - when 100MB are free, I see little reason in not using at least a 10MB buffer, instead of the current seek-time-excercise of HD's).

Larry: there is a typo in your text: "fundimentally"

Mike Dimmick: working set is not "an idea", it is precisely defined replacement algorithm developed by Denning (http://cne.gmu.edu/pjd/PUBS/bvm.pdf). According to publically available information, Windows kernel uses modified version of working set algorithm.

Linux, on the other hand, uses MACH/BSD active/inactive queues and physical scanning as a basis for its per-zone page replacement, which is _quite_ different from working set.

2.2 Linux kernel did use virtual-based scanning that is somewhat closer to working set.