Myth: Without /3GB a single program can't allocate more than 2GB of virtual memory

“Out Of Memory” Does Not Refer to Physical Memory

Fabulous Adventures In Coding — Thu, 11 Jun 2009 10:05:10 GMT

I started programming on x86 machines during a period of large and rapid change in the memory management

re: Why is the page size on ia64 8K?

The Old New Thing — Wed, 08 Sep 2004 18:22:00 GMT

re: Myth: Without /3GB a single program can't allocate more than 2GB of virtual memory

Raymond Chen — Sun, 22 Aug 2004 15:11:00 GMT

"Allocate means it is available in my address space."

No! You missed the whole point of this entry.

"Mapped" means it is available in your address space. "Allocate" means that the memory is committed to you.

You can "actually" use the memory - my ReadByte function shows how. It's not convenient, but it is nevertheless actual.

re: Myth: Without /3GB a single program can't allocate more than 2GB of virtual memory

Jack Tripper — Sun, 22 Aug 2004 14:51:00 GMT

Something that you are failing to deal with in this series of articles, that makes all their information lost:

explain the different between allocated memory and mapped memory.

If i want to allocate 2.5GB of memory, and i don't have the /3GB switch, then i can't use 2.5GB of memory.

You try to talk about tricks of mapping parts of files and whatnot. But i want to map a 2.5GB file, i want to access it all; and i can't. And that is because without the /3GB switch i cannot allocate more than 2GB.

Dwell on the difference between *technically* somewhere there is more than 2GB of memory set aside for stuff, and *actually* not being able to use more than 2GB.

Allocate means it is available in my address space.

The story of the 3GB server switch

Savage Nomads — Fri, 20 Aug 2004 20:44:00 GMT

  As Evan already mentioned on his blog, Raymond Chen has a great series on /3GB switch on his blog. What is really cool is that Raymond takes on some myths about the /3GB switch and  the fact that he...

/3GB Switch

Welcome to Flaphead.com @ Home — Wed, 11 Aug 2004 16:09:00 GMT

re: Myth: Without /3GB a single program can't allocate more than 2GB of virtual memory

Simon Cooke [exMSFT] — Wed, 11 Aug 2004 15:44:00 GMT

Andreas wrote:
> Yeah, I got that, but if you do this setup
> at a time when there's only one thread (e.g.
> at startup) it should be safe, right?

Because of load-time DLL injection, and depending on the version of the OS and which libraries it needs, there's no real way of 100% ensuring that.

Paul Walker wrote:
> Simon - I'm not sure that mapping the buffer
> twice helps speed that much. Okay, it means
> that you don't have to access byte-by-byte
> the whole time, but in practice you wouldn't
> do that anyway, and you still have to do the
> comparison to buffer-size in order to wrap
> once you have gone off the end.

You only have to do the comparison to buffer size when you add or remove data from the buffer - which is infrequently. And even then, you only need to adjust your head pointer when you remove data.

Compare it with the other ring buffer implementations out there on CodeProject et al. Either you can make things really complicated, and deal with the buffer in two halves, or you access it a byte at a time.

Also, another advantage of this is that if you use it with overlapped IO, you can set things up such that you can be feeding data into one end of the buffer, consuming it from the other, and whenever you try to do a Send or Write, you can always send the whole contents of the buffer in one go. If you're using zero-copy writes/sends, this becomes quite important (eg. high-perf networking).

> Regarding the interleaving of memory - with
> all the stuff Windows lets you do to its
> internals, isn't there a call to temporarily
> lock the scheduler?

Nope. At least, not in user-mode code. That'd kind of defeat the point of the pre-emptiveness of the scheduler. You can temporarily boost yourself to realtime priority, but that (again) isn't 100% foolproof - you could have other realtime threads to compete with. Another option is to suspend all other threads in your process, and then resume them, but you risk setting up race conditions and losing state that way

re: Myth: Without /3GB a single program can't allocate more than 2GB of virtual memory

Raymond Chen — Wed, 11 Aug 2004 14:47:00 GMT

Alas that's not how the i386 works; more on this is scheduled for a future entry.

re: Myth: Without /3GB a single program can't allocate more than 2GB of virtual memory

dave stokes — Wed, 11 Aug 2004 14:16:00 GMT

Raymond -

"Please elaborate on what you mean by "address it directly". You can do that today. It's called a 32-bit pointer."

Yeah, I'm just a dumb guy. But I'd have thought my meaning was actually pretty clear from everything else I wrote, I'm talking about addressing the 4GB you've mapped to what you've chosen to call "virtual memory". You can't address that directly, only as you say yourself by mapping parts of it to a window in your 2GB address space.

With the hardware mechanisms I've mentioned in a previous thread, data in a seperate address space can have a full range of say 4GB which can be directly addressed by suitable hardware registers, for what it's worth.

re: Myth: Without /3GB a single program can't allocate more than 2GB of virtual memory

Paul Walker — Wed, 11 Aug 2004 13:15:00 GMT

Regarding all of this... Linux makes distinctions between various different types of address, such as:

* user virtual addresses (seen by user-space code)
* physical addresses (used between CPU and memory)
* bus addresses (used between peripherals and mem)
* kernel logical addresses
* kernel virtual addresses

I'd imagine Windows makes at least the same distinctions.

For what it's worth, I've never heard physical memory used to refer to anything except RAM chips. Certainly not swap.