Fabulous Adventures In Coding : Memory Management

Absence of evidence is not evidence of absence

Eric Lippert — Mon, 12 Oct 2009 13:51:00 GMT

Today, two more subtly incorrect myths about C#.

As you probably know, C# requires all local variables to be explicitly assigned before they are read, but assumes that all class instance field variables are initially assigned to default values. An explanation of why that is that I sometimes hear is "the compiler can easily prove that a local variable is not assigned, but it is much harder to prove that an instance field is not assigned. And since the class's default constructor automatically assigns all instance fields to default values, you don't need to do the analysis for fields."

Both statements are subtly incorrect.

The first statement is incorrect because the compiler in fact cannot and does not prove that a local variable is not assigned. Proving that is (1) impossible, and (2) does not give us any useful information we can act upon. It's impossible because proving that a given variable is assigned a value is equivalent to solving the Halting Problem:

int x;
if (/*condition requiring solution of the halting problem here*/) x = 10;
print(x);

If what we wanted to do was prove that x was unassigned then we would have to at compile time prove that the condition was false. Our compiler is not that sophisticated!

But the deeper point here is that we're not interested in proving for certain that x is unassigned. We're interested in proving for certain that x is assigned! If we can prove that for certain, then x is "definitely assigned". If we cannot prove that for certain then x is "not definitely assigned". We're only interested in "definitely unassigned" insofar as "definitely unassigned" is a stronger version of "not definitely assigned". If x is read from when it is "not definitely assigned", that's a bug.

That is, we're attempting to prove that x is assigned, and our failure to prove that at every point where it is read is what motivates the error. That failure could be because of a bona fide bug in your program, or it could be because our flow analyzer is extremely conservative. For example:

int x, y = 0;
if (0 * y == 0) x = 10;
print(x);

You and I know that x is definitely assigned, but in C# 3 the compiler is deliberately not smart enough to prove that. (Interestingly enough, it was smart enough in C# 2. I broke that to bring the compiler into line with the spec; being smarter but in violation of the spec is not necessarily a good thing.)

This example again shows that we do not prove that x is unassigned; if we did prove that, then clearly our prover would contain an error, since you and I both know that x is definitely assigned. Rather, we fail to prove that x is assigned.

This is an interesting twist on the believers vs skeptics argument that goes like this: the skeptic says "there's no reliable evidence that bigfoot exists, therefore, bigfoot does not exist". The believer says "absence of reliable evidence is not itself evidence of absence; and yes, bigfoot does exist". In both cases, reasoning from a position of lacking reliable evidence is seldom good reasoning! But in our case, it is precisely because we lack reliable evidence that we are coming to the conclusion that we do not know enough to allow you to read from x.

(The relevant principle for tentatively concluding that bigfoot is mythical based on a lack of reliable evidence is "extraordinary claims require extraordinary evidence". It is reasonable to assume that an extraordinary claim is false until reliable evidence is produced. When overwhelmingly reliable evidence is produced of an extraordinary claim -- say, the extraordinary claim that time itself slows down when you move faster -- then it makes sense to believe the extraordinary claim. Overwhelming evidence has been provided for the theory of relativity, but not for the theory of bigfoot.)

The second myth is that the default constructor of a class initializes the fields to their default values. This can be shown to be false by several arguments.

First, a class need not have a default constructor, and yet its fields are always observed to be initially assigned. If there is no default constructor, then something else must be initializing the fields.

Second, even if a class does have a default constructor, there's no guarantee that it will be called. Some other constructor could be called.

Third, the field initializers of a class run before any constructor body runs, therefore it cannot be the constructor body that does the initialization; that would be wiping out the results of the field initializers.

Fourth, constructors can call other constructors; if each of those constructors was initializing the fields to zero, then that would be wasteful; we'd be unneccessarily re-initializing already-wiped-out fields.

What actually happens is that the CLI memory allocator guarantees that the memory allocated for a given class instance will be initialized to all zeros before the constructor is called. By the time the constructors run the object is already freshly zeroed out and ready to go.

Iterator Blocks, Part Six: Why no unsafe code?

Eric Lippert — Mon, 27 Jul 2009 17:07:00 GMT

There are three good reasons to disallow unsafe blocks inside an iterator block.

First, it is an incredibly unlikely scenario. The purpose of iterator blocks is to make it easy to write an iterator that walks over some abstract data type. This is highly likely to be fully managed code; it's simply not a by-design scenario.

Second, the scenario is a violent mixing of "levels." You think of the level of abstraction of a programming language feature as how "far from the machine" the feature is. Unsafe pointer manipulation is as close to the machine as it gets in C#. You are running without a safety net, poking and peeking at the raw bytes in the virtual memory space of the process. All the protections of verifiable type safety are completely gone and you have total responsibility over every bit in the user-mode address space.

Iterator blocks, by contrast, are the highest level of abstraction in C# 2.0. There is no immediately obvious mapping from the source code to the generated code; it's a magic box that does the right thing because the compiler is doing some pretty massive transformations on the code, generating classes and implementing interfaces for you.

Mixing those two levels of abstraction in one method seems like a really bad idea.

Third, we would have to do some extra work to make "fixed" blocks work correctly, work that would be directly opposed to our guidelines for the correct and efficient use of the "fixed" block.

I suppose I should briefly explain what the "fixed" block does. When you call unmanaged code that needs to, say, write into the unmanaged address that is the beginning of an array of 16 bit chars, somehow you need to tell the garbage collector "hey, you, GC, running on that other thread over there -- if you get the urge to reorganize memory to be more efficient, I need you to please not move this array for a moment, because unmanaged code on this thread is going to be writing into that address". This operation is called "fixing" or "pinning" the object.

Clearly it is a bad idea to pin lots of stuff and leave it pinned for a long time. The GC works as well as it does because it is able to reorganize memory to be more efficient; pinning blocks of memory across collections impedes the GC's ability to get work done. We therefore recommend that you pin as little as possible for as little time as possible.

The CLR provides us two ways to pin something. The hard, flexible, expensive way is to explicitly get a GCHandle on the object, tell the garbage collector "this object is pinned", take the address of the object, do what you need to do, and then unpin it.

The easy way is to mark a local variable as a "pinned local" using the "fixed" statement. The C# compiler will generate special code that marks the local as "pinned". When the garbage collector does a collection, clearly the GC needs to look at all the locals on the current stack frame, because those locals are all "alive". When the GC sees that a local is pinned, it not only notes that it is alive, it also makes a note to itself that the referent of the contents of this variable are not to be moved during the collection. This is a cheap and easy alternative to explicitly getting a GC handle, but it depends on the variable in question being a local.

In an iterator block, locals are all realized by hoisting them to fields. Because we cannot pin fields, we'd have to change the code generation for the fixed block so that it took out a handle to the object, pinned it, and ensured that the object was unpinned at some point.

That's bad enough; now consider what happens when you yield while an object is pinned. Arbitrarily much time could pass between the yield and the unpin! This directly violates our guidance that things should be pinned for as little time as possible.

For all those reasons, no unsafe code is allowed in iterator blocks. In the unlikely event that you need to dereference pointers in an iterator block, you can always extract the unsafe code to its own helper method and call it from the iterator block.

***********

I hope you've enjoyed this little trip into the weird corner cases of iterator blocks. This is a complicated feature with lots of interesting design choice points.

I'm flying south to Canada to spend some time with my family on the beaver-shark infested shores of the great inland sea known as Lake Huron, so the next few fabulous adventures will be pre-recorded. See you when I'm back.

Iterator Blocks, Part Two: Why no ref or out parameters?

Eric Lippert — Mon, 13 Jul 2009 16:35:00 GMT

A long and detailed discussion of how exactly we implement iterator blocks would take me quite a while, and would duplicate work that has been done well by others already. I encourage you to start with Raymond’s series, which is a pretty gentle introduction: part 1, part 2, part 3. If you want a more detailed look at how this particular sausage is made, Jon’s article is quite in-depth.

To make a long story short, we implement iterators by:

Spitting a class that implements the relevant interfaces.
Hoisting locals of the iterator block to become fields of the class.
Rewriting the iterator block as the “MoveNext” method of the class.
Tracking the point where the last yield happened in the MoveNext method, and branching to that point directly using a switched “goto” when MoveNext is called.
Moving the logic in “finally” blocks (or equivalents, like the auto-generated finally blocks associated with lock statements and using statements) into the Dispose method, to ensure that stuff gets cleaned up when the iteration is completed or abandoned.

This is not the only implementation strategy we could have pursued. As I said last time, we could have implemented full-blown continuations and built the whole thing out of them; any conceivable control flow can be built out of continuations. But getting continuations right requires a lot of support from the framework, and would be very expensive. We can get by with less, so we do.

Similarly, we could have built this out of “coroutine fibers”. A fiber is like a thread in that it has its own stack, but the code on the fiber itself decides when the fiber is done running. Again, this is a strategy – implement coroutines – that I mentioned last time which we rejected as too costly and difficult to get right in the CLR. Again, we can get by with less, so we do.

But this choice of a feasible, cost-effective implementation strategy then drives restrictions back into the design space; we have to lose some generality in order to make this strategy work.

The first thing that we lose is the ability to have iterator blocks be in methods which take ref or out parameters. In order to preserve the values of local variables and formal parameters across calls to MoveNext, we hoist the locals and parameters into fields of a compiler-generated class.

As I discussed earlier, the CLR designers had an implementation goal of being able to take advantage of the performance characteristics of the system stack. What happens when you store a reference in a field? The reference could be to something on the stack, but the field could live longer than the thing on the stack. The designers of the CLR were faced with four choices:

Build a system which is as fragile and horrid as unmanaged C code, a language in which accidentally storing a reference to something with too-short lifetime is a frequent source of awful crashing bugs and data corruptions.
Disallow (*) taking references to locals; any local taken as a reference must be hoisted to become a field on a garbage-collected reference type. Safe, but slow.
Disallow storing references in fields.
Invent something else that solves this problem.

The CLR designers chose the third option. Which means that our choice of hoisting parameters to fields immediately runs into a problem; we have no safe way of storing references to other variables in a field. The reference could be to something on the stack, but the iterator could live longer than the variable that is on the stack. A confluence of implementation choices has now driven an unfortunate and seemingly arbitrary restriction into the design. We lose a bit of generality in order to gain simplicity of implementation and higher performance in common cases.

Now think about that decision from the perspective of the larger goal of the feature. Remember, the feature is not “make any possible method into an iterator block”. The feature is “make it easier to write code that enumerates the members of a collection”. An “out” or “ref” parameter exists solely to enable a method to mutate an outside variable and therefore pass back information to the caller when the method terminates.

Why would you want to do that in a method whose sole purpose is to produce the sequence of values stored in a collection? What would the meaning of mutations to the variable even be if the iterator block mutates the variable multiple times as the iterator is running? Normally that kind of thing is not visible (modulo goofy multithreading scenarios) until the method is done, but an iterator block returns control to its caller many times before its done.

This seems like a scenario which is uncommon, strange and hard to implement. And furthermore, there are ways to achieve mutation of outer variables during iteration without passing in a ref to a variable; you could pass in an Action delegate. Instead of this illegal program:

IEnumerable<int> Frob(out string s)
{
yield return 1;
s = "goodbye";
}
void Grob()
{
string s = "hello";
foreach(int i in Frob(out s)) …
}

you can always do the equivalent thing with this legal program:

IEnumerable<int> Frob(Action<string> setter)
{
yield return 1;
setter("goodbye");
}
void Grob()
{
string s = "hello";
foreach(int i in Frob(x=>{s=x;})) …
}

Since making the more general feature work given our implementation constraints is (1) hard or impossible, (2) clearly a corner case, not a mainline scenario, and (3) has a possible workaround, it’s a no-brainer to put a restriction on the design and disallow ref and out parameters.

Next time, why no yield in a finally block?

***********

(*) Or, weaker, make it allowed but unverifiable. That is pretty much the same thing from the compiler implementers perspective; we try to never generate unverifiable code unless it is specifically marked “unsafe”.

The void is invariant

Eric Lippert — Mon, 29 Jun 2009 17:30:00 GMT

[UPDATES below]

A while back I described a kind of variance that we’ve supported since C# 2.0. When assigning a method group to a delegate type, such that both the selected method and the delegate target agree that their return type is a reference type, then the conversion is allowed to be covariant. That is, you can say:

Giraffe GetGiraffe() { … }
…
Func<Animal> f = GetGiraffe;

This works logically because anyone who calls f must be able to handle any animal that comes back. The actual method claims to only return animals, and in fact, makes the stronger claim to only return giraffes.

This works out in the CLR because the bits that make up a reference to an instance of Giraffe are exactly the same bits that make up a reference to that Giraffe interpreted as an instance of Animal. We can allow this magical conversion to happen because the CLR guarantees that it will all just work out without going in there and having to futz around with the bits.

This is why this trick only works with reference types. A method that returns, say, a double cannot be converted via a covariant conversion to a delegate type that expects the method to return an object. Somewhere there would have to be code emitted that takes the returned double and boxes it to object; the bits of a double and the bits of a reference to an object boxing a double are completely different.

But why doesn’t this trick work with void types? Here we have a method that returns some sort of success or failure code. Maybe we don’t care what it returns.

static bool DoSomething(bool b)
{
if (b) return DoTheThing();
else return DoTheOtherThing();
}

Action<bool> action = DoSomething;

This doesn’t work. Why not? The caller of the action is not even going to use the returned value, so it doesn’t matter one bit what it is! Shouldn’t “void” be considered a supertype of all possible types for the purposes of covariant return type conversions from method groups to delegate types?

No, and I’ll tell you why.

Consider what happens when you do this:

bool x = DoSomething(true);

We spit out IL that does the following:

(1) put true on the IL stack – the stack gets one deeper
(2) call DoSomething – the argument is removed from the stack and the return value is placed on the stack. Net, the stack stays the same size as before
(3) stuff whatever on top of the stack into local variable x – the stack now returns to its original depth.

Now consider what happens when you do this:

DoSomething(true);

We spit out IL that does the first two steps as before. But we cannot stop there! There is now a bool on the IL stack which needs to be removed. We generate a pop instruction to represent the fact that the returned bool has been discarded.

Now consider what happens when you do this:

action(true);

The compiler believes that action is a void-returning method, so it does not generate a pop instruction. If we allowed you to stuff DoSomething into the action, then we would be allowing you to misalign the IL stack!

But didn’t I say “the stack is an implementation detail?” Yes, but that’s a different stack. The CLI specification describes a “virtual machine” which passes around arguments and returned values on a stack. An implementation of the CLI is required to make something that behaves like the specified machine, but it is not required to do so in any particular manner. It is not required to use the million-bytes-per-thread stack supplied to each thread by the operating system as its implementation of the IL stack; that’s a convenient structure to use, of course, but it’s an implementation detail that it does so.

(As an aside: when we implemented the script engines, we also first specified our own private stack-based virtual machine. When we implemented it, we decided to put the information about “return addresses” – that is, “what code do I run next?” on the system stack, but we put arguments and return values of script functions in a stack-shaped block of memory that we allocated on our own. This made building the JScript garbage collector easier.)

In practice, the jitter uses the system stack for some things and registers for other things. Return values are actually often sent back in a register, not on the stack. But that implementation detail doesn’t help us out when deciding what the conversion rules are; we have to assume that the implementation can do no more than what the CLI specification says. Had the CLI specification said “the returned value of any function is passed back in a ‘virtual register’” rather than having it pushed onto the stack, then we could have made void-returning delegates compatible with functions that returned anything. You can always just ignore the value in the register. But that’s not what the CLI specified, so that’s not what we can do.

[UPDATE]

A number of people have asked in the comments why we do not simply generate a helper method that does what you want. That is, when you say

Action<bool> action = DoSomething;

realize that as

static void DoSomethingHelper(bool b)
{
bool result = DoSomething(b); // result is ignored
}
...
Action<bool> action = DoSomethingHelper;

We could do that. But where would you like the line to be drawn? Should you be able to assign a reference to a method that returns an int to a Func<Nullable<int>>? We could spit a helper method that converts the int to a nullable int. What about Func<double>? We could spit a helper method that converts the int to a double. What about Func<object>? We could spit a helper method that boxes the int, unexpectedly allocating memory off the heap every time you call it. What about a Func<Foo> where there is a user-defined implicit conversion from int to Foo?

We could be spitting arbitrarily complex fixer-upper methods that would seamlessly "do what you meant to say", and we have to stop somewhere. The exact semantics of what we do and do not fix up would have to be designed, specified, implemented, tested, documented, shipped to customers and maintained forever. Those are costs. Plus, every time we add a new conversion rule to the language we add breaking changes. The costs of those breaking changes to our customers have to be factored in.

But more fundamentally, one of the design principles of C# is "if you say something wrong then we tell you rather than trying to guess what you meant". JScript is deliberately a "muddle on through and do the best you can" language; C# is not. If what you want to do is make a delegate to a helper method then you express that intention by going right ahead and making that method.

“Out Of Memory” Does Not Refer to Physical Memory

Eric Lippert — Mon, 08 Jun 2009 17:04:00 GMT

I started programming on x86 machines during a period of large and rapid change in the memory management strategies enabled by the Intel processors. The pain of having to know the difference between “extended memory” and “expanded memory” has faded with time, fortunately, along with my memory of the exact difference.

As a result of that early experience, I am occasionally surprised by the fact that many professional programmers seem to have ideas about memory management that haven’t been true since before the “80286 protected mode” days.

For example, I occasionally get the question “I got an ‘out of memory’ error but I checked and the machine has plenty of RAM, what’s up with that?”

Imagine, thinking that the amount of memory you have in your machine is relevant when you run out of it! How charming! :-)

The problem, I think, with most approaches to describing modern virtual memory management is that they start with assuming the DOS world – that “memory” equals RAM, aka “physical memory”, and that “virtual memory” is just a clever trick to make the physical memory seem bigger. Though historically that is how virtual memory evolved on Windows, and is a reasonable approach, that’s not how I personally conceptualize virtual memory management.

So, a quick sketch of my somewhat backwards conceptualization of virtual memory. But first a caveat. The modern Windows memory management system is far more complex and interesting than this brief sketch, which is intended to give the flavour of virtual memory management systems in general and some mental tools for thinking clearly about what the relationship between storage and addressing is. It is not by any means a tutorial on the real memory manager. (For more details on how it actually works, try this MSDN article.)

I’m going to start by assuming that you understand two concepts that need no additional explanation: the operating system manages processes, and the operating system manages files on disk.

Each process can have as much data storage as it wants. It asks the operating system to create for it a certain amount of data storage, and the operating system does so.

Now, already I am sure that myths and preconceptions are starting to crowd in. Surely the process cannot ask for “as much as it wants”. Surely the 32 bit process can only ask for 2 GB, tops. Or surely the 32 bit process can only ask for as much data storage as there is RAM. Neither of those assumptions are true. The amount of data storage reserved for a process is only limited by the amount of space that the operating system can get on the disk. (*)

This is the key point: the data storage that we call “process memory” is in my opinion best visualized as a massive file on disk.

So, suppose the 32 bit process requires huge amounts of storage, and it asks for storage many times. Perhaps it requires a total of 5 GB of storage. The operating system finds enough disk space for 5GB in files and tells the process that sure, the storage is available. How does the process then write to that storage? The process only has 32 bit pointers, but uniquely identifying every byte in 5GB worth of storage would require at least 33 bits.

Solving that problem is where things start to get a bit tricky.

The 5GB of storage is split up into chunks, typically 4KB each, called “pages”. The operating system gives the process a 4GB “virtual address space” – over a million pages - which can be addressed by a 32 bit pointer. The process then tells the operating system which pages from the 5GB of on-disk storage should be “mapped” into the 32 bit address space. (How? Here’s a page where Raymond Chen gives an example of how to allocate a 4GB chunk and map a portion of it.)

Once the mapping is done then the operating system knows that when the process #98 attempts to use pointer 0x12340000 in its address space, that this corresponds to, say, the byte at the beginning of page #2477, and the operating system knows where that page is stored on disk. When that pointer is read from or written to, the operating system can figure out what byte of the disk storage is referred to, and do the appropriate read or write operation.

An “out of memory” error almost never happens because there’s not enough storage available; as we’ve seen, storage is disk space, and disks are huge these days. Rather, an “out of memory” error happens because the process is unable to find a large enough section of contiguous unused pages in its virtual address space to do the requested mapping.

Half (or, in some cases, a quarter) of the 4GB address space is reserved for the operating system to store it’s process-specific data. Of the remaining “user” half of the address space, significant amounts of it are taken up by the EXE and DLL files that make up the application’s code. Even if there is enough space in total, there might not be an unmapped “hole” in the address space large enough to meet the process’s needs.

The process can deal with this situation by attempting to identify portions of the virtual address space that no longer need to be mapped, “unmap” them, and then map them to some other pages in the storage file. If the 32 bit process is designed to handle massive multi-GB data storages, obviously that’s what its got to do. Typically such programs are doing video processing or some such thing, and can safely and easily re-map big chunks of the address space to some other part of the “memory file”.

But what if it isn’t? What if the process is a much more normal, well-behaved process that just wants a few hundred million bytes of storage? If such a process is just ticking along normally, and it then tries to allocate some massive string, the operating system will almost certainly be able to provide the disk space. But how will the process map the massive string’s pages into address space?

If by chance there isn’t enough contiguous address space then the process will be unable to obtain a pointer to that data, and it is effectively useless. In that case the process issues an “out of memory” error. Which is a misnomer, these days. It really should be an “unable to find enough contiguous address space” error; there’s plenty of memory because memory equals disk space.

I haven’t yet mentioned RAM. RAM can be seen as merely a performance optimization. Accessing data in RAM, where the information is stored in electric fields that propagate at close to the speed of light is much faster than accessing data on disk, where information is stored in enormous, heavy ferrous metal molecules that move at close to the speed of my Miata. (**)

The operating system keeps track of what pages of storage from which processes are being accessed most frequently, and makes a copy of them in RAM, to get the speed increase. When a process accesses a pointer corresponding to a page that is not currently cached in RAM, the operating system does a “page fault”, goes out to the disk, and makes a copy of the page from disk to RAM, making the reasonable assumption that it’s about to be accessed again some time soon.

The operating system is also very smart about sharing read-only resources. If two processes both load the same page of code from the same DLL, then the operating system can share the RAM cache between the two processes. Since the code is presumably not going to be changed by either process, it's perfectly sensible to save the duplicate page of RAM by sharing it.

But even with clever sharing, eventually this caching system is going to run out of RAM. When that happens, the operating system makes a guess about which pages are least likely to be accessed again soon, writes them out to disk if they’ve changed, and frees up that RAM to read in something that is more likely to be accessed again soon.

When the operating system guesses incorrectly, or, more likely, when there simply is not enough RAM to store all the frequently-accessed pages in all the running processes, then the machine starts “thrashing”. The operating system spends all of its time writing and reading the expensive disk storage, the disk runs constantly, and you don’t get any work done.

This also means that "running out of RAM" seldom(***) results in an “out of memory” error. Instead of an error, it results in bad performance because the full cost of the fact that storage is actually on disk suddenly becomes relevant.

Another way of looking at this is that the total amount of virtual memory your program consumes is really not hugely relevant to its performance. What is relevant is not the total amount of virtual memory consumed, but rather, (1) how much of that memory is not shared with other processes, (2) how big the "working set" of commonly-used pages is, and (3) whether the working sets of all active processes are larger than available RAM.

By now it should be clear why “out of memory” errors usually have nothing to do with how much physical memory you have, or how even how much storage is available. It’s almost always about the address space, which on 32 bit Windows is relatively small and easily fragmented.

And of course, many of these problems effectively go away on 64 bit Windows, where the address space is billions of times larger and therefore much harder to fragment. (The problem of thrashing of course still occurs if physical memory is smaller than total working set, no matter how big the address space gets.)

This way of conceptualizing virtual memory is completely backwards from how it is usually conceived. Usually it’s conceived as storage being a chunk of physical memory, and that the contents of physical memory are swapped out to disk when physical memory gets too full. But I much prefer to think of storage as being a chunk of disk storage, and physical memory being a smart caching mechanism that makes the disk look faster. Maybe I’m crazy, but that helps me understand it better.

*************

(*) OK, I lied. 32 bit Windows limits the total amount of process storage on disk to 16 TB, and 64 bit Windows limits it to 256 TB. But there is no reason why a single process could not allocate multiple GB of that if there’s enough disk space.

(**) Numerous electrical engineers pointed out to me that of course the individual electrons do not move fast at all; it's the field that moves so fast. I've updated the text; I hope you're all happy with it now.

(***) It is possible in some virtual memory systems to mark a page as “the performance of this page is so crucial that it must always remain in RAM”. If there are more such pages than there are pages of RAM available, then you could get an “out of memory” error from not having enough RAM. But this is a much more rare occurrence than running out of address space.

The Stack Is An Implementation Detail, Part Two

Eric Lippert — Mon, 04 May 2009 20:57:27 GMT

A number of people have asked me, in the wake of my earlier posting about value types being on the stack, why it is that value types go on the stack but reference types do not.

The short answer is “because they can”. And since the stack is cheap, we do put them on the stack when possible.

The long answer is… long.

I’ll need to give a high-level description of the memory management strategies that we call “stack” and “heap”. Heap first.

The CLRs garbage-collected heap is a marvel of engineering with a huge amount of detail; this sketch is not actually how it works, but it’s a good enough approximation to get the idea.

The idea is that there is a large block of memory reserved for instances of reference types. This block of memory can have “holes” – some of the memory is associated with “live” objects, and some of the memory is free for use by newly created objects. Ideally though we want to have all the allocated memory bunched together and a large section of “free” memory at the top.

If we’re in that situation when new memory is allocated then the “high water mark” is bumped up, eating up some of the previously “free” portion of the block. The newly-reserved memory is then usable for the reference type instance that has just been allocated. That is extremely cheap; just a single pointer move, plus zeroing out the newly reserved memory if necessary.

If we have holes then we can maintain a “free list” – a linked list of holes. We can then search the free list for a hole of appropriate size and fill it in. This is a bit more expensive since there is a list search. We want to avoid this suboptimal situation if possible.

When a garbage collection is performed there are three phases: mark, sweep and compact. In the “mark” phase, we assume that everything in the heap is “dead”. The CLR knows what objects were “guaranteed alive” when the collection started, so those guys are marked as alive. Everything they refer to is marked as alive, and so on, until the transitive closure of live objects are all marked. In the “sweep” phase, all the dead objects are turned into holes. In the “compact” phase, the block is reorganized so that it is one contiguous block of live memory, free of holes.

This sketch is complicated by the fact that there are actually three such arenas; the CLR collector is generational. Objects start off in the “short lived” heap. If they survive they eventually move to the “medium lived” heap, and if they survive there long enough, they move to the “long lived” heap. The GC runs very often on the short lived heap and very seldom on the long lived heap; the idea is that we do not want to have the expense of constantly re-checking a long-lived object to see if it is still alive. But we also want short-lived objects to be reclaimed swiftly.

The GC has a huge amount of carefully tuned policy that ensures high performance; it attempts to balance the memory and time costs of having a Swiss-cheesed heap against the high cost of the compaction phase. Extremely large objects are stored in a special heap that has different compaction policy. And so on. I don’t know all the details, and fortunately, I don’t need to. (And of course, I have left out lots of additional complexity that is not germane to this article – pinning and finalization and weak refs and so on.)

Now compare this to the stack. The stack is like the heap in that it is a big block of memory with a “high water mark”. But what makes it a “stack” is that the memory on the bottom of the stack always lives longer than the memory on the top of the stack; the stack is strictly ordered. The objects that are going to die first are on the top, the objects that are going to die last are on the bottom. And with that guarantee, we know that the stack will never have holes, and therefore will not need compacting. We know that the stack memory will always be “freed” from the top, and therefore do not need a free list. We know that anything low-down on the stack is guaranteed alive, and so we do not need to mark or sweep.

On a stack, the allocation is just a single pointer move – the same as the best (and typical) case on the heap. But because of all those guarantees, the deallocation is also a single pointer move! And that is where the huge time performance savings is. A lot of people seem to think that heap allocation is expensive and stack allocation is cheap. They are actually about the same, typically. It’s the deallocation costs – the marking and sweeping and compacting and moving memory from generation to generation – that are massive for heap memory compared to stack memory.

Clearly it is better to use a stack than a GC heap if you can. When can you? Only when you can guarantee that all the necessary conditional that make a stack work are actually achieved. Local variables and formal parameters of value type are the sweet spot that achieve that. The locals of frames on the bottom of the stack clearly live longer than the locals on the frames of the top of the stack. Locals of value type are copied by value, not by reference, so the local is the only thing that references the memory; there is no need to track who is referencing a particular value to determine its lifetime. And the only way to take a ref to a local is to pass it as a ref or out parameter, which just passes the ref on up the stack. The local is going to be alive anyway, so the fact that there is a ref to it “higher up” the call stack doesn’t change its lifetime.

aside
{

A few asides:

This explains why you cannot make a “ref int” field. If you could then you could store a ref to the value of a short-lived local inside a long-lived object. Were that legal then using the stack as a memory management technique would no longer be a viable optimization; value types would be just another kind of reference type and would have to be garbage collected.

Anonymous function closures and iterator block closures are implemented behind-the-scenes by turning locals and formal parameters into fields. So now you know why it is illegal to capture a ref or out formal parameter in an anonymous function or iterator block; doing so would not be a legal field.

Of course we do not want to have ugly and bizarre rules in the language like “you can close over any local or value parameter but not a ref or out parameter”. But because we want to be able to get the optimization of putting value types on the stack, we have chosen to put this odd restriction into the language. Design is, as always, the art of finding compromises.

Finally, the CLR does allow “ref return types”; you could in theory have a method “ref int M() { … }” that returned a reference to an integer variable. If for some bizarre reason we ever decided to allow that in C#, we’d have to fix up the compiler and verifier so that they ensured that it was only possible to return refs to variables that were known to be on the heap, or known to be “lower down” on the stack than the callee.

}

So there you go. Local variables of value type go on the stack because they can. They can because (1) “normal” locals have strict ordering of lifetime, and (2) value types are always copied by value and (3) it is illegal to store a reference to a local anywhere that could live longer than the local. By contrast, reference types have a lifetime based on the number of living references, are copied by reference, and those references can be stored anywhere. The additional flexibility that reference types give you comes at the cost of a much more complex and expensive garbage collection strategy.

But again, all of this is an implementation detail. Using the stack for locals of value type is just an optimization that the CLR performs on your behalf. The relevant feature of value types is that they have the semantics of being copied by value, not that sometimes their deallocation can be optimized by the runtime.

The Stack Is An Implementation Detail, Part One

Eric Lippert — Mon, 27 Apr 2009 16:15:00 GMT

" style="BORDER-TOP-WIDTH: 0px; DISPLAY: inline; BORDER-LEFT-WIDTH: 0px; BORDER-BOTTOM-WIDTH: 0px; MARGIN-LEFT: 0px; MARGIN-RIGHT: 0px; BORDER-RIGHT-WIDTH: 0px" height=240 alt="Stack" src="http://blogs.msdn.com/blogfiles/ericlippert/WindowsLiveWriter/TheStackIsAnImplementationDetail_C978/Stack_thumb_1.jpg" width=156 align=left border=0 mce_src="http://blogs.msdn.com/blogfiles/ericlippert/WindowsLiveWriter/TheStackIsAnImplementationDetail_C978/Stack_thumb_1.jpg">I blogged a while back about how “references” are often described as “addresses” when describing the semantics of the C# memory model. Though that’s arguably correct, it’s also arguably an implementation detail rather than an important eternal truth. Another memory-model implementation detail I often see presented as a fact is “value types are allocated on the stack”. I often see it because of course, that’s what our documentation says.

Almost every article I see that describes the difference between value types and reference types explains in (frequently incorrect) detail about what “the stack” is and how the major difference between value types and reference types is that value types go on the stack. I’m sure you can find dozens of examples by searching the web.

I find this characterization of a value type based on its implementation details rather than its observable characteristics to be both confusing and unfortunate. Surely the most relevant fact about value types is not the implementation detail of how they are allocated, but rather the by-design semantic meaning of “value type”, namely that they are always copied “by value”. If the relevant thing was their allocation details then we’d have called them “heap types” and “stack types”. But that’s not relevant most of the time. Most of the time the relevant thing is their copying and identity semantics.

I regret that the documentation does not focus on what is most relevant; by focusing on a largely irrelevant implementation detail, we enlarge the importance of that implementation detail and obscure the importance of what makes a value type semantically useful. I dearly wish that all those articles explaining what “the stack” is would instead spend time explaining what exactly “copied by value” means and how misunderstanding or misusing “copy by value” can cause bugs.

Of course, the simplistic statement I described is not even true. As the MSDN documentation correctly notes, value types are allocated on the stack sometimes. For example, the memory for an integer field in a class type is part of the class instance’s memory, which is allocated on the heap. A local variable is hoisted to be implemented as a field of a hidden class if the local is an outer variable used by an anonymous method(*) so again, the storage associated with that local variable will be on the heap if it is of value type.

But more generally, again we have an explanation that doesn’t actually explain anything. Leaving performance considerations aside, what possible difference does it make to the developer whether the CLR’s jitter happens to allocate memory for a particular local variable by adding some integer to the pointer that we call “the stack pointer” or adding the same integer to the pointer that we call “the top of the GC heap”? As long as the implementation maintains the semantics guaranteed by the specification, it can choose any strategy it likes for generating efficient code.

Heck, there’s no requirement that the operating system that the CLI is implemented on top of provide a per-thread one-meg array called “the stack”. That Windows typically does so, and that this one-meg array is an efficient place to store small amounts of short-lived data is great, but it’s not a requirement that an operating system provide such a structure, or that the jitter use it. The jitter could choose to put every local “on the heap” and live with the performance cost of doing so, as long as the value type semantics were maintained.

Even worse though is the frequently-seen characterization that value types are “small and fast” and reference types are “big and slow”. Indeed, value types that can be jitted to code that allocates off the stack are extremely fast to both allocate and deallocate. Large structures heap-allocated structures like arrays of value type are also pretty fast, particularly if you need them initialized to the default state of the value type. And there is some memory overhead to ref types. And there are some high-profile cases where value types give a big perf win. But in the vast majority of programs out there, local variable allocations and deallocations are not going to be the performance bottleneck.

Making the nano-optimization of making a type that really should be a ref type into a value type for a few nanoseconds of perf gain is probably not worth it. I would only be making that choice if profiling data showed that there was a large, real-world-customer-impacting performance problem directly mitigated by using value types. Absent such data, I’d always make the choice of value type vs reference type based on whether the type is semantically representing a value or semantically a reference to something.

UPDATE: Part two is here

*******

(*) Or in an iterator block.

References are not addresses

Eric Lippert — Tue, 17 Feb 2009 21:10:00 GMT

[NOTE: Based on some insightful comments I have updated this article to describe more clearly the relationships between references, pointers and addresses. Thanks to those who commented.]

I review a fair number of C# books; in all of them of course the author attempts to explain the difference between reference types and value types. Unfortunately, most of them do so by saying something like "a variable of reference type stores the address of the object". I always object to this. The last time this happened the author asked me for a more detailed explanation of why I always object, which I shall share with you now:

We have the abstract concept of "a reference". If I were to write about "Beethoven's Ninth Symphony", those two-dozen characters are not a 90-minute long symphonic masterwork with a large choral section. They're a reference to that thing, not the thing itself. And this reference itself contains references -- the word "Beethoven" is not a long-dead famously deaf Romantic Period composer, but it is a reference to one.

Similarly in programming languages we have the concept of "a reference" distinct from "the referent".

The inventor of the C programming language, oddly enough, chose to not have the concept of references at all. Rather, Ritchie chose to have "pointers" be first-class entities in the language. A pointer in C is like a reference in that it refers to some data by tracking its location, but there are more smarts in a pointer; you can perform arithmetic on a pointer as if it were a number, you can take the difference between two pointers that are both in the interior of the same array and get a sensible result, and so on.

Pointers are strictly "more powerful" than references; anything you can do with references you can do with pointers, but not vice versa. I imagine that's why there are no references in C -- it's a deliberately austere and powerful language.

The down side of pointers-instead-of-references is that pointers are hard for many novices to understand, and make it very very very easy to shoot yourself in the foot.

Pointers are typically implemented as addresses. An address is a number which is an offset into the "array of bytes" that is the entire virtual address space of the process (or, sometimes, an offset into some well-known portion of that address space -- I'm thinking of "near" vs. "far" pointers in win16 programming. But for the purposes of this article let's assume that an address is a byte offset into the whole address space.) Since addresses are just numbers you can easily perform pointer arithmetic with them.

Now consider C#, a language which has both references and pointers. There are some things you can only do with pointers, and we want to have a language that allows you to do those things (under carefully controlled conditions that call out that you are doing something that possibly breaks type safety, hence "unsafe".) But we also do not want to force anyone to have to understand pointers in order to do programming with references.

We also want to avoid some of the optimization nightmares that languages with pointers have. Languages with heavy use of pointers have a hard time doing garbage collection, optimizations, and so on, because it is infeasible to guarantee that no one has an interior pointer to an object, and therefore the object must remain alive and immobile.

For all these reasons we do not describe references as addresses in the specification. The spec just says that a variable of reference type "stores a reference" to an object, and leaves it completely vague as to how that might be implemented. Similarly, a pointer variable stores "the address" of an object, which again, is left pretty vague. Nowhere do we say that references are the same as addresses.

So, in C# a reference is some vague thing that lets you reference an object. You cannot do anything with a reference except dereference it, and compare it with another reference for equality. And in C# a pointer is identified as an address.

By contrast with a reference, you can do much more with a pointer that contains an address. Addresses can be manipulated mathematically; you can subtract one from another, you can add integers to them, and so on. Their legal operations indicate that they are "fancy numbers" that index into the "array" that is the virtual address space of the process.

Now, behind the scenes, the CLR actually does implement managed object references as addresses to objects owned by the garbage collector, but that is an implementation detail. There's no reason why it has to do that other than efficiency and flexibility. C# references could be implemented by opaque handles that are meaningful only to the garbage collector, which, frankly, is how I prefer to think of them. That the "handle" happens to actually be an address at runtime is an implementation detail which I should neither know about nor rely upon. (Which is the whole point of encapsulation; the client doesn't have to know.)

I therefore have three reasons why authors should not explain that "references are addresses".

1) It's close to a lie. References cannot be treated as addresses by the user, and in fact, they do not necessarily contain an address in the implementation. (Though our implementation happens to do so.)

2) It's an explanation that explains nothing to novice programmers. Novice programmers probably do not know that an "address" is an offset into the array of bytes that is all process memory. To understand what an "address" is with any kind of depth, the novice programmer already has to understand pointer types and addresses -- basically, they have to understand the memory model of many implementations of C. This is one of those "it's clear only if it's already known" situations that are so common in books for beginners.

3) If these novices eventually learn about pointer types in C#, their confused understanding of references will probably make it harder, not easier, to understand how pointers work in C#. The novice could sensibly reason "If a reference is an address and a pointer is an address, then I should be able to cast any reference to a pointer in unsafe code, right?" But you cannot.

If you think of a reference is actually being an opaque GC handle then it becomes clear that to find the address associated with the handle you have to somehow "fix" the object. You have to tell the GC "until further notice, the object with this handle must not be moved in memory, because someone might have an interior pointer to it". (There are various ways to do that which are beyond the scope of this screed.)

Basically what I'm getting at here is that an understanding of the meaning of "addresses" in any language requires a moderately deep understanding of the memory model of that language. If an author does not provide an explanation of the memory model of either C or C#, then explaining references in terms of addresses becomes an exercise in question begging. It raises more questions than it answers.

This is one of those situations where the author has the hard call of deciding whether an inaccurate oversimplification serves the larger pedagogic goal better than an accurate digression or a vague hand-wave.

In the counterfactual world where I am writing a beginner C# book, I would personally opt for the vague hand-wave. If I said anything at all I would say something like "a reference is actually implemented as a small chunk of data which contains information used by the CLR to determine precisely which object is being referred to by the reference". That's both vague and accurate without implying more than is wise.