Maoni's WebLog : General

So, what’s new in the CLR 4.0 GC?

maoni — Thu, 20 Nov 2008 04:02:00 GMT

PDC 2008 happened not long ago so I get to write another “what’s new in GC” blog entry. For quite a while now I’ve been working on a new concurrent GC that replaces the existing one. And this new concurrent GC is called “background GC”.

First of all let me apologize for having not written anything for so long. It’s been quite busy working on the new GC and other things.

Let me refresh your memory on concurrent GC. Concurrent GC has existed since CLR V1.0. For a blocking GC, ie, a non concurrent GC we always suspend managed threads, do the GC work then resume managed threads. Concurrent GC, on the other hand, runs concurrently with the managed threads to the following extend:

§ It allows you to allocate while a concurrent GC is in progress.

However you can only allocate so much – for small objects you can allocate at most up to end of the ephemeral segment. Remember if we don’t do an ephemeral GC, the total space occupied by ephemeral generations can be as big as a full segment allows so as soon as you reached the end of the segment you will need to wait for the concurrent GC to finish so managed threads that need to make small object allocations are suspended.

§ It still needs to stop managed threads a couple of times during a concurrent GC.

During a concurrent GC we need to suspend managed threads twice to do some phases of the GC. These phases could possibly take a while to finish.

We only do concurrent GCs for full GCs. A full GC can be either a concurrent GC or a blocking GC. Ephemeral GCs (ie, gen0 or gen1 GCs) are always blocking.

Concurrent GC is only available for workstation GC. In server GC we always do blocking GCs for any GCs.

Concurrent GC is done on a dedicated GC thread. This thread times out if no concurrent GC has happened for a while and gets recreated next time we need to do concurrent GC.

When the program activity (including making allocations and modifying references) is not really high and the heap is not very large concurrent GC works well – the latency caused by the GC is reasonable. But as people start writing larger applications with larger heaps that handle more stressful situations, the latency can be unacceptable.

Background GC is an evolution to concurrent GC. The significance of background GC is we can do ephemeral GCs while a background GC is in progress if needed. As with concurrent GC, background GC is also only applicable to full GCs and ephemeral GCs are always done as blocking GCs, and a background GC is also done on its dediated GC thread. The ephemeral GCs done while a background GC is in progress are called foreground GCs.

So when a background GC is in progress and you’ve allocated enough in gen0, we will trigger a gen0 GC (which may stay as a gen0 GC or get elevated as a gen1 GC depending on GC’s internal tuning). The background GC thread will check at frequent safe points (ie, when we can allow a foreground GC to happen) and see if there’s a request for a foreground GC. If so it will suspend itself and a foreground GC can happen. After this foreground GC is finished, the background GC thread and the user threads can resume their work.

Not only does this allow us to get rid of dead objects in young generations, it also lifts the restriction of having to stay in the ephemeral segment – if we need to expand the heap while a background GC is going on, we can do so in a gen1 GC.

We also made some performance improvement in background GC which does better at doing more things concurrently so the time we need to suspend managed threads is also shorter.

We are not offering background GC for server GC in V4.0. It’s under consideration – we recognize how important it is for server applications (which usually have much larger heaps than client apps) to benefit from smaller latency but the work did not fit in our V4.0 timeframe. For now for server applications, I would recommend you to look at the full GC notification feature we added in .NET 3.5 SP1. It’s explained here: http://msdn.microsoft.com/en-us/library/cc713687.aspx. Basically you register to get notified when a full GC is approaching and when it’s finished. This allows you to do software load balancing between different server instances – when a full GC is about to happen in one of the server instances, you can redirect new requests to other instances.

He’s live… he’s live not… he’s live…

maoni — Wed, 21 Mar 2007 08:17:00 GMT

I was making some code changes today and thought this was interesting to share. As you know, the WeakReference class has a getter and a setter method to get and set the Target which is what the weakref points to. See Using GC Efficiently – Part 3 for more details on WeakReference.

Note that the code below is only for illustration purposes – it is not necessarily what’s in the production code.

So let’s say the code used to look like this in the WeakReference class:

internal IntPtr m_handle;

public Object Target

{

get

{

IntPtr h = m_handle;

if (IntPtr.Zero == h)

return false;

Object o = GCHandle.InternalGet(h);

h = Thread.VolatileRead(ref m_handle);

GC.KeepAlive (this);

return (h == IntPtr.Zero) ? null : o;

}

...

}

m_handle is the weak GCHandle that we create to implemente the weakref funtionality. It’s a weak handle that points to the object that you want your weakref object to point to.

The problem is Thread.VolatileRead kind of a heavy weight thing - not very performant (there was a reason why we used this API in the first place…not necessarily a good one but it’s what we ended up with).

First of all let’s take a look at the old code. Notice that we have a GC.KeepAlive in there. Why would we do that? I mean if during the call of Thread.VolatileRead, the weakref object is dead and its finalizer sets m_handle to 0, we’d just read 0. That’s fine right?

But imagine this code:

Object o = new Object();

while (true)

{

WeakReference wr = new WeakReference (o);

if (wr.Target == null)

{

...

}

o.GetHashCode();

We know that o is live during the while loop which means it would be really nice if wr.Target is null (some would consider it a bug if wr.Target was ever null in this case). If we didn’t have the KeepAlive, object wr could be considered dead as soon as its address is passed in to the Thread.VolatileRead call as the argument. So then h could be 0 and we’d return null. So we want to make sure that if the object the weakref points to is guaranteed to be live during the getter call, you will always get back that object instead of null.

Wouldn’t that achieve the same effect since m_handle is an instance data member and if it’s used, the *this* object should be kept alive where the last statement is?

Well, actually since m_handle is not a volatile, jit could generate some code that stores the value of m_handle in a register so it will not need to read the value of m_handle again when it’s at that last statement.

Another interesting thing about this code is that it does the IntPtr.Zero check after it first read the value of m_handle. But how can m_handle be 0? m_handle is only set to 0 in the WeakReference class’s finalizer code. If we are already KeepAlive-ing the weakref object, it means the object should be live therefore by definition the finalizer should have not been run, right?

Well, unfortunately there is a case where m_handle can be 0 while we are in the getter which is when the getter is called in an object’s finalizer. Imagine you have this object hireachy:

public class ObjectA

{

WeakReference wr;

public ObjectA(WeakReference wr0)

{

...

wr = wr0;

}

...

public ~ObjectA()

{

...

if (wr.Target == null)

{

...

}

...

}

ObjectA a = new ObjectA();

When a is not used anymore, at some point both a’s and wr0’s finalizer (assuming a is the only object that contains a reference to wr0) will be put on the finalize queue.

Now if a’s finalizer gets to run first, we are fine ‘cause when we are in wr0’s getter, m_handle is still valid. But if wr0’s finalizer gets to run first, then when a’s finalizer is run, m_handle is already set to 0. Of course as we’ve been saying that a finalizer should do no more than releasing native resources, this shouldn’t be a common scenario.

So, the idea is to change m_handle to volatile and eliminate the need for calling Thread.VolatileRead. The resulting code looks like this:

internal volatile IntPtr m_handle;

public Object Target

{

get

{

IntPtr h = m_handle;

if (IntPtr.Zero == h)

return null;

Object o = GCHandle.InternalGet(h);

return (m_handle == IntPtr.Zero) ? null : o;

}

...

}

Notice that KeepAlive is gone because we are reading a volatile value which means the weakref object will be kept alive though out the getter. We still need to check if h is IntPtr.Zero at the beginning because we are still subject to be called from another object’s finalizer.

Some people don’t use volatile’s in fear of losing performance ‘cause volatile’s can’t be optimzed by the compiler. In reality though, if you read the volatile into a local when you need to access it frequently and that you are fine with the cached value, no reason to be afraid of using volatile’s.

Check out the CLR Lead Architect's New Blog

maoni — Wed, 22 Nov 2006 23:38:00 GMT

Many people know Patrick Dussud by his outstanding work on Garbage Collection. But did you know he was one of the founders of the CLR? In his intro blog entry he talks about how the CLR came to life. I am sure it will be a great read for those of you who are curious about it.

New MSDN Article - Investigating Memory Issues

maoni — Mon, 23 Oct 2006 08:35:00 GMT

We have a new MSDN article out in the November issue that talks about investigating managed memory issues.

http://msdn.microsoft.com/msdnmag/issues/06/11/CLRInsideOut/default.aspx?loc=en

Take a look and let me know what you think.

Oh, and it's also in 6 other languages (German, Spanish, French, Russian, Portuguese and Chinese) for readers that prefer one of those languages.

I Am a Happy Janitor – Part 1: Finding garbage

maoni — Sat, 19 Aug 2006 02:14:00 GMT

Indeed what I do is very much like the job of the janitors – like the ones who clean your building, or janitors you see at a food court, or yourself when you are taking care of garbage at your house. Doubtful you say? Let me prove it to you.

Finding garbage

What do you do when you are having lunch with your friends at a food court and suddenly you need to go somewhere for a short while in the middle of it (like, you asked for hot sauce but were given not-so-hot-sauce and you went back to the cashier to tell him how it’s so not up to your standard)? Then the nice janitor guy comes along and he sees your plate and tries to take it (you were so hungry you already ate most of your food before your remembered to use the sauce…). What happens now?

One of your friends says to the janitor guy, “my friend is not finished yet.”. The janitor then goes away without taking your plate.

Same principle for the GC, the difference is GC’s friends tell it what’s not garbage, ie, what’s live (seriously..who wants garbage…). GC’s friends include:

· JIT

· EE stack walker

· Handle table

· Finalize queue

What happens is this:

GC says: I need to start a collection on gen X…My super friends please report to me what is live now for gen X.

JIT says: Roger that. I will take care of the live roots on managed stack frames. When I jitted methods I had already built up a table that tracked which locals were alive at which point so I will report the ones that are currently live to you. I am quite aggressive at determining if something should be dead or not in retail builds – locals could be treated as dead as soon as I determine it’s not in use anymore (not guaranteed). However in debuggable builds I am much more lenient – I will let all locals live till the end of the scope to make debugging more convenient.

EE stack walker says: I can take care of the locals on the unmanaged frames when managed frames call into the EE (for lock implementation and whatnot).

Finalize queue says: Indeed, the almighty me brought back some dead objects back to life so I am telling you about these objects.

Handle table says: Reporting handles in use in gen X and all its youngest generations now.

GC waits patiently while the super friends are reporting what’s live.

The only time GC needs to participate in finding what’s live is when GC needs to find out if there are objects in older generations that are referring to objects in younger generations. For example if GC is collecting gen 1, it will need to go find out if there are any gen2 objects that are referring to gen1 and gen0.

Suspending and resuming threads for GC

maoni — Thu, 08 Jun 2006 04:02:00 GMT

First of all, suspension and resumption of threads is not really part of the GC. GC calls functions to do the suspension and the resumption as a service provided in the CLR. Other components in the CLR also use this service such as the debugger implementation. But it’s true that suspending and resuming because of the GC is the most significant usage of this service.

Secondly, how threads are suspended and resumed works the same way in Workstation GC (with or without Concurrent GC) and Server GC. Workstation GC with Concurrent GC just suspends and resumes threads more times than the other two cases.

Now let's talk about suspension. A managed thread could be running either managed code or native code at the time. If it’s running native code which can either be from

· the EE (Execution Engine) of the CLR – for example, EE implements some helper function for your managed call which is on the stack at the time. In this case the EE code tells the suspension code how it should be suspended. A common scenario is the EE code will wait for GC to finish so the suspension code doesn’t need to suspend it (it already suspends itself). Or

· the native code is not from the EE, which means it’s running some native code via Interop, we don’t need to suspend this thread because it shouldn’t be doing anything with managed objects that can are not pinned. When this thread needs to return to managed code, however, it will need to check if a GC is in progress and if so it needs to wait for GC to finish.

For the managed threads that are running managed code it can be in either

§ a mode where it can be stopped at any time (because the JIT recorded enough info for each instruction) in which case it can be suspended right there. We then redirect it to some function that will make this thread wait for GC to complete. Or

· a mode where it can’t be stopped at any time, in which case the thread’s return address will be hijacked, meaning we replace the return address by a stub which will suspend this thread. What this stub does is it will suspend this thread to wait for GC to finish.

In both case the function that was used to suspend the thread knows how to resume them correctly when GC finishes.

Resumption works much simpler. When a GC is finished, all the threads that were suspended will wake up (they were waiting on an event for GC to finish) to resume execution.

Large Object Heap

maoni — Wed, 19 Apr 2006 07:51:00 GMT

LOH (Large Object Heap) contains objects that are 85,000 bytes or bigger (there’s also some objects that are less than 85,000 bytes that are allocated on the LOH by the runtime itself but usually they are very small and we’ll ignore them for this discussion).

The way LOH is implemented changed dramatically from 1.0 to 1.1. In 1.0 we used a malloc type of allocator for allocating large objects; in 1.1 and beyond we use the same allocator for both large and small objects: we acquire memory from the OS by heap segments and commit on a segment as needed. There’s very little difference between 1.1 and 2.0 for LOH.

LOH is not compacted – however if you want a large object to be pinned you should make sure to pin it because whether LOH is compacted or not is an implementation detail that could be changed in the future. Free blocks between live large objects are threaded into a free list which will be used to satisfy large object allocation requests. Note that this is a real advantage for managed heap – we are able to be very efficient to manage fragmentation because we can coalesce adjacent free objects into a big free block. If a free block is too large we call MEM_RESET on it to tell the OS to not bother backing it up. So far I haven’t heard of any fragmentation problems from production code (of course you write a program that specifically fragments the LOH) – I think one reason is we are doing a good job controlling fragmentation; the other reason is people usually don’t churn LOH too much – one typical pattern I’ve seen people use LOH is to allocate some large arrays that hold on to small objects.

When I talked about weak references in Using GC Efficiently – Part 3, I mentioned a performance implication of using weak refs which is when you use them to refer to really small objects that are comparable in size. Make sure that you are fine with the ratio of the space that weak ref objects take up over the size of objects that they refer to. There’s no difference between using a weak reference to refer to a large and a small object aside from the fact that by definition this ratio is naturally smaller for large objects because of their sizes.

In this comment, it was asked why “explicitly setting a reference to null would prevent the LOH from growing faster than the collection rate”. Actually I think what likely happened was the large objects were held live by something therefore GC was not reclaiming the memory. By setting these objects to null just lets GC know that those objects are garbage now and can be reclaimed. You can use either CLRProfiler or the !SOS.gcroot command to verify this.

So, what’s new in the CLR 2.0 GC?

maoni — Tue, 04 Oct 2005 09:30:00 GMT

Certainly that’s one of the most frequently asked questions I get (at the PDC too!). So since PDC already happened I can tell the rest of you about the new stuff happened in GC in CLR 2.0. The slides can be downloaded here. And I will be referring to some of the slides. I must apologize for your having to click on the link to see the slide each time I refer to one since I don’t have a separated site where I can store pictures.

Most of what I am going to talk about was covered in my PDC GC talk. BTW, just a few words about the 2005 PDC (skip this paragraph if you are interested in only technical stuff :-)). I was very pleased with the way it went. I got to talk to lots of customers who told me how awesome our GC was which certainly made me feel very good! There were a couple of customers who did tell us about the problems they had with our GC. One problem we already addressed in Whidbey and the other one was something we were perfectly aware of and had put on our todo list. So no surprises there. I was happy to see the number of people at my talk considering it was the latest possible and lots of people had already left the conference before that. So all in all it was great.

Some terminology before we start:

Ephemeral generations – gen0 and gen1.

Ephemeral segment – the segment that gen0 and gen1 live in (they always live in one segment) and there can only be one ephemeral segment for each heap. So for server GC, if we have 4 heaps we have 4 ephemeral segments (refer to my Using GC Efficiently – Part 2 for explaination for different flavors of GCs).

Small object heap – you know LOH (if you don’t, it’s covered in Using GC Efficiently – Part 1), so the rest of the managed heap is called small object heap :-)

Fragmentation control

If you ask me what was the most significant improvement in the CLR 2.0 GC from the 1.1 GC, I’d have to say it’s the work we did in reducing fragmentation caused by pinning.

One of the biggest problems we faced for managed applications was wasted free space on the managed heap. And sometimes you get OOM while there’s plenty of free space on the managed heap. So reducing fragmentation, in other words, having less space wasted on the managed heap, was crucial. It was observed often in applications that perform IO which needs to pin the buffers for the native APIs to read or write to them. As such, we’ve done a lot of work in CLR 2.0 to help solve this

problem.

Slide 25 demonstrates the fragmentation problem caused by pinning. Let’s say we have an application where each request performs some type of socket or file IO. Before GC number X we have a heap that has one segment, which is our ephemeral segment, has bunch of stuff allocated in gen0, including some pinned objects. Now a GC is triggered. After the collection what survived from gen1 was promoted to gen2. For a simplistic view let’s just say nothing from gen0 except the pins survived. In reality what survived is the pinned objects and the other data associated with the requests that pinned those objects for this scenario. So after the collection, gen1 just has the pins ‘cause that’s all that survived from gen0. Gen0 then starts right after the 2^nd pin.

You can imagine that if this process keeps repeating itself, and let’s say that the pins survived N more GCs, after these GCs we may have a situation where we have expanded the heap and the old ephemeral segment now is a gen2 segment and we acquired a new ephemeral segment. Again we start the new gen0 after the pins in the ephemeral segment.

This is a really bad situation because we now have lots of wasted space on our heap. Both our gen2 and gen1 look quite empty. Obviously the less wasted space we can make it, the better.

The fragmentation reduction work can be categorized into 2 things: one is called demotion which is to prevent fragmentation from getting into higher generation as much as we can; the other one is reusing existing gen2 segments so we can reuse the free space that has already made into gen2. Let’s talk about them in more details.

Demotion, as the opposite of promotion, means the object doesn’t end up in a generation that it’s supposed to be in. Imagine if after the compaction, there’s plenty of space between some pinned objects at the end of the ephemeral segment, it’s more productive to leave them in gen0 instead of promoting them to gen1 because when allocation requests come in the free space before the pins can be used to satisfy allocations. This is demonstrated with slide 26.

Segment reuse is relatively straightforward. It’s to take advantage of the existing gen2 segments that have a lot free space but not yet empty (because if they were empty they would have been deleted). Slide 27 demonstrates the problem without segment reuse. So before GC we are running out of space in the ephemeral segment and we need to expand the heap. We have 2 pinned objects. And I didn’t mark their generations because it’s not significant to indicate their generations - they can not be moved so they will stay in this segment and become gen2 objects by definition. There might be pinning on the gen2 segments as well but since that doesn’t affect illustrating the point I preferred to keep the picture simple and left them out.

So after GC we allocate a new segment. The old gen1 in the old ephemeral segment was promoted to gen2 and the old gen0 now lives in the new ephemeral segment as the new gen1 and we will start allocating after gen1 on this segment. The pinned objects are left in the old ephemeral seg since they can not be moved. They are part of gen2 now because they live in a gen2 segment.

Slide 28 demonstrates segment reuse. Same picture for before GC. During GC we found that segment 3 is empty enough so instead of allocating a new segment we decide to reuse this segment as the new ephemeral seg. The old ephemeral segment becomes a gen2 segment – as we said there can only be one ephemeral segment – and seg3 becomes the new ephemeral segment. The old gen0 now lives in this segment as the new gen1 and again, we start allocating in gen0 after that.

Fixed premature OOM bugs

Premature OOM means you have lots of free space yet you are getting an OOM exception. As I said above, having fragmentation can be a form of premature OOM because you can get OOM while you still have lots of free space on your managed heap. Besides that we also fixed some other premature OOM bugs so if you were getting premature OOM I say definitely try CLR 2.0 out if possible. The premature OOM bugs included bugs for both large and small object heap.

VM hoarding

First of all, this is a feature I would recommand you to not use unless absolutely necessary. The reason we added this feature was for 2 situations – one is when segments are created and deleted very frequently. If you simply can not avoid this, you can specify the VM hoarding feature which instead of releasing the memory back to the OS it puts the segment on a standby list. Note that we don’t do this for the segments that are larger than the normal segment size (generally this is 16MB, for server GC the segments are larger). We will use these segments later to satisfy new segment requests. So next time we need a new segment we will use one from this standby list if we can find one that’s big enough.

This feature is also useful for apps that worry about fragmenting the VM space too much and want to hold onto the segments that they already acquired, like some server apps that need to frequently load and unload small DLLs and they want to keep what they already reserved so the DLLs don’t fragment VM all over the place.

Since the feature should be used very conservatively it’s only available via hosting API – you can turn it on by specifying the STARTUP_HOARD_GC_VM flag.

That’s all I wanted to talk about for this blog entry. There were of course other changes and it’s not possible to list them all but I think those are (mainly the first two) what affect users the most.

GC talk at the 2005 PDC

maoni — Thu, 28 Jul 2005 01:31:00 GMT

I will be giving a GC talk at the PDC this September. This talk is to give you a close up view of the CLR GC so I hope to see all you hard core .NET developers there!

I will talk about some internal details of generations, allocations, different flavors of GC and fragmentation (what we have done in the GC and what you can do in your applications to reduce it). I'll also mention some advanced programming techniques to improve your managed memory usage.

WinDev in Boston

maoni — Tue, 07 Sep 2004 03:19:00 GMT

I am going to WinDev this year to give 2 performance talks. Check out:

http://butrain.bu.edu/windev/track2.asp (it's from Oct 25 to 29)

Below is the description:

C5 - CLR Performance
Write faster managed code! Learn about performance engineering, tools and issues specific to managed code including: garbage collection (GC), managed code deployment and common managed code pitfalls. The session also covers some of the performance improvements scheduled for the next version of the CLR.

C9 - CLR Interoperability Performance
Learn about the different types of interoperability supported by the CLR and the basic performance guidelines associated with each. Topics include: pinvoke performance issues, reverse pinvoke performance issues, and COM interop performance issues.

Hope to see you there!

Hello World

maoni — Fri, 04 Jun 2004 17:23:00 GMT

Yep, now I have a blog too.

I work on the CLR Performance Team so naturally I will be writing about performance.