As you all know, in CLR memory management is done by Garbage collector (GC). When GC can't find memory in preallocated memory chunk (GC heap) for new objects and can't book enough memory from the OS to expand GC heap, it throws OutOfMemoryException (OOM).
The problem
From time to time, I've heard complaints about OOM - people analyze code and monitor memory usage, find out that sometimes their .NET applications throw OOM when there's enough free memory. In most cases I've seen, the problems are:
A glance of GC heap
Usually managed heap shouldn't suffer from fragmentation problem because the heap is compacted during GC. Blow shows an oversimplified model of CLR's GC heap:
|---------|
|free |
|_________|
|Object B |
| |
|Object A |
| ... |
|Object C |
|Object C | (marked)
|Object A | (marked)
Free space in GC heap
In above model, you can see that GC actually does a good job to defragment the heap. Free space is always at top of the heap and available for new allocation. But in real production, free space could reside among allocated objects. That is because:
before GC: after GC:
|---------| |---------|
|Object C | (pinned, reachable) |Object C | (pinned)
|_________| |_________|
|Object B | (unreachable) | free |
| | | |
|Object A | (reachable) |Object A |
| ... | | ... |
How pinning could fragment GC heap
if an application keeps pinning objects in this pattern: pin a new object, do some allocation, pin another object, do some allocation ... and all pinned objects remain pinned for long time, a lot of free space will be created, showed below:
|Pinned 1 |
|Pinned 2 |
|Pinned n |
| free |
Such a process could create a GC heap with a lot of free slots. Those free slots are being partially reused for allocation but when they are too small or when their remainder is too small, GC can’t use them as long as the objects are pinned. This would prevent GC from using the heap efficiently and might cause OOM eventually.
One thing makes the situation worse is that although a developer may not use pinned objects directly, some .Net libraries use them under the hood, like asynchronized IO. For example, in V1.0 and V1.1 the buffer passed to Socket.BeginReceive is pinned by the library so that unmanaged code could access the buffer. Consider a socket server application which handles thousands of socket requests per second and each request could take several minutes because of slow connection, GC heap could be fragmented a lot because of large amount of pinned objects and long lifetime some objects are pinned; then OOM could happen.
How to diagnose the problem
To determine if GC heap is fragmented, SOS is the best tool. Sos.dll is a debugger extension shipped with .NET framework which could check some underlying data structure in CLR. For example, “DumpHeap” could traverse GC heap and dump every object in the heap like this:
0:000>!dumpheap
Address MT Size
00a71000 0015cde8 12 Free
00a7100c 0015cde8 12 Free
00a71018 0015cde8 12 Free
00a71024 5ba58328 68
00a71068 5ba58380 68
00a710ac 5ba58430 68
00a710f0 5ba5dba4 68
...
00a91000 5ba88bd8 2064
00a91810 0019fe48 2032 Free
00a92000 5ba88bd8 4096
00a93000 0019fe48 8192 Free
00a95000 5ba88bd8 4096
total 1892 objects
Statistics:
MT Count TotalSize Class Name
5ba7607c 1 12 System.Security.Permissions.HostProtectionResource
5ba75d54 1 12 System.Security.Permissions.SecurityPermissionFlag
5ba61f18 1 12 System.Collections.CaseInsensitiveComparer
0015cde8 6 10260 Free
5ba57bf8 318 18136 System.String
In this example, “DumpHeap” shows that there are 3 small free slots (They appear as special “Free” objects) at the beginning of the heap, followed by some objects with size 68 bytes. More interestingly, the statistics shows that there are 10,260 bytes Free objects (free space among live objects), and 18,136 bytes of string totally in the heap. If you find the Free objects take a very big percentage of the heap, the heap is fragmented (in whidbey, "DumpHeap" would do more analysis about heap fragmentation). In this case, you want to check the objects nearby the free space to see what they are and who holds their roots, you could do it using “DumpObj” and “GCRoot”:
0:000>!dumpobj 00a92000
Name: System.Byte[]
MethodTable 0x00992c3c
EEClass 0x00992bc4
Size 4096(0x1000) bytes
Array: Rank 1, Type System.Byte
Element Type: System.Byte
0:000>!gcroot 00a92000
Scan Thread 0 (728)
Scan Thread 1 (730)
ESP:88cf548:Root:05066b48(System.IO.MemoryStream)->00a92000 (System.Byte[])
ESP:88cf568:Root:05066b48(System.IO.MemoryStream)->00a92000 (System.Byte[])
Scan HandleTable 9b130
Scan HandleTable 9ff18
HANDLE(Pinned):d41250:Root: 00a92000 (System.Byte[])
This shows that the object at address 00a92000 is a byte array, it's rooted by local variables in thread 1(to be precise, !GCRoot's output of roots in stack can't be trusted) and a pinned handle.
And the command "ObjSize" list all handles including pinned ones:
0:000>!objsize
HANDLE(Pinned):d41250: sizeof(00a92000) = 4096 ( 0x1000) bytes (System.Byte[])
HANDLE(Pinned):d41254: sizeof(00a95000) = 4096 ( 0x1000) bytes (System.Byte[])
HANDLE(Pinned):d41258: sizeof(00ac8b5b0) = 16 ( 0x10) bytes (System.Byte[])
Using those Sos commands, you could get a clear picture if the heap is fragmented and how. I believe Michael will have more blogs about details of Sos.
Solution
In Everett a lot of work is done in GC to recognize fragmentation caused by pinning and alleviate the situation, more work is already done in Whidbey. So hopefully, the problem won't show up in Whidbey. But besides change in the platform, user code could do something to avoid the issue too. From above analysis, we could tell:
So if pinning becomes an issue which causes OOM for a .NET application, instead of creating new object to pin every time, developers could consider preallocating the to-be-pinned objects and reusing them. That way those objects would live close to each other in older part of GC heap and the heap won’t be fragmented that much. For example, if an application keeps pinning 1K buffers (consider the socket server case), we could use such a buffer pool to get the buffers:
public class BufferPool { private const int INITIAL_POOL_SIZE = 512; // initial size of the pool private const int BUFFER_SIZE = 1024; // size of the buffers // pool of buffers private Queue m_FreeBuffers; // singleton instance private static BufferPool m_Instance = new BufferPool (); // Singleton attribute public static BufferPool Instance { get { return m_Instance; } } protected BufferPool() { m_FreeBuffers = new Queue (INITIAL_POOL_SIZE); for (int i = 0; i < INITIAL_POOL_SIZE; i++) { m_FreeBuffers.Enqueue (new byte[BUFFER_SIZE]); } } // check out a buffer public byte[] Checkout (uint size) { if (m_FreeBuffers.Count > 0) { lock (m_FreeBuffers) { if (m_FreeBuffers.Count > 0) return (byte[])m_FreeBuffers.Dequeue (); } } // instead of creating new buffer, // blocking waiting or refusing request may be better return new byte [BUFFER_SIZE]; } // check in a buffer public void Checkin (byte[] buffer) { lock (m_FreeBuffers) { m_FreeBuffers.Enqueue (buffer); } } }
public class BufferPool
{
private const int INITIAL_POOL_SIZE = 512; // initial size of the pool
private const int BUFFER_SIZE = 1024; // size of the buffers
// pool of buffers
private Queue m_FreeBuffers;
// singleton instance
private static BufferPool m_Instance = new BufferPool ();
// Singleton attribute
public static BufferPool Instance
get {
return m_Instance;
}
protected BufferPool()
m_FreeBuffers = new Queue (INITIAL_POOL_SIZE);
for (int i = 0; i < INITIAL_POOL_SIZE; i++)
m_FreeBuffers.Enqueue (new byte[BUFFER_SIZE]);
// check out a buffer
public byte[] Checkout (uint size)
if (m_FreeBuffers.Count > 0)
lock (m_FreeBuffers)
return (byte[])m_FreeBuffers.Dequeue ();
// instead of creating new buffer,
// blocking waiting or refusing request may be better
return new byte [BUFFER_SIZE];
// check in a buffer
public void Checkin (byte[] buffer)
m_FreeBuffers.Enqueue (buffer);
This posting is provided "AS IS" with no warranties, and confers no rights. Use of included samples are subject to the terms specified at http://www.microsoft.com/info/cpyright.htm"