Some time ago I saw a problem from a partner team in Microsoft that an InvalidOperationException is thrown from WeakReference.IsAlive. WeakReference wraps weak GC handle implemented in CLR's Execution Engine (GC handle is also exposed by System.Runtime.InteropServices.GCHandle which supports not only weak handles, but other types too). A weak GC handle will be allocated and assigned to the WeakReference object when the WeakReference object is created. As described by Jeffrey Richiter, the weak GC handle contains pointer to an object, if the object is collected by GC, the GC handle will be cleared to NULL. Most of time WeakReference.IsAlive returns true or false to indicate whether the tracked object is alive. The check is based on whether the underlying GC handle contains a non-NULL pointer or NULL. Similarly, WeakReference.get_Target will return a valid object reference or null. But after the WeakReference object itself becomes unrooted and finalized, the underlying GC handle will be destroyed and any call to IsAlive or get_Target on the WeakReference object will throw InvalidOperationException in V1.X.

How would any method being called on a finalized object? Well, if one object O has a field WR as WeakReference, when O become unrooted and there are no other roots for WR, both objects are considered to be dead and will be put into F-reachable queue for finalization(check also check Jeffrey's article). Since there are no guarantee about order of finalizers (things are a little bit different for critical finalizer), when O's finalizer is executed, WR may already be finalized. Thus inside O's finalizer or after O is resurrected, it could call methods on the finalized WR. In the example I mentioned at the beginning, the problem is some object's finalizer is calling IsAlive on its WeakReference field.

The guideline for finalization says not to use any finalizable field in finalizer, so it's fair for WeakReference's properties to throw exception if they are called in finalizer. But it might be hard for people to understand why IsAlive needs to throw. After all it's only used to check status and doesn't need to access the tracked object if it's already collected. I think the reasoning is that IsAlive is meant to check whether the underlying GC handle tracks a live object, but if the GC handle is already gone during WeakReference's finalization, we can't answer the question.

The most interesting part is to look at history of this design decision. In V1.X, both IsAlive and get_Target property throws exception after the WeakRerence object is finalized; in Beta2 of V2.0, IsAlive still throws, but get_Target won't throw, it will return null after finalization; after Beta2, we made a change so that IsAlive won't throw either, it will return false after finalization. Partly because there are too many people calling WeakReference.IsAlive in finalizers, and it is not a really dangerous thing to do. Note that WeakReference.set_Target always throws after finalization.

So on CLR V2.0 offical released build, you could safely use WeakReference in finalizer.But it is still good practice not to use finalizable objects in finalizer, including WeakReference. 

With knowledge in my previous blog, we could avoid some mistakes in .NET programming.

A C++ Thread is very resource heavy. It is associated with a lot of dynamically allocated memory and some OS handles. So it had better to be cleaned up ASAP after its corresponding OS thread dies. C++ Thread class has a reference count. For its object to be deleted, the ref count has to be dropped to 0 (Rotor: Thread::DecExternalCount in vm\threads.cpp). One interesting point is that the C# Thread object actually keeps a reference to its associated C++ Thread, so a live C# Thread object could keep its C++ Thread from being deleted even if the OS thread is already dead. (On the other hand, C++ Thread also has a reference to C# Thread, but it will break the circle when its own ref count drops to 1). Because C# Thread is a managed object, its lifetime is mostly determined by users. Plus, C# Thread class has a finalizer, so its lifetime will be extended at least one GC.  So if user code caches the C# Thread objects or have some ill-behaved finalizers (in another blog entry, I mentioned wrong-doing finalizer on one object could prevent all other object's fianlizer from running), "dead" C++ Thread objects may accumulate over time and some "memory leak" will be observed. 

I have an example here to demo the problem and how to debug it using windbg + SOS. In this process, there are 202 C++ Thread objects. Among which 160 are "dead", meaning their associated OS threads are dead. Number of total threads in Thread Store and dead/unstarted threads are showed in "!threads" output. For a "live" thread, OS and debugger thread ID are printed out for the entry, for a "dead" thread, "XXX" is marked at beginning of the line:

0:043> !threads
ThreadCount: 202
UnstartedThread: 0
BackgroundThread: 1
PendingThread: 0
DeadThread: 160
                                  PreEmptive   GC Alloc                     Lock    
        ID  ThreadOBJ       State     GC       Context           Domain     Count APT Exception
  0 0x1138 0x0015a298        0x20 Enabled  0x00000000:0x00000000 0x00149ac8     1 Ukn
  1 0x1148 0x00152530      0x1220 Enabled  0x00000000:0x00000000 0x00149ac8     0 Ukn (Finalizer)
  3 0x114c 0x00177548   0x2001020 Enabled  0x00000000:0x00000000 0x00149ac8     0 Ukn
  4 0x1150 0x00177878   0x2001020 Enabled  0x00000000:0x00000000 0x00149ac8     0 Ukn
  5 0x1154 0x00177c08   0x2001020 Enabled  0x00000000:0x00000000 0x00149ac8     0 Ukn
…
 42 0x11e4 0x00180460   0x2001020 Enabled  0x00000000:0x00000000 0x00149ac8     0 Ukn
 22 0x11e8 0x00180838   0x2001020 Enabled  0x00000000:0x00000000 0x00149ac8     0 Ukn
XXX      0 0x00180c10      0x1820 Enabled  0x00000000:0x00000000 0x00149ac8     0 Ukn
XXX      0 0x00180fe8      0x1820 Enabled  0x00000000:0x00000000 0x00149ac8     0 Ukn
XXX      0 0x001813c0      0x1820 Enabled  0x00000000:0x00000000 0x00149ac8     0 Ukn
XXX      0 0x00181750      0x1820 Enabled  0x00000000:0x00000000 0x00149ac8     0 Ukn
XXX      0 0x00181b28      0x1820 Enabled  0x00000000:0x00000000 0x00149ac8     0 Ukn
XXX      0 0x00181f00      0x1820 Enabled  0x00000000:0x00000000 0x00149ac8     0 Ukn
XXX      0 0x001822d8      0x1820 Enabled  0x00000000:0x00000000 0x00149ac8     0 Ukn
… //continue with a huge list

Now I want to find out why all the "dead" C++ Thread objects are still around. First I could check its ref count if I have symbols for mscorwks.

//0x00180c10 is a dead ThreadOBJ I picked from !Threads output
0:043> dt mscorwks!Thread 0x00180c10 m_ExternalRefCount
   +0x0cc m_ExternalRefCount : 1

Since the ref count is 1, if this C++ Thread object has a C# Thread object associated with it, the C# object must be the last reference. I could verify if that is the case by checking the C++ object's m_ExposedObject field. It is a weak GC handle (a unmovable pointer to GC reference which doesn't counted as root of the GC object), so dereference it will get the managed object. As mentioned before, C++ Thread object also has a strong handle (m_StrongHndToExposedObject field) to the C# object, but it already cleared the strong handle when ref count drops to 1 to avoid circular reference.

0:043> dt mscorwks!Thread 0x00180c10 m_ExposedObject
   +0x0c0 m_ExposedObject : 0x00a71054

0:043> dp 0x00a71054 l1
00a71054  00c5c714

0:043> !do 00c5c714
Name: System.Threading.Thread
MethodTable 0x79bb8384
EEClass 0x79bb85b0
Size 60(0x3c) bytes
GC Generation: 0
mdToken: 0x020000eb  (c:\windows\microsoft.net\framework\v1.1.4322\mscorlib.dll)
FieldDesc*: 0x79bb8614
        MT      Field     Offset                 Type       Attr      Value Name
0x79bb8384 0x4000330      0x4                CLASS   instance 0x00000000 m_Context
0x79bb8384 0x4000331      0x8                CLASS   instance 0x00000000 m_LogicalCallContext
0x79bb8384 0x4000332      0xc                CLASS   instance 0x00000000 m_IllogicalCallContext
0x79bb8384 0x4000333     0x10                CLASS   instance 0x00000000 m_Name
0x79bb8384 0x4000334     0x14                CLASS   instance 0x00000000 m_ExceptionStateInfo
0x79bb8384 0x4000335     0x18                CLASS   instance 0x00000000 m_Delegate
0x79bb8384 0x4000336     0x1c                CLASS   instance 0x00000000 m_PrincipalSlot
0x79bb8384 0x4000337     0x20                CLASS   instance 0x00000000 m_ThreadStatics
0x79bb8384 0x4000338     0x24                CLASS   instance 0x00000000 m_ThreadStaticsBits
0x79bb8384 0x4000339     0x28                CLASS   instance 0x00000000 m_CurrentCulture
0x79bb8384 0x400033a     0x2c                CLASS   instance 0x00000000 m_CurrentUICulture
0x79bb8384 0x400033b     0x30         System.Int32   instance 2 m_Priority
0x79bb8384 0x400033c     0x34         System.Int32   instance 1575952 DONT_USE_InternalThread
0x79bb8384 0x400033d        0                CLASS     shared   static m_LocalDataStoreMgr
    >> Domain:Value 0x00149ac8:0x00c05338 <<

Then I want to check root of the C# Thread object to see who keeps it alive:

0:043> !gcroot 00c5c714
Scan Thread 0 (0x1138)
ESP:12f69c:Root:0xc5b3f4(System.Object[])->0xc5c714(System.Threading.Thread)
…

So there is an array who keeps a reference to a "dead" C# Thread. This looks interesting. I could check all other C# Thread objects in the process using !DumpHeap command. !DumpHeap could dump objects in GC heap for a particular type specified by "-type" option:

0:043> !DumpHeap -type System.Threading.Thread
   Address         MT     Size  Gen
0x00c054c8 0x79bb8384       60    1 System.Threading.Thread
0x00c5b730 0x79bc81d4       28    2 System.Threading.ThreadStart
0x00c5b74c 0x79bb8384       60    2 System.Threading.Thread
0x00c5b7bc 0x79bc81d4       28    2 System.Threading.ThreadStart
0x00c5b7d8 0x79bb8384       60    2 System.Threading.Thread
0x00c5b820 0x79bc81d4       28    2 System.Threading.ThreadStart
0x00c5b83c 0x79bb8384       60    2 System.Threading.Thread
0x00c5b884 0x79bc81d4       28    2 System.Threading.ThreadStart
0x00c5b8a0 0x79bb8384       60    2 System.Threading.Thread
0x00c5b8e8 0x79bc81d4       28    2 System.Threading.ThreadStart
0x00c5b904 0x79bb8384       60    2 System.Threading.Thread
0x00c5b94c 0x79bc81d4       28    2 System.Threading.ThreadStart
0x00c5b968 0x79bb8384       60    2 System.Threading.Thread
0x00c5b9b0 0x79bc81d4       28    2 System.Threading.ThreadStart
…//long list
total 401 objects

Because !DumpHeap match type by string, so it also dumps ThreadStart objects. Because every C# Thread object created by user code always has a ThreadStart object (but C# Thread created by System.Thread.CurrentThread may not have a ThreadStart), so they show up as a pair. among 401 such objects, 200 are C# Thread objects, roughly match the number of C++ Thread objects (the number doesn't have to be the same because not every C++ Thread object has a C# counterpart created). Generation for most of C# Thread objects are 2, meaning they already survive at least 2 GCs. When I track roots of those C# Thread objects, they all point to the array. In this case, we need to look closely to the source to see whether it is necessary to cache all the C# Thread objects in an array.

Another related topic is that CLR relies on DLL_THREAD_DETACH notification to mscorwks.dll's DllMain (Rotor: EEDllMain in vm\ceemain.cpp) to know an OS thread is dead, thus detach the related C++ Thread. Using TerminateThread API is already notoriously bad in unmanaged programming, here we see another reason not to call it in managed code: if TerminateThread is called on a managed OS thread, among other bad effect (e.g. back out code not executed), CLR will not get thread detach notification. Because C++ Thread object has references to OS thread's stack address, failing to detach it from the OS thread will cause crash at random place.

This posting is provided "AS IS" with no warranties, and confers no rights.

Question: How many threads does a typical managed process have when it just starts to run?  

Answer: regardless how many threads the user creates, there are at least 3 threads for a common managed process after CLR starts up: a main thread which starts CLR and run user's Main method, CLR debugger helper thread which provides debugging service for interop debuggers like Visual Studio, and the finalizer thread which runs finalizers for unreachable objects. Depends on what the program does, CLR might create more threads to perform special tasks.   

Sometimes it is important to know what "special" threads would be created in CLR so we could understand better the implicit impact of our managed programs. Here is a list of most common special threads: 

1. Finalizer thread. The thread is to run finalizers for "dead" objects. This thread is created when GC heap is initialized during EE start up. In Rotor, the thread proc for the thread is GCHeap::FinalizerThreadStart in vm\gcee.cpp. Because GC is undeterministic and finalizers are executed in a separate thread, you can't predict when exactly an object will be finalized. Because there is only one thread to run all finalizers, if one finalizer is blocked, no other finalizers could run. So it is discouraged to take any lock in finalizer. Also see Maoni Stephens's blog for details about finalizer thread.

2. Debugger helper thread. As its name suggests, this thread helps debuggers (mixed mode and managed debugger, but not pure unmanaged debug like windbg) to get information of the managed process and to execute certain debugging operations. The thread is created when EE initializes debugger during start up. In Rotor, the thread proc for this thread is DebuggerRCThread::ThreadProcStatic (debug\ee\Rcthread.cpp). Also see Mike Stall's blog about impact of this helper thread。

3. Concurrent GC thread (doesn't exist in Rotor). As explained in Maoni and Chris Lyon's blog, concurrent GC is a special GC mode which allows garbage to be collected while managed threads are running simultaneously. To achieve this goal, CLR creates a thread to perform GC concurrently with user threads. The thread is only created when CLR decides to do a concurrent GC (even when concurrent GC mode is on, not every GC is concurrent, read Maoni's blog for details) and will be recycled when there are no concurrent GC work to do.

4. Server GC threads (doesn't exist in Rotor). Maoni and Chris also explained Server GC mode where on multi-process machine CLR creates one GC heap for each CPU and one thread to do GC for each heap. When Server GC mode is enabled, server GC threads will be created at EE start up time when GC heaps are initialized.

5. App Domain unload helper thread. In CLR V1.X, when a thread requests to unload an App Domain and the thread is in that App Domain itself, it needs to create a worker thread to do the unloading work. The worker thread will be dead once the target AD is unloaded. In Rotor, the thread starts with UnloadThreadWorker.ThreadStart (bcl\system\Appdomain.cs). In Whidbey, all AD unload work is performed in a special thread regardless whether the requesting thread is in the unloading domain. The helper thread is created when first non-default App Domain is created (default domain is never unloaded) and will stay alive since then. Also see Chris Brumme's blog about details of AD unload.

6. Threadpool threads. Depends on how a program use CLR threadpool, CLR might create threads of a varieties of types. There is only one thread for some thread type. For other types, number of threads is related to number of CPUs, the work load, and some user configurable settings. The thread types including wait threads (threads to perform asynchronized wait, could be more than one); worker threads (threads to execute user work item, could be more than one); Completion port threads (threads wait for completion port IO in Windows, could be more than one, doesn't exist in Rotor); Gate thread (thread help to monitor status of completion port threads and worker threads, only one); Timer thread (thread manages timer queue, only one).

As a C++ fan, I'm a long time admirer for deterministic finalization. I think introduction of garbage collection to C style language by Java and .Net is a huge improvement. However, I found lose of deterministic destructor is almost unacceptable when I first enter Java/.Net world. Of course I'm used to it now, but it's still quite confusing to me for C# to use C++ destructor syntax for Finalizer. And in managed extension of C++, destructor becomes something totally different for managed data type. I bet a lot of experienced C++ developers make mistake to use finalizer as if it was destructor when they first try .Net. So when I read the new changes in Whidbey version of C++ from Stan Lippman's blog, I'm very excited and can't wait to give it a try.

But before we look into the new features, let's go over how old version of managed C++ handles destructors. I wrote this simple program:

#using <mscorlib.dll>

using namespace System;

__gc class RefT
{
public:
   RefT () {Console::WriteLine ("RefT::RefT");}
   ~RefT () {Console::WriteLine ("RefT::~RefT");}
};

 

__value class ValueT
{ 
    ValueT () {Console::WriteLine ("ValueT::ValueT");}
    //destructor is not allowed for value type
};

int main()
{ 
    { //1. auto-generated finalizer will be called in asynchronizied fashion
        RefT * prt = new RefT;
    }

   { //2. Dispose is called at “delete” and finalizer will be suppressed
       RefT * prt = new RefT;
       delete prt;
   }

  {
      ValueT vt;
   }
 
  {
      //value type can't be created in GC heap
      ValueT * pvt = __nogc new ValueT;
      delete pvt;
   }
   return 0;
}

I compiled it with V1.1 C++ compiler and checked generated IL code using ildasm. RefT is compiled to something like this:

class RefT
{
    RefT ()
   {
       Console.WriteLine (“RefT::RefT”);
   }

   void Finalize ()
   {
       Console.WriteLine (“Ref::~Ref”);
   }

   void __dtor ()
   {
       GC.SuppressFinalize (this);
       Finalize ();
   }
}

Here we could see that C++ destructor is mapped to CLR finalizer and a method “__dtor” is added to call finalizer and SupressFinalize.

In main, the first block creates a RefT object in heap and leaves it as garbage. Sometime later, finalizer (~RefT) will run and the object will be collected. In the second block, we “delete” the object. This is translated into a call to __dtor in IL. So “delete” acts more like Dispose method recommended by IDisposable pattern: the object is not freed but the contents are disposed and finalizer won't run on the object later.

Test.cs (compiled to DelegateExample.exe): 

using System;
using System.Threading;
using System.Runtime.InteropServices;

class Test
{
 delegate uint ThreadProc (IntPtr arg);

 private uint m;

 public Test (uint n)
 {
  m = n;
 }

 uint Reflect (IntPtr arg)
 {
  Console.WriteLine (m);
  return m;
 }

 static void Main ()
 {
  Test t = new Test (1);
  ThreadProc tp = new ThreadProc (t.Reflect);
  NewThread (tp);
  Thread.Sleep (1000);
 }

 [DllImport("UsingCallback")]
 static extern void NewThread (ThreadProc proc);
} 

UsingCallback.cpp (compiled to UsingCallback.dll):

_stdcall  void NewThread (LPTHREAD_START_ROUTINE cb)
{
 DWORD id = 0;
 CreateThread (NULL, 0, cb, NULL, 0,&id);
}

Yes, here is the problem: in the cs file, the managed code passes a Delegate object to unmanaged code which will create a new thread to call the delegate. Since unmanaged code has no way to tell CLR how it plans to use the object, from CLR's point of view, there are no live roots for the Delegate object after the line “NewThread (tp);”. Thus the object is eligible to be garbage collected (GC) after the call, even if the new thread might not start yet. So it's possible for the Delegate to become trash before unmanaged code invokes it and cause unspecified failure. A fix is to add GC.KeepAlive before Main returns:

         Test t = new Test (1);
        ThreadProc tp = new ThreadProc (t.Reflect);
        NewThread (tp);
        Thread.Sleep (1000);
        GC.KeepAlive (tp);

One thing annoying about this kind of bug is that GC is nondeterministic, the program could work just fine 99% of time, but only crashes under stress situation. Another thing make the problem hard to find is that it's not very intuitive by looking at the source. You might know the theory that when a variable is not used anymore, it will not be reported as a live root; but it would take quite some time to figure out at which point which variable is dead, plus there's no way to verify if CLR agrees with your analysis.

I'll show you how to use SOS.dll to check the internal data structure used by CLR to determine variable lifetime. When JIT compiles code, it generates such variable aliveness information (GC info) for each method and saves it along with the machine code of the method. When GC happens, it will check GC info for every method in the stack to find out which variable is alive and will use the live variables as object roots. GC info is highly compacted, but SOS.dll has “!GCInfo” command to crack it and show it in a human-readable way. This approach only works in assembly level, so stop reading if you are not interested in assembly language. :)

I compiled test.cs above using Visual studio to a "Debug" build, and launched the program under Windbg. After the Main method is JITted, I could disassemly the generated native code of the method using "!SOS.u" command:

0:000> !u 02f00058
Will print '>>> ' at address: 02f00058
Normal JIT generated code
[DEFAULT] Void Test.Main()
Begin 02f00058, size 60
>>> 02f00058 55               push    ebp
02f00059 8bec             mov     ebp,esp
02f0005b 83ec08           sub     esp,0x8
02f0005e 57               push    edi
02f0005f 56               push    esi
02f00060 53               push    ebx
02f00061 33ff             xor     edi,edi
02f00063 33db             xor     ebx,ebx
02f00065 b9e850ad00       mov     ecx,0xad50e8 (MT: Test)
02f0006a e8a91fbcfd       call    00ac2018 (JitHelp: nc)
02f0006f 8bf0             mov     esi,eax
02f00071 8bce             mov     ecx,esi
02f00073 ba01000000       mov     edx,0x1
02f00078 ff152051ad00     call    dword ptr [00ad5120] (Test..ctor)
02f0007e 8bfe             mov     edi,esi
02f00080 b9c451ad00       mov     ecx,0xad51c4 (MT: Test/ThreadProc)
02f00085 e88e1fbcfd       call    00ac2018 (JitHelp: nc)
02f0008a 8bf0             mov     esi,eax
02f0008c 689350ad00       push    0xad5093
02f00091 8bd7             mov     edx,edi
02f00093 8bce             mov     ecx,esi
02f00095 ff152452ad00     call    dword ptr [00ad5224] (Test/ThreadProc..ctor)
02f0009b 8bde             mov     ebx,esi
02f0009d 8bcb             mov     ecx,ebx
02f0009f ff152c51ad00     call    dword ptr [00ad512c] (Test.NewThread)
02f000a5 b9e8030000       mov     ecx,0x3e8
02f000aa ff155084bb79  call dword ptr [mscorlib_79990000+0x228450 (79bb8450)] (System.Threading.Thread.Sleep)
02f000b0 90               nop
02f000b1 5b               pop     ebx
02f000b2 5e               pop     esi
02f000b3 5f               pop     edi
02f000b4 8be5             mov     esp,ebp
02f000b6 5d               pop     ebp
02f000b7 c3               ret 

SOS's "!u" is similar to Windbg's "u", but it shows more data because managed code is self-describable. For example, for those indirect calls, if the target is a managed function, "!u" could tell us the function name. 

To check its GC info, we need to get the method's MethodDesc first (Method descriptor, CLR's data structure to keep all information about one method). We could use "!ip2md" to find out the method desc from any instruction pointer in the method: 

0:000> !ip2md 02f00058
MethodDesc: 0x00ad50a8
Jitted by normal JIT
Method Name : [DEFAULT] Void Test.Main()
MethodTable ad50e8
Module: 151ad0
mdToken: 06000003 (D:\projects\DelegateExample\bin\Debug\DelegateExample.exe)
Flags : 10
Method VA : 02f00058

This command shows some important information about the method. To check GC info, we only need to pass the MethodDesc pointer itself to “!GCInfo”: 

0:000> !gcinfo 0x00ad50a8
Normal JIT generated code
Method info block:
    method      size   = 0060
    prolog      size   =   9
    epilog      size   =   7
    epilog     count   =   1
    epilog      end    = yes
    saved reg.  mask   = 000F
    ebp frame          = yes
    fully interruptible=yes
    double align       = no
    security check     = no
    exception handlers = no
    local alloc        = no
    edit & continue    = yes
    varargs            = no
    argument   count   =   0
    stack frame size   =   2
    untracked count    =   0
    var ptr tab count  =   0
    epilog        at   0059
60 E5 C0 45    |
 

Pointer table:
F0 7B          | 000B        reg EDI becoming live
5A             | 000D        reg EBX becoming live
F0 42          | 0017        reg EAX becoming live
72             | 0019        reg ESI becoming live
4A             | 001B        reg ECX becoming live
F0 03          | 0026        reg EAX becoming dead
08             | 0026        reg ECX becoming dead
F0 44          | 0032        reg EAX becoming live
30             | 0032        reg ESI becoming dead
72             | 0034        reg ESI becoming live
57             | 003B        reg EDX becoming live
4A             | 003D        reg ECX becoming live
06             | 0043        reg EAX becoming dead
08             | 0043        reg ECX becoming dead
10             | 0043        reg EDX becoming dead
4C             | 0047        reg ECX becoming live
0E             | 004D        reg ECX becoming dead
30             | 004D        reg ESI becoming dead
F1 1B          | 0060        reg EBX becoming dead
38             | 0060        reg EDI becoming dead
FF             | 

Output of the command has 2 sections, the first part is method info block, which contains some basic information about the JITted code, like size of the method, size of the prolog, and etc. The 2^nd part is pointer table, on which I'll spend most of time. Pointer table describes lifetime of every GC reference inside the method. It has 3 columns, the first one is byte encodings, which we don't need to care about; the second column is the offset in the JITted code. E.g, this method's code starts from 02f00058, so 000B means the instruction at 02f00058+B = 2F00063, “xor     ebx,ebx”;  the third column tells us change of lifetime for a GC pointer at that instruction. E,g, “000B        reg EDI becoming live” means starts from 2F00063, register EDI is a live root; similiarly, we can see EDI becomes dead at offset 60, end of the method (0060        reg EDI becoming dead). So whatever variable EDI is used to store, its lifetime is from beginning to end of the method. To understand the pointer table, it's better to interweave the table with the JITted code. In Whidbey, SOS has "!u -gcinfo" to do the job; but for Everett, I have to manually put them together and show it along with the source code:

02f00058 55               push    ebp
02f00059 8bec             mov     ebp,esp
02f0005b 83ec08           sub     esp,0x8
02f0005e 57               push    edi
02f0005f 56               push    esi
02f00060 53               push    ebx
02f00061 33ff             xor     edi,edi
    GCInfo:      000B        reg EDI becoming live
02f00063 33db             xor     ebx,ebx
    GCInfo:      000D        reg EBX becoming live

Test t = new Test (1); 

02f00065 b9e850ad00       mov     ecx,0xad50e8 (MT: Test)
02f0006a e8a91fbcfd       call    00ac2018 (JitHelp: nc)
    GCInfo:      0017        reg EAX becoming live
02f0006f 8bf0             mov     esi,eax
    GCInfo:      0019        reg ESI becoming live
02f00071 8bce             mov     ecx,esi
    GCInfo:      001B        reg ECX becoming live
02f00073 ba01000000       mov     edx,0x1
02f00078 ff152051ad00     call    dword ptr [00ad5120] (Test..ctor)
    GCInfo:      0026        reg EAX becoming dead
    GCInfo:      0026        reg ECX becoming dead
02f0007e 8bfe             mov     edi,esi

ThreadProc tp = new ThreadProc (t.Reflect);

02f00080 b9c451ad00       mov     ecx,0xad51c4 (MT: Test/ThreadProc)
02f00085 e88e1fbcfd       call    00ac2018 (JitHelp: nc)
    GCInfo:      0032        reg EAX becoming live
    GCInfo:      0032        reg ESI becoming dead
02f0008a 8bf0             mov     esi,eax
    GCInfo:      0034        reg ESI becoming live
02f0008c 689350ad00       push    0xad5093
02f00091 8bd7             mov     edx,edi
    GCInfo:      003B        reg EDX becoming live
02f00093 8bce             mov     ecx,esi
    GCInfo:      003D        reg ECX becoming live
02f00095 ff152452ad00     call    dword ptr [00ad5224] (Test/ThreadProc..ctor)
    GCInfo:      0043        reg EAX becoming dead
    GCInfo:      0043        reg ECX becoming dead
    GCInfo:      0043        reg EDX becoming dead

NewThread (tp);

02f0009b 8bde             mov     ebx,esi
02f0009d 8bcb             mov     ecx,ebx
    GCInfo:      0047        reg ECX becoming live
02f0009f ff152c51ad00     call    dword ptr [00ad512c] (Test.NewThread)
    GCInfo:      004D        reg ECX becoming dead
    GCInfo:      004D        reg ESI becoming dead

Thread.Sleep (1000);

02f000a5 b9e8030000       mov     ecx,0x3e8
02f000aa ff155084bb79  call dword ptr [mscorlib_79990000+0x228450 (79bb8450)] (System.Threading.Thread.Sleep)
02f000b0 90               nop
02f000b1 5b               pop     ebx
02f000b2 5e               pop     esi
02f000b3 5f               pop     edi
02f000b4 8be5             mov     esp,ebp
02f000b6 5d               pop     ebp
02f000b7 c3               ret

    GCInfo:      0060        reg EBX becoming dead
    GCInfo:      0060        reg EDI becoming dead

Let's assume a thread triggers GC when another thread is excuting code 02f0009d. From the table, we know at this time, register ESI, EBX, and EDI will be reported as live roots to GC. With some disassembly, we could see at this moment both ESI and EBX contain reference to Delegate tp, and EDI is variable t. It might surprise you that both EBX and EDI keep alive until the function ends. That means variable t and tp are actually alive for the whole function, so the code has no bug?!

The tricky point is that a variable is eligible to be dead once it's not used anymore. However, it's up to JIT to determine whether it really wants to report the variable to be dead. In fact, for debuggable code, JIT extends lifetime for every variable to end of the function.

To prove the premature GC bug really exists in the sample, I compiled it to "Release" build and repeated all the steps above:

0:000> !u 02df0058
Loaded Son of Strike data table version 5 from "C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\mscorwks.dll"
Will print '>>> ' at address: 02df0058
Normal JIT generated code
[DEFAULT] Void Test.Main()
Begin 02df0058, size 46
>>> 02df0058 57               push    edi
02df0059 56               push    esi
02df005a b9e850ad00       mov     ecx,0xad50e8 (MT: Test)
02df005f e8b41fcdfd       call    00ac2018 (JitHelp: nc)
02df0064 8bf0             mov     esi,eax
02df0066 c7460401000000   mov     dword ptr [esi+0x4],0x1
02df006d b9bc51ad00       mov     ecx,0xad51bc (MT: Test/ThreadProc)
02df0072 e8a11fcdfd       call    00ac2018 (JitHelp: nc)
02df0077 8bf8             mov     edi,eax
02df0079 689350ad00       push    0xad5093
02df007e 8bd6             mov     edx,esi
02df0080 8bcf             mov     ecx,edi
02df0082 ff151c52ad00     call    dword ptr [00ad521c] (Test/ThreadProc..ctor)
02df0088 8bcf             mov     ecx,edi
02df008a ff152c51ad00     call    dword ptr [00ad512c] (Test.NewThread)
02df0090 b9e8030000       mov     ecx,0x3e8
02df0095 ff155084bb79  call dword ptr [mscorlib_79990000+0x228450 (79bb8450)] (System.Threading.Thread.Sleep)
02df009b 5e               pop     esi
02df009c 5f               pop     edi
02df009d c3               ret

0:000> !ip2md 02df0058
MethodDesc: 0x00ad50a8
Jitted by normal JIT
Method Name : [DEFAULT] Void Test.Main()
MethodTable ad50e8
Module: 151ad0
mdToken: 06000003 (D:\projects\DelegateExample\bin\Release\DelegateExample.exe)
Flags : 10
Method VA : 02df0058

0:000> !gcinfo 0x00ad50a8
Normal JIT generated code
Method info block:
    method      size   = 0046
    prolog      size   =   2
    epilog      size   =   3
    epilog     count   =   1
    epilog      end    = yes
    saved reg.  mask   = 0003
    ebp frame          = no 
    fully interruptible=no
    double align       = no
    security check     = no
    exception handlers = no
    local alloc        = no
    edit & continue    = no
    varargs            = no
    argument   count   =   0
    stack frame size   =   0
    untracked count    =   0
    var ptr tab count  =   0
    epilog        at   0043
46 21          | 

Pointer table:
A9             | 001F        call 0 [ ESI ]
07             | 0026        push
CB             | 0030        call 1 [ EDI ]
FF             |

Here is mixed source code, native code and pointer table: 

02df0058 57               push    edi
02df0059 56               push    esi

Test t = new Test (1);

02df005a b9e850ad00       mov     ecx,0xad50e8 (MT: Test)
02df005f e8b41fcdfd       call    00ac2018 (JitHelp: nc)
02df0064 8bf0             mov     esi,eax
02df0066 c7460401000000   mov     dword ptr [esi+0x4],0x1

ThreadProc tp = new ThreadProc (t.Reflect);

02df006d b9bc51ad00       mov     ecx,0xad51bc (MT: Test/ThreadProc)
02df0072 e8a11fcdfd       call    00ac2018 (JitHelp: nc)
    GCInfo:      001F        call 0 [ ESI ]
02df0077 8bf8             mov     edi,eax
02df0079 689350ad00       push    0xad5093
    GCInfo:      0026        push
02df007e 8bd6             mov     edx,esi
02df0080 8bcf             mov     ecx,edi
02df0082 ff151c52ad00     call    dword ptr [00ad521c] (Test/ThreadProc..ctor)
    GCInfo:      0030        call 1 [ EDI ]

EM>NewThread (tp);

02df0088 8bcf             mov     ecx,edi
02df008a ff152c51ad00     call    dword ptr [00ad512c] (Test.NewThread)

Thread.Sleep (1000);

02df0090 b9e8030000       mov     ecx,0x3e8
02df0095 ff155084bb79  call dword ptr [mscorlib_79990000+0x228450 (79bb8450)] (System.Threading.Thread.Sleep)
02df009b 5e               pop     esi
02df009c 5f               pop     edi
02df009d c3               ret 

In release build, the JIIted code is smaller, and the pointer table in GC info is much smaller. I need to explain some new syntax in this pointer table: “call 0 [ESI]” means the method calls a function with 0 argument, and ESI is a live variable at this point; "push" just indicates change of the stack, which is important information for GC to unwind this frame, but doesn't affect pointer aliveness.  

One thing interesting about this new pointer table is that it only reports GC references when the method calls into other methods. What if a GC happens somewhere else? For example, at line 02df0079, EDI contains object tp and ESI contains object t, if a GC happens while a thread is excuting 02df0079 but GCInfo doesn't report those two variables, will they be collected? The answer is GC can't happen at that place. There is an important field in method info block called "fully interruptable". In debug version, this field for the method is true but in release version the field is false. A method is fully interruptable means GC could stop a thread (and perform collection) at any point if the thread is executing this method. If a method is not fully interruptable, GC can't start at arbitrary point if a thread is executing this method. It has to wait until a point when the thread returns from a call (e.g, return from ThreadProc's constructor back to Main) or when the method calls into another method which allows GC to happen (calls into unmanaged code via PInvoke always allow GC to happen). That's why we only need to report varaibles at calls. For performance reason, such trivial methods like Test.Main is usually non-fully interruptable in release build.

With all the knowledge, now we know that if a GC happens when a thread is just returning from the call "new ThreadProc" at instruction 02df0072, GC knows ESI (variable t) is a live root; if a GC happens when a thread is just returning from the call to ThreadProc's constructor at 02df0082, GC knows EDI (variable tp) is a live root. Variable t is not reported this time(but it's kept alive by object tp). However if a GC happens when a thread returns from the call to NewThread at 02df008a, neither t nor tp will be reported so they could be collected by GC. If the new thread hasn't start then, it will have trouble when using the objects.

So next time if you suspect there might be some premature GC bug, you could try "!SOS.GCInfo" to see how exactly CLR thinks about the GC object's lifetime.

 This posting is provided "AS IS" with no warranties, and confers no rights.

I've been quiet for 3 months and probably won't have much time for blogs for next several months down the road. Today I got a chance to update my post OutOfMemoryException and Pinning to correct a mistake pointed out by our GC architect Patrick Dussud. Also I want to remind everyone that Michael Stanton posted a great article in April about how to traverse the GC heap using psscor.dll. This technique is quite useful to understand the internal of CLR's GC. Talking about GC heap, I want to tell another GC heap corruption story.

My test buddy Chris Lyon debugged a customer program with me recently and found a very interesting user bug. He already posted the problem in an internal blog, I think it's worthy to share it with the public community too. The problem is in this code:

SomeStruct[] arr = new SomeStruct[SomeSize];
GCHandle handle = GCHandle.Alloc(Marshal.SizeOf(typeof(SomeStruct)) * SomeSize, GCHandleType.Pinned);
IntPtr fixedAddr = handle.AddrOfPinnedObject ();
... //pass fixedAddr to unmanaged code to filled the array

This code tries to create a pinned GC handle for an array then pass the array's address to unmanaged code to fill in. The code seems to assume GCHandle.Alloc has similiar semantics as malloc - tell it how much memory we need and it will allocate it for us. But it's not true. GCHandle.Alloc only allocates a (pointer-sized) GC handle for a given object without allocating memory for the object. The first argument for this method is supposed to be the Object which the handle will point to. This program passes an Int32 (size of the array) to the method, but the compiler won't give us any error because the nice handy auto-boxing feature. At runtime what would happen is CLR will box the integer to be a heap object, allocate a pinned handle for this integer object, then address of this object will be passed to unmanaged code. The unmanaged code will copy SomeSize * sizeof(SomeStruct) byte data to the 4 byte integer and overwrites other objects in the heap. We could just relax and wait the program to crash.

This example looks quite trivial, but it demos how easy GC heap could be corrupted by user error with help of unmanaged code. One suggestion: watch out any IntPtr you passed to unmanaged code! 

This posting is provided "AS IS" with no warranties, and confers no rights.

An exsample of FCall

My friend Joel Pobar had a great post to demo how to add new code to Rotor which exposes more EE(Execution Engine) internal information to managed world. This is a very good example covers both BCL and EE, and how the two parts interact with each other. As showed in this example, BCL code could call into EE by a special type of method called “FCall“, like this:

FCIMPL1(MethodBody *, COMMember::GetMethodBody, MethodDesc **ppMethod)
{
   MethodDesc* pMethod = *ppMethod;
   METHODBODYREF MethodBodyObj = NULL;
  
   HELPER_METHOD_FRAME_BEGIN_RET_0();
   GCPROTECT_BEGIN(MethodBodyObj);

   TypeHandle thMethodBody(g_Mscorlib.FetchClass(CLASS__METHOD_BODY));
   MethodBodyObj = (METHODBODYREF)AllocateObject(thMethodBody.GetMethodTable());
   Module* pModule = pMethod->GetModule();

   COR_ILMETHOD_DECODER MethodILHeader(pMethod->GetILHeader(), pModule->GetMDImport(), TRUE);
   MethodBodyObj->maxStackSize = MethodILHeader.GetMaxStack();
  
   GCPROTECT_END();
   HELPER_METHOD_POLL();
   HELPER_METHOD_FRAME_END();

   return (MethodBody*)OBJECTREFToObject(MethodBodyObj);
}
FCIMPLEND

As I said before I'd try to explain some Rotor code in my blog, so let me start by analyzing a small problem in that piece of code. I won't call it a bug because it happens to be harmless here. But it does violate some CLR coding rules.

Some basic bricks of an FCall

First let's take a glance of those amazing macros:

• FCIMPL1/FCIMPLEND: defined in vm/fcall.h. They just tweak calling convention between managed code (BCL) and unmanaged code (EE). You can see different flavors of FCIMPL, which serve calls with different argument numbers or types (to match the argument passing and enregistering rules in managed world).
• HELPER_METHOD_FRAME_BEGIN_RET_0/HELPER_METHOD_FRAME_END: defined in vm/fcall.h. For operations like GC (Garbage Collection) and EH (Exception Handling) to work correctly, some frames have to be set up in the stack, especially at the boundary between managed part and unmanaged part. Frames are a topic could take a blog entry itself, so I won't cover it too much here. What you need to know now is that for performance reason, an FCall doesn't set up a frame by default. So when an FCall wants to throw an exception or allow a GC to happen, it has to set up a HelperMethodFrame (vm/frames.h) first. This job is done by HELPER_METHOD_FRAME_BEGIN* macro.HELPER_METHOD_FRAME_END is to tear down the frame from stack. All GC or exception throwing has to happen in the range guarded by this frame. In the code above, some operations could trigger a GC (at least AllocateObject could do so, not sure about others. It would be a hard job to trace into each code path to find out whether a GC could happen), so a HelperMethodFrame has to be established before that call.  
• GCPROTECT_BEGIN/GCPROTECT_END: defined in vm/frames.h. Similar to HELPER_METHOD_FRAME_BEGIN*, GCPROTECT_BEGIN is used to set up a GCFrame (vm/frames.h) and GCPROTECT_END pop the frame out. When a GC happens, it needs to find out all object references in stack to trace which objects in the managed heap are still alive, and when it moves the object (to compact the heap) it needs to update the references in stack with the new location of the objects. For managed code, JIT generates all information needed by GC. But for unmanaged part of CLR, the code whoever has references to managed objects is responsible to report all references itself. A GCFrame serves for this purpose. If an unmanaged method pushes a GCFrame to stack, the frame will report the protected reference (the argument to GCPROTECT_BEGIN) during GC. In our example, MethodBodyObj is an object reference so we set up a GCFrame for it.
• HELPER_METHOD_POLL: defined in vm/fcall.h. This macro is meant to do a GC poll in range of a HelperMethodFrame. GC poll is another complicated thing I don't want to talk here. Basically it allows GC to happen in another thread, without a poll another thread that wants to perform a GC might be blocked, thus all managed threads in the application will be blocked.
• OBJECTREFToObject: defined in vm/vars.hpp. It's used to take a pure object pointer out from an ObjectRef. An ObjectRef is a naked pointer in free build, but a wrapper with some very useful checking in debug build.

The problem

1. GC hole. In COMMember::GetMethodBody, a HELPER_METHOD_POLL is put after GCPROTECT_END. As I explained above, GCPROTECT_END will pop up the GC frame which is protecting the object reference, but HELPER_METHOD_POLL allows a GC to happen in another thread. So there are chances (although very small) that after the GC frame is popped up, another thread performs a GC. In such a GC, MethodBodyObj won't be reported. So GC might not know the object referenced by MethodBodyObj is still alive (if there's no other reference to the object) and collect it; or (if there are other references) GC might move the object but not update MethodBodyObj with the new address thus MethodBodyObj would hold a “stale” object pointer. Either case, COMMember::GetMethodBody will return a bogus object and the program might crash later in an unexpected way. We call this kind of errors “GC holes” in CLR. They are hard to detect because GC is non-deterministic. The funny thing is that nothing bad would happen in this method because HELPER_METHOD_POLL is actually defined as a no-op in this version:

// This is the fastest way to do a GC poll if you have already erected a HelperMethodFrame
// #define HELPER_METHOD_POLL()             { __helperframe.Poll(); INDEBUG(__fCallCheck.SetDidPoll()); }

#define HELPER_METHOD_POLL()             { }

I don't know why we use an empty macro for HELPER_METHOD_POLL but I'm sure it's supposed to be the version which is commented out. In later versions we may uncomment the above line to make HELPER_METHOD_POLL take effect. So although this FCall doesn't cause any trouble for now, it might later. The corrected version should be:

HELPER_METHOD_FRAME_BEGIN_RET_0();
GCPROTECT_BEGIN(MethodBodyObj);

...   

HELPER_METHOD_POLL();
GCPROTECT_END();
HELPER_METHOD_FRAME_END();

2. If you dig deep into the code of  HELPER_METHOD_FRAME_BEGIN*, you will find that those macros do a GC poll themselves. So unless the FCall does some very time consuming work, there's no need for another poll. Thus a refined version of our sample would be:

HELPER_METHOD_FRAME_BEGIN_RET_0();
GCPROTECT_BEGIN(MethodBodyObj);

...   

GCPROTECT_END();
HELPER_METHOD_FRAME_END();

3. Because setting up HelperMethodFrame usually means the code wants to allow GC, for convenience we have versions of HELPER_METHOD_FRAME_BEGIN* to protect object references. Then a GCFrame is not needed. So the FCall could be written this way:

HELPER_METHOD_FRAME_BEGIN_RET_1(MethodBodyObj);

...   

HELPER_METHOD_FRAME_END();

I hope you already got a taste how FCall works in CLR by reading this blog. Actually most thing I talked here can be found in comments at the beginning of vm/fcall.h. And if you want to see more examples of FCall, just search FCIMPL in vm directory, you will get plenty of them. Then you will see how CLR build a beautiful object-oriented world by the old fashion and kinda dirty way.

 This posting is provided "AS IS" with no warranties, and confers no rights.

As you all know, in CLR memory management is done by Garbage collector (GC). When GC can't find memory in preallocated memory chunk (GC heap) for new objects and can't book enough memory from the OS to expand GC heap, it throws OutOfMemoryException (OOM).

The problem

From time to time, I've heard complaints about OOM - people analyze code and monitor memory usage, find out that sometimes their .NET applications throw OOM when there's enough free memory. In most cases I've seen, the problems are:

1. The virtual address space of the OS is fragmented. This is usually caused by some unmanaged components in the application. This issue exists in unmanaged world for long time, but it could hit GC hard. GC heap is managed in unit of segments, whose size is 16MB for workstation version and 32MB for server version in V1.0 and V1.1. That means when CLR needs to expand GC heap, it has to find 32MB consecutive free virtual memory for a server application. Usually this is not a problem in a system with 2GB address space for user mode. But if there are some unmanaged DLLs in the application manipulating virtual memory without carefulness, the virtual address space could be divided into small blocks of free and reserved memory. Thus GC would fail to find a big enough piece of free memory although the total free memory is enough. This kind of problems could be found out by looking through the whole virtual address space to see which block is reserved by which component.
2. The GC heap itself is fragmented, meaning GC can't allocate objects in already reserved segments which actually have enough free space inside. I want to focus on this problem in this blog. 

A glance of GC heap

Usually managed heap shouldn't suffer from fragmentation problem because the heap is compacted during GC. Blow shows an oversimplified model of CLR's GC heap: 

• All objects are adjacent to each other; the top of heap is free space. 

       |---------|

       |free     |

       |_________|

       |Object B |

       |         |

       |_________|

       |Object A |

       |_________|

       |   ...   | 

• New objects are allocated in free space. Allocation always happens at top, just as a stack.

       |---------|

       |free     |

       |_________|

       |Object C |

       |_________|

       |Object B |

       |         |

       |_________|

       |Object A |

       |_________|

       |   ...   | 

• When free space is used up, a GC happens. During GC, reachable objects are marked. 

       |---------|

       |Object C | (marked)

       |_________|

       |Object B |

       |         |

       |_________|

       |Object A | (marked)

       |_________|

       |   ...   | 

• After GC, heap is compacted, live (reachable) objects are relocated, dead (unreachable) objects are swept out. 

       |---------|

       |free     |

       |_________|

       |Object C |

       |_________|

       |Object A |

       |_________|

       |   ...   |

Free space in GC heap

In above model, you can see that GC actually does a good job to defragment the heap. Free space is always at top of the heap and available for new allocation. But in real production, free space could reside among allocated objects. That is because:

1. Sometimes GC could choose not to compact part of the heap when it's not necessary. Since relocating all objects could be expensive, GC might avoid doing so under some conditions. In that case, GC will keep a list of free space in heap for future compaction. This won't cause heap fragmentation because GC has full control over the free space. GC could fill up those blocks anytime later when necessary.
2. Pinned objects are not movable. So if a pinned object survives a GC, it could create a block of free space, like this:

       before GC:                              after GC:

       |---------|                             |---------|

       |Object C | (pinned, reachable)         |Object C | (pinned)

       |_________|                             |_________|

       |Object B | (unreachable)               | free    |

       |         |                             |         |

       |_________|                             |_________|

       |Object A | (reachable)                 |Object A |

       |_________|                             |_________|

       |   ...   |                             |   ...   |

How pinning could fragment GC heap

if an application keeps pinning objects in this pattern: pin a new object, do some allocation, pin another object, do some allocation ... and all pinned objects remain pinned for long time, a lot of free space will be created, showed below: 

• A new object is pinned

       |---------|

       |free     |

       |_________|

       |Pinned 1 |

       |_________|

       |Object A |

       |_________|

       |   ...   | 

• After some allocation, another object is pinned

       |---------|

       |free     |

       |_________|

       |Pinned 2 |

       |_________|

       |   ...   |

       |_________|

       |Pinned 1 |

       |_________|

       |Object A |

       |_________|

       |   ...   | 

• More objects are pinned, with unpinned objects in between 

       |_________|

       |Pinned n |

       |_________|

       |   ...   |

       |_________|

       |Pinned 2 |

       |_________|

       |   ...   |

       |_________|

       |Pinned 1 |

       |_________|

       |Object A |

       |_________|

       |   ...   | 

• A GC happens, because pinned objects can't be relocated, free space remains in the heap 

       |_________|

       |Pinned n |

       |_________|

       | free    |

       |_________|

       |Pinned 2 |

       |_________|

       | free    |

       |_________|

       |Pinned 1 |

       |_________|

       | free    |

       |_________|

       |   ...   |

Such a process could create a GC heap with a lot of free slots. Those free slots are being partially reused for allocation but when they are too small or when their remainder is too small, GC can’t use them as long as the objects are pinned. This would prevent GC from using the heap efficiently and might cause OOM eventually.

One thing makes the situation worse is that although a developer may not use pinned objects directly, some .Net libraries use them under the hood, like asynchronized IO. For example, in V1.0 and V1.1 the buffer passed to Socket.BeginReceive is pinned by the library so that unmanaged code could access the buffer. Consider a socket server application which handles thousands of socket requests per second and each request could take several minutes because of slow connection, GC heap could be fragmented a lot because of large amount of pinned objects and long lifetime some objects are pinned; then OOM could happen.

How to diagnose the problem

To determine if GC heap is fragmented, SOS is the best tool. Sos.dll is a debugger extension shipped with .NET framework which could check some underlying data structure in CLR. For example, “DumpHeap” could traverse GC heap and dump every object in the heap like this:

     0:000>!dumpheap

     Address       MT     Size

     00a71000 0015cde8       12 Free

     00a7100c 0015cde8       12 Free

     00a71018 0015cde8       12 Free

     00a71024 5ba58328       68

     00a71068 5ba58380       68

     00a710ac 5ba58430       68

     00a710f0 5ba5dba4       68

     ...

     00a91000 5ba88bd8     2064

     00a91810 0019fe48     2032 Free

     00a92000 5ba88bd8     4096

     00a93000 0019fe48     8192 Free

     00a95000 5ba88bd8     4096

     ...

     total 1892 objects

     Statistics:

           MT    Count TotalSize Class Name

     5ba7607c        1        12 System.Security.Permissions.HostProtectionResource

     5ba75d54        1        12 System.Security.Permissions.SecurityPermissionFlag

     5ba61f18        1        12 System.Collections.CaseInsensitiveComparer

     ...

     0015cde8        6     10260      Free

     5ba57bf8      318     18136 System.String

     ...

In this example, “DumpHeap” shows that there are 3 small free slots (They appear as special “Free” objects) at the beginning of the heap, followed by some objects with size 68 bytes. More interestingly, the statistics shows that there are 10,260 bytes Free objects (free space among live objects), and 18,136 bytes of string totally in the heap. If you find the Free objects take a very big percentage of the heap, the heap is fragmented (in whidbey, "DumpHeap" would do more analysis about heap fragmentation). In this case, you want to check the objects nearby the free space to see what they are and who holds their roots, you could do it using “DumpObj” and “GCRoot”:

     0:000>!dumpobj 00a92000

     Name: System.Byte[]

     MethodTable 0x00992c3c

     EEClass 0x00992bc4

     Size 4096(0x1000) bytes

     Array: Rank 1, Type System.Byte

     Element Type: System.Byte

     0:000>!gcroot 00a92000

     Scan Thread 0 (728)

     Scan Thread 1 (730)

     ESP:88cf548:Root:05066b48(System.IO.MemoryStream)->00a92000 (System.Byte[])

     ESP:88cf568:Root:05066b48(System.IO.MemoryStream)->00a92000 (System.Byte[])

     ...

     Scan HandleTable 9b130

     Scan HandleTable 9ff18

     HANDLE(Pinned):d41250:Root: 00a92000 (System.Byte[])

This shows that the object at address 00a92000 is a byte array, it's rooted by local variables in thread 1(to be precise, !GCRoot's output of roots in stack can't be trusted) and a pinned handle.

And the command "ObjSize" list all handles including pinned ones:

     0:000>!objsize

     ...

     HANDLE(Pinned):d41250: sizeof(00a92000) = 4096 ( 0x1000) bytes (System.Byte[])

     HANDLE(Pinned):d41254: sizeof(00a95000) = 4096 ( 0x1000) bytes (System.Byte[])

     HANDLE(Pinned):d41258: sizeof(00ac8b5b0) = 16 ( 0x10) bytes (System.Byte[])

     ...

Using those Sos commands, you could get a clear picture if the heap is fragmented and how. I believe Michael will have more blogs about details of Sos.

Solution

In Everett a lot of work is done in GC to recognize fragmentation caused by pinning and alleviate the situation, more work is already done in Whidbey. So hopefully, the problem won't show up in Whidbey. But besides change in the platform, user code could do something to avoid the issue too. From above analysis, we could tell:

1. If the pinned objects are allocated around same time, the free slots between each two objects would be smaller, and the situation is better.
2. If pinning happens on older objects, it could cause fewer problems. Because older objects live at bottom of heap but most of free space is generated on top of heap.
3. The shorter the objects are pinned, the easier GC could compact the heap

So if pinning becomes an issue which causes OOM for a .NET application, instead of creating new object to pin every time, developers could consider preallocating the to-be-pinned objects and reusing them. That way those objects would live close to each other in older part of GC heap and the heap won’t be fragmented that much. For example, if an application keeps pinning 1K buffers (consider the socket server case), we could use such a buffer pool to get the buffers: 

public class BufferPool

{

    private const int INITIAL_POOL_SIZE = 512; // initial size of the pool

    private const int BUFFER_SIZE = 1024; // size of the buffers

 

    // pool of buffers

    private Queue m_FreeBuffers;

     

    // singleton instance

    private static BufferPool m_Instance = new BufferPool ();

           

    // Singleton attribute

    public static BufferPool Instance

    {  

        get {

            return m_Instance;

        }

    }

 

    protected BufferPool()

    {

        m_FreeBuffers = new Queue (INITIAL_POOL_SIZE);

        for (int i = 0; i < INITIAL_POOL_SIZE; i++)

        {

            m_FreeBuffers.Enqueue (new byte[BUFFER_SIZE]);

        }

    }

 

    // check out a buffer

    public byte[] Checkout (uint size)

    {

        if (m_FreeBuffers.Count > 0)

        {

            lock (m_FreeBuffers)

            {

                if (m_FreeBuffers.Count > 0)                           

                    return (byte[])m_FreeBuffers.Dequeue ();

            }

        }

        // instead of creating new buffer,

        // blocking waiting or refusing request may be better

        return new byte [BUFFER_SIZE];

    }

 

    // check in a buffer

    public void Checkin (byte[] buffer)

    {

        lock (m_FreeBuffers)

        {

            m_FreeBuffers.Enqueue (buffer);

        }

    }

}

This posting is provided "AS IS" with no warranties, and confers no rights. Use of included samples are subject to the terms specified at http://www.microsoft.com/info/cpyright.htm"