Yun Jin's WebLog : Rotor code explanation

Thread, System.Threading.Thread, and !Threads (III)

yunjin — Tue, 30 Aug 2005 19:55:00 GMT

I got email asking me to explain !Threads output in details. I think this is a good question and a good topic for another installment to the series.

Here is an example I'll use for this post:

0:055> !threads
ThreadCount: 202
UnstartedThread: 95
BackgroundThread: 1
PendingThread: 0
DeadThread: 47
                                  PreEmptive   GC Alloc                     Lock
        ID ThreadOBJ       State     GC       Context           Domain     Count APT Exception
0 0xed0 0x0014f260   0x2000020 Enabled 0x00000000:0x00000000 0x00149aa0     1 Ukn
1 0xa3c 0x00157d28   0x2001220 Enabled 0x00000000:0x00000000 0x00149aa0     0 Ukn (Finalizer)
XXX      0 0x00166378      0x1820 Enabled 0x00000000:0x00000000 0x00149aa0     0 Ukn
4 0x12cc 0x00166540   0x2001020 Enabled 0x00000000:0x00000000 0x00149aa0     0 STA
5 0x12dc 0x00166708   0x2001020 Enabled 0x00000000:0x00000000 0x00149aa0     0 Ukn
3 0xe7c 0x00175b70   0x2001020 Enabled 0x00000000:0x00000000 0x00149aa0     0 Ukn
XXX      0 0x00175d38      0x1820 Enabled 0x00000000:0x00000000 0x00149aa0     0 Ukn
XXX      0 0x00175f00      0x1820 Enabled 0x00000000:0x00000000 0x00149aa0     0 Ukn
XXX      0 0x001760c8      0x1820 Enabled 0x00000000:0x00000000 0x00149aa0     0 Ukn
XXX      0 0x00176290      0x1820 Enabled 0x00000000:0x00000000 0x00149aa0     0 Ukn System.InvalidOperationException
XXX      0 0x00176458      0x1820 Enabled 0x00000000:0x00000000 0x00149aa0     0 Ukn
XXX      0 0x00176620      0x1820 Enabled 0x00000000:0x00000000 0x00149aa0     0 Ukn
XXX      0 0x001767e8      0x1820 Enabled 0x00000000:0x00000000 0x00149aa0     0 Ukn
12 0x7e0 0x001769b0   0x2001020 Enabled 0x00000000:0x00000000 0x00149aa0     0 Ukn
13 0x15e8 0x00178008   0x2001020 Enabled 0x00000000:0x00000000 0x00149aa0     0 Ukn
14 0x4d0 0x001781d0   0x2001020 Enabled 0x00000000:0x00000000 0x00149aa0     0 Ukn
XXX      0 0x00178398      0x1820 Enabled 0x00000000:0x00000000 0x00149aa0     0 Ukn
XXX      0 0x00178560      0x1820 Enabled 0x00000000:0x00000000 0x00149aa0     0 Ukn
XXX      0 0x00178728      0x1820 Enabled 0x00000000:0x00000000 0x00149aa0     0 Ukn
XXX      0 0x001788f0      0x1820 Enabled 0x00000000:0x00000000 0x00149aa0     0 Ukn
XXX      0 0x00178ab8      0x1820 Enabled 0x00000000:0x00000000 0x00149aa0     0 Ukn
XXX      0 0x00178c80      0x1820 Enabled 0x00000000:0x00000000 0x00149aa0     0 Ukn
XXX      0 0x00178e48      0x1820 Enabled 0x00000000:0x00000000 0x00149aa0     0 Ukn
21 0x14f0 0x00179010   0x2001020 Enabled 0x00000000:0x00000000 0x00149aa0     0 Ukn
22 0x1708 0x001791d8   0x2001020 Enabled 0x00000000:0x00000000 0x00149aa0     0 Ukn
23 0x11f8 0x001793a0   0x2001020 Enabled 0x00000000:0x00000000 0x00149aa0     0 Ukn
24 0x224 0x00179568   0x2001020 Enabled 0x00000000:0x00000000 0x00149aa0     0 Ukn
XXX      0 0x00179730      0x1820 Enabled 0x00000000:0x00000000 0x00149aa0     0 Ukn
XXX      0 0x001798f8      0x1820 Enabled 0x00000000:0x00000000 0x00149aa0     0 Ukn
XXX      0 0x00179ac0      0x1820 Enabled 0x00000000:0x00000000 0x00149aa0     0 Ukn System.InvalidOperationException
XXX      0 0x00179c88      0x1820 Enabled 0x00000000:0x00000000 0x00149aa0     0 Ukn System.InvalidOperationException
XXX      0 0x00179e50      0x1820 Enabled 0x00000000:0x00000000 0x00149aa0     0 Ukn
...

First !Threads gives some statistics about Thread Store.

ThreadCount: number of total C++ Thread objects in Thread Store.

UnstartedThread: number of C++ Thread objects marked as unstarted. Recall I mentioned in previous blog, if a user creates a C# Thread object, CLR will create an "unstarted" C++ Thread object. When Thread.Start is called on the C# object, CLR will create an OS thread and remove "unstarted" flag from the C++ Thread object.

BackgroundThread: number of C++ Threads (and the corresponding OS threads) considered as background. Being background simply means CLR won't wait the thread for shutting down. Threads created explicitly by using System.Threading.Thread.Start are by default foreground threads; whereas threads wandering into CLR from unmanaged world are by default background threads (Rotor: SetupThread in vm/threads.cpp calls SetBackground(TRUE)). However, whether a thread is background could be changed by using IsBackground property in C# Thread object.

PendingThreads: If an OS thread is created but its ThreadProc hasn't be executed to the place to decrement unstarted counter in Thread Store, the thread is considered to be pending. Number of this type of threads should be quite low.

DeadThreads: Number of C++ Thread objects whose OS threads are already dead but the C++ objects themselves are not deleted yet.

In Rotor, all the five numbers are actually stored in ThreadStore (vm/threads.h) object as its fields.

Then it comes a table of all C++ Thread objects in Thread Store. Let me explain each field.

The first column doesn't have a header. It is the OS thread ID given by debugger just for debugging readability. Because the numbers only exist in debugger process, not the debuggee process, you may see the number being different when you look at a live session than when you debug a dump taken from the same live session. For a "dead" or "unstarted" thread, this column is "XXX".

ID: this is the thread ID assigned by OS, it remains consistent during debugger sessions, but OS could recycle it.

ThreadOBJ: address of C++ Thread object. You could see contents of the object by "dt mscorwks!Thread <address>" if you have symbols for mscorwks.dll.

State: one of the most important fields of the table. For Rotor, it is the C++ Thread's m_State field. It is combination of bit masks to indicate what the status the Thread currently is. All possible states (bit masks) are defined as enum ThreadState in vm/Threads. We already covered several states like TS_Background, TS_Unstarted, and TS_Dead. More states include TS_AbortRequested (this thread is requested to be aborted), TS_AbortInitiated (abort process is already started for this thread), TS_GCSuspendPending (GC is trying to suspend this thread), and etc.

Preemptive GC: also very important. In Rotor, this is m_fPreemptiveGCDisabled field of C++ Thread class. It indicates what GC mode the thread is in: "enabled" in the table means the thread is in preemptive mode where GC could preempt this thread at any time; "disabled" means the thread is in cooperative mode where GC has to wait the thread to give up its current work (the work is related to GC objects so it can't allow GC to move the objects around). When the thread is executing managed code (the current IP is in managed code), it is always in cooperative mode; when the thread is in Execution Engine (unmanaged code), EE code could choose to stay in either mode and could switch mode at any time; when a thread are outside of CLR (e.g, calling into native code using interop), it is always in preemptive mode.

GC Alloc context: allocate context GC might use when it tries to allocate object for this thread. In Rotor, it is m_alloc_context in C++ Thread object.

Domain: which AppDomain the thread is currently in (Rotor: m_pDomain field of C++ Thread class). You could use !DumpDomain or "dt mscorwks!AppDomain" to dump details of the domain. A thread can only be in one domain at a time, but it could switch into different domains. Speical marks will be put on thread's stack to when it transit to another domain.

Lock count: how many locks this thread has taken (Rotor: m_dwLockCount field of C++ Thread class). The locks it tracks include the managed monitors (taken by lock(obj) in C#), BCL's ReaderWriterLock, and certain locks inside CLR's unmanaged code.

APT: COM apartment for the thread, whether the thread is in a single-threaded apartment (STA), multithreaded apartment(MTA) or unknown.

Exception: the last managed exception thrown from this thread. It is saved in a GC handle in the C++ Thread object (Rotor: m_LastThrownObjectHandle).

The last column also indicates which special thread this thread is. However, !Threads only recognize a limited type of special threads for this field, including Finalizer thread, GC thread, Threadpool Worker thread, and Threadpool Completion Port thread. And for special threads which doesn't have a C++ Thread object (a special thread doesn't need to run managed code like debugger helper thread and server GC thread), they can not be displayed here. In Whidbey, a "-special" option is added to !Threads command which will show all special threads in the process as a separate list. Here is a sample output:

0:007> !threads -special
ThreadCount: 4
UnstartedThread: 0
BackgroundThread: 3
PendingThread: 0
DeadThread: 0
Hosted Runtime: no
                                      PreEmptive   GC Alloc           Lock
       ID OSID ThreadOBJ    State     GC       Context       Domain   Count APT Exception
   0    1 828 0029a030      a020 Disabled 06907c38:069081d4 0f59e038     2 MTA
   4    2 16fc 0029e980      b220 Enabled 0690424c:069061d4 0021f4a8     0 MTA (Finalizer)
   5    3 1c1c 002e71e8      1220 Enabled 028f20f8:028f3f94 0021f4a8     0 Ukn
   6    4 1244 0f6fa778    80a220 Enabled 00000000:00000000 0021f4a8     0 MTA (Threadpool Completion Port)

       OSID     Special thread type
    1    e20    DbgHelper
    2   1e1c    GC
    3   1ed4    GC
    4   16fc    Finalizer
    5   1c1c    ADUnloadHelper
    6   1244    Timer

This posting is provided "AS IS" with no warranties, and confers no rights.

Thread, System.Threading.Thread, and !Threads (II)

yunjin — Mon, 29 Aug 2005 19:30:00 GMT

With knowledge in my previous blog, we could avoid some mistakes in .NET programming.

A C++ Thread is very resource heavy. It is associated with a lot of dynamically allocated memory and some OS handles. So it had better to be cleaned up ASAP after its corresponding OS thread dies. C++ Thread class has a reference count. For its object to be deleted, the ref count has to be dropped to 0 (Rotor: Thread::DecExternalCount in vm\threads.cpp). One interesting point is that the C# Thread object actually keeps a reference to its associated C++ Thread, so a live C# Thread object could keep its C++ Thread from being deleted even if the OS thread is already dead. (On the other hand, C++ Thread also has a reference to C# Thread, but it will break the circle when its own ref count drops to 1). Because C# Thread is a managed object, its lifetime is mostly determined by users. Plus, C# Thread class has a finalizer, so its lifetime will be extended at least one GC. So if user code caches the C# Thread objects or have some ill-behaved finalizers (in another blog entry, I mentioned wrong-doing finalizer on one object could prevent all other object's fianlizer from running), "dead" C++ Thread objects may accumulate over time and some "memory leak" will be observed.

I have an example here to demo the problem and how to debug it using windbg + SOS. In this process, there are 202 C++ Thread objects. Among which 160 are "dead", meaning their associated OS threads are dead. Number of total threads in Thread Store and dead/unstarted threads are showed in "!threads" output. For a "live" thread, OS and debugger thread ID are printed out for the entry, for a "dead" thread, "XXX" is marked at beginning of the line:

0:043> !threads
ThreadCount: 202
UnstartedThread: 0
BackgroundThread: 1
PendingThread: 0
DeadThread: 160
                                  PreEmptive   GC Alloc                     Lock
        ID ThreadOBJ       State     GC       Context           Domain     Count APT Exception
0 0x1138 0x0015a298        0x20 Enabled 0x00000000:0x00000000 0x00149ac8     1 Ukn
1 0x1148 0x00152530      0x1220 Enabled 0x00000000:0x00000000 0x00149ac8     0 Ukn (Finalizer)
3 0x114c 0x00177548   0x2001020 Enabled 0x00000000:0x00000000 0x00149ac8     0 Ukn
4 0x1150 0x00177878   0x2001020 Enabled 0x00000000:0x00000000 0x00149ac8     0 Ukn
5 0x1154 0x00177c08   0x2001020 Enabled 0x00000000:0x00000000 0x00149ac8     0 Ukn
…
42 0x11e4 0x00180460   0x2001020 Enabled 0x00000000:0x00000000 0x00149ac8     0 Ukn
22 0x11e8 0x00180838   0x2001020 Enabled 0x00000000:0x00000000 0x00149ac8     0 Ukn
XXX      0 0x00180c10      0x1820 Enabled 0x00000000:0x00000000 0x00149ac8     0 Ukn
XXX      0 0x00180fe8      0x1820 Enabled 0x00000000:0x00000000 0x00149ac8     0 Ukn
XXX      0 0x001813c0      0x1820 Enabled 0x00000000:0x00000000 0x00149ac8     0 Ukn
XXX      0 0x00181750      0x1820 Enabled 0x00000000:0x00000000 0x00149ac8     0 Ukn
XXX      0 0x00181b28      0x1820 Enabled 0x00000000:0x00000000 0x00149ac8     0 Ukn
XXX      0 0x00181f00      0x1820 Enabled 0x00000000:0x00000000 0x00149ac8     0 Ukn
XXX      0 0x001822d8      0x1820 Enabled 0x00000000:0x00000000 0x00149ac8     0 Ukn
… //continue with a huge list

Now I want to find out why all the "dead" C++ Thread objects are still around. First I could check its ref count if I have symbols for mscorwks.

//0x00180c10 is a dead ThreadOBJ I picked from !Threads output
0:043> dt mscorwks!Thread 0x00180c10 m_ExternalRefCount
+0x0cc m_ExternalRefCount : 1

Since the ref count is 1, if this C++ Thread object has a C# Thread object associated with it, the C# object must be the last reference. I could verify if that is the case by checking the C++ object's m_ExposedObject field. It is a weak GC handle (a unmovable pointer to GC reference which doesn't counted as root of the GC object), so dereference it will get the managed object. As mentioned before, C++ Thread object also has a strong handle (m_StrongHndToExposedObject field) to the C# object, but it already cleared the strong handle when ref count drops to 1 to avoid circular reference.

0:043> dt mscorwks!Thread 0x00180c10 m_ExposedObject
   +0x0c0 m_ExposedObject : 0x00a71054

0:043> dp 0x00a71054 l1
00a71054 00c5c714

0:043> !do 00c5c714
Name: System.Threading.Thread
MethodTable 0x79bb8384
EEClass 0x79bb85b0
Size 60(0x3c) bytes
GC Generation: 0
mdToken: 0x020000eb (c:\windows\microsoft.net\framework\v1.1.4322\mscorlib.dll)
FieldDesc*: 0x79bb8614
        MT      Field     Offset                 Type       Attr      Value Name
0x79bb8384 0x4000330      0x4                CLASS   instance 0x00000000 m_Context
0x79bb8384 0x4000331      0x8                CLASS   instance 0x00000000 m_LogicalCallContext
0x79bb8384 0x4000332      0xc                CLASS   instance 0x00000000 m_IllogicalCallContext
0x79bb8384 0x4000333     0x10                CLASS   instance 0x00000000 m_Name
0x79bb8384 0x4000334     0x14                CLASS   instance 0x00000000 m_ExceptionStateInfo
0x79bb8384 0x4000335     0x18                CLASS   instance 0x00000000 m_Delegate
0x79bb8384 0x4000336     0x1c                CLASS   instance 0x00000000 m_PrincipalSlot
0x79bb8384 0x4000337     0x20                CLASS   instance 0x00000000 m_ThreadStatics
0x79bb8384 0x4000338     0x24                CLASS   instance 0x00000000 m_ThreadStaticsBits
0x79bb8384 0x4000339     0x28                CLASS   instance 0x00000000 m_CurrentCulture
0x79bb8384 0x400033a     0x2c                CLASS   instance 0x00000000 m_CurrentUICulture
0x79bb8384 0x400033b     0x30         System.Int32   instance 2 m_Priority
0x79bb8384 0x400033c     0x34         System.Int32   instance 1575952 DONT_USE_InternalThread
0x79bb8384 0x400033d        0                CLASS     shared   static m_LocalDataStoreMgr
    >> Domain:Value 0x00149ac8:0x00c05338 <<

Then I want to check root of the C# Thread object to see who keeps it alive:

0:043> !gcroot 00c5c714
Scan Thread 0 (0x1138)
ESP:12f69c:Root:0xc5b3f4(System.Object[])->0xc5c714(System.Threading.Thread)
…

So there is an array who keeps a reference to a "dead" C# Thread. This looks interesting. I could check all other C# Thread objects in the process using !DumpHeap command. !DumpHeap could dump objects in GC heap for a particular type specified by "-type" option:

0:043> !DumpHeap -type System.Threading.Thread
   Address         MT     Size Gen
0x00c054c8 0x79bb8384       60    1 System.Threading.Thread
0x00c5b730 0x79bc81d4       28    2 System.Threading.ThreadStart
0x00c5b74c 0x79bb8384       60    2 System.Threading.Thread
0x00c5b7bc 0x79bc81d4       28    2 System.Threading.ThreadStart
0x00c5b7d8 0x79bb8384       60    2 System.Threading.Thread
0x00c5b820 0x79bc81d4       28    2 System.Threading.ThreadStart
0x00c5b83c 0x79bb8384       60    2 System.Threading.Thread
0x00c5b884 0x79bc81d4       28    2 System.Threading.ThreadStart
0x00c5b8a0 0x79bb8384       60    2 System.Threading.Thread
0x00c5b8e8 0x79bc81d4       28    2 System.Threading.ThreadStart
0x00c5b904 0x79bb8384       60    2 System.Threading.Thread
0x00c5b94c 0x79bc81d4       28    2 System.Threading.ThreadStart
0x00c5b968 0x79bb8384       60    2 System.Threading.Thread
0x00c5b9b0 0x79bc81d4       28    2 System.Threading.ThreadStart
…//long list
total 401 objects

Because !DumpHeap match type by string, so it also dumps ThreadStart objects. Because every C# Thread object created by user code always has a ThreadStart object (but C# Thread created by System.Thread.CurrentThread may not have a ThreadStart), so they show up as a pair. among 401 such objects, 200 are C# Thread objects, roughly match the number of C++ Thread objects (the number doesn't have to be the same because not every C++ Thread object has a C# counterpart created). Generation for most of C# Thread objects are 2, meaning they already survive at least 2 GCs. When I track roots of those C# Thread objects, they all point to the array. In this case, we need to look closely to the source to see whether it is necessary to cache all the C# Thread objects in an array.

Another related topic is that CLR relies on DLL_THREAD_DETACH notification to mscorwks.dll's DllMain (Rotor: EEDllMain in vm\ceemain.cpp) to know an OS thread is dead, thus detach the related C++ Thread. Using TerminateThread API is already notoriously bad in unmanaged programming, here we see another reason not to call it in managed code: if TerminateThread is called on a managed OS thread, among other bad effect (e.g. back out code not executed), CLR will not get thread detach notification. Because C++ Thread object has references to OS thread's stack address, failing to detach it from the OS thread will cause crash at random place.

This posting is provided "AS IS" with no warranties, and confers no rights.

Thread, System.Threading.Thread, and !Threads (I)

yunjin — Thu, 25 Aug 2005 20:26:00 GMT

If you use SOS’s !Threads command during debugging a lot, you should be familiar with such output:

0:003> !threads
PDB symbol for mscorwks.dll not loaded
Loaded Son of Strike data table version 5 from "C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\mscorwks.dll"
ThreadCount: 12
UnstartedThread: 5
BackgroundThread: 1
PendingThread: 0
DeadThread: 5
                                  PreEmptive   GC Alloc                     Lock
        ID ThreadOBJ       State     GC       Context           Domain     Count APT Exception
0 0xb74 0x0014f230        0x20 Enabled 0x00000000:0x00000000 x00149aa8     1 Ukn
2 0xb58 0x00157cf8      0x1220 Enabled 0x00000000:0x00000000 x00149aa8     0 Ukn (Finalizer)
XXX      0 0x001665f0      0x1820 Enabled 0x00000000:0x00000000 x00149aa8     0 Ukn
XXX      0 0x0016d348      0x1400 Enabled 0x00000000:0x00000000 x00149aa8     0 Ukn
XXX      0 0x0016d510      0x1820 Enabled 0x00000000:0x00000000 x00149aa8     0 Ukn
XXX      0 0x0016d9d0      0x1400 Enabled 0x00000000:0x00000000 x00149aa8     0 Ukn
XXX      0 0x0016db98      0x1820 Enabled 0x00000000:0x00000000 x00149aa8     0 Ukn
XXX      0 0x0016e248      0x1400 Enabled 0x00000000:0x00000000 x00149aa8     0 Ukn
XXX      0 0x0016e410      0x1820 Enabled 0x00000000:0x00000000 x00149aa8     0 Ukn
XXX      0 0x0016e740      0x1400 Enabled 0x00000000:0x00000000 x00149aa8     0 Ukn
XXX      0 0x0016e908      0x1820 Enabled 0x00000000:0x00000000 x00149aa8     0 Ukn
XXX      0 0x0016ec98      0x1400 Enabled 0x00000000:0x00000000 x00149aa8     0 Ukn

Have you ever wondered what exactly are in the list? Why the number of threads listed here doesn’t match the number of "real" threads in the process (in the example above, I only have 4 threads in the process, but !Threads shows 12)? Why some "real" threads have entries here, some don't? Maybe they are managed System.Threading.Thread objects (but the number in the list might not match number of Thread objects either)? Answers to those questions are tied to how CLR implements threads and manages thread information.

CLR provides classes in System.Threading namespace as threading APIs. As you know, threading is implemented by utilizing native threads in underlying OS (Windows, in CLR case). CLR just piggybacks managed threads to native OS threads. Users use BCL System.Threading.Thread objects (I’ll call it as C# Thread below) to control managed threads just as thread HANDLE is used to control to windows native threads. If you ever checked contents of a C# Thread object, you will find it is quite small (here I use SOS !DumpObj command, you could also see it using Visual Studio or other managed debuggers):

0:003> !DumpObj 0x00c03248
Name: System.Threading.Thread
MethodTable 0x79bb8384
EEClass 0x79bb85b0
Size 60(0x3c) bytes
GC Generation: 0
mdToken: 0x020000eb (c:\windows\microsoft.net\framework\v1.1.4322\mscorlib.dll)
FieldDesc*: 0x79bb8614
        MT      Field     Offset                 Type       Attr      Value Name
0x79bb8384 0x4000330      0x4                CLASS   instance 0x00000000 m_Context
0x79bb8384 0x4000331      0x8                CLASS   instance 0x00000000 m_LogicalCallContext
0x79bb8384 0x4000332      0xc                CLASS   instance 0x00000000 m_IllogicalCallContext
0x79bb8384 0x4000333     0x10                CLASS   instance 0x00000000 m_Name
0x79bb8384 0x4000334     0x14                CLASS   instance 0x00000000 m_ExceptionStateInfo
0x79bb8384 0x4000335     0x18                CLASS   instance 0x00000000 m_Delegate
0x79bb8384 0x4000336     0x1c                CLASS   instance 0x00000000 m_PrincipalSlot
0x79bb8384 0x4000337     0x20                CLASS   instance 0x00000000 m_ThreadStatics
0x79bb8384 0x4000338     0x24                CLASS   instance 0x00000000 m_ThreadStaticsBits
0x79bb8384 0x4000339     0x28                CLASS   instance 0x00000000 m_CurrentCulture
0x79bb8384 0x400033a     0x2c                CLASS   instance 0x00000000 m_CurrentUICulture
0x79bb8384 0x400033b     0x30         System.Int32   instance 2 m_Priority
0x79bb8384 0x400033c     0x34         System.Int32   instance 1498008 DONT_USE_InternalThread
0x79bb8384 0x400033d        0                CLASS     shared   static m_LocalDataStoreMgr

CLR actually needs much more information about a thread than fields of the C# Thread class, and such information is needed in CLR's unmanaged part where it's not easy to access managed objects. So inside Execution Engine (so far Execution Engine is still written in unmanaged code), there is an unmanaged C++ class also called Thread (let me call it C++ Thread) to keep all information for an OS native thread (OS thread). In Rotor, this class is defined in vm\threads.h.

CLR needs to create a C++ Thread object for every OS thread EE knows of, that is, every thread which ever ran managed code. Such threads could either be (a) created explicitly by users using C# Thread.Start, (b) an unmanaged OS thread who has ever visited managed world, e.g, through interop, or (c) a special OS thread in CLR which might run managed code. For case a, when a user creates a C# Thread object, CLR will create a C++ Thread object, link it to the C# Thread object, and mark it as unstarted (Rotor: SetupUnstartedThread in vm\threads.cpp). Once Start method is called on the C# Thread object, CLR will create an OS thread, in the OS thread’s ThreadPproc (Rotor: ThreadNative::KickOffThread in vm\ComSynchronizable.cpp), CLR will save the C++ Thread object’s address to the OS thread's TLS (Thread Local Storage) and mark it to be started (Rotor: ThreadStore::TransferStartedThread in vm\threads.cpp); For case b, at entry point for an unmanaged OS thread to managed world, CLR will create a C++ Thread object and also use TLS to associate it with the OS thread (Rotor: SetupThread in vm\threads.cpp); Case c is similar to case b, except CLR might set up C++ Thread earlier.

In any case, the C++ Thread object is the primary place for CLR to store information regarding to a managed thread. When CLR needs to access the information, it will just fetch the object from the OS thread's TLS. In that sense, the C# Thread is more like a managed proxy for the C++ Thread. In fact, for case b and c, there will be no C# Thread objects created unless users call System.Threading.Thread.CurrentThread. C++ Thread tracks its corresponding C# Thread using a GCHandle (m_ExposedObject field in C++ Thread object); C# Thread tracks its C++ Thread using a native pointer(DONT_USE_InternalThread field in C# Thread). You could verify this circular reference in debugger if you have symbols for mscorwks.dll:

0:003> !do 0x00c03248
Name: System.Threading.Thread
…
MT Field Offset Type Attr Value
…
0x79bb8384 0x400033c 0x34 System.Int32 instance 1498008 DONT_USE_InternalThread
…

0:013> dt mscorwks!Thread 0n1498008
…
+0x0c0 m_ExposedObject : 0x00a710e4
…

0:013> dp 0x00a710e4 l1
00a710e4 00c03248

CLR uses a data structure called Thread Store (Rotor: ThreadStore in vm\threads.h) to keep all C++ Threads. What you see in output of !Threads command is actually list of C++ Threads in Thread Store. The “ThreadOBJ” field is address of a C++ Thread object. Other fields are important information about the C++ Thread, including OS thread ID for the corresponding OS thread. Here you could see not every live OS thread has a C++ Thread, which could be explained by the fact that CLR only creates a C++ Thread for an OS thread which ever runs managed code. Meanwhile you may also find some C++ Threads don't have an OS thread associated with them (the ID fields are “XXX”). They are either (a)unstarted, (b) failed to started, (c)used to represent an live OS thread which is now dead. For case (a), CLR will wait until C# Thread.Start is called and create an OS thread for it then; for (b) and (c), the C++ Threads object will be deleted some time later.

This posting is provided "AS IS" with no warranties, and confers no rights.

Special threads in CLR

yunjin — Tue, 05 Jul 2005 19:52:00 GMT

Question: How many threads does a typical managed process have when it just starts to run?

Answer: regardless how many threads the user creates, there are at least 3 threads for a common managed process after CLR starts up: a main thread which starts CLR and run user's Main method, CLR debugger helper thread which provides debugging service for interop debuggers like Visual Studio, and the finalizer thread which runs finalizers for unreachable objects. Depends on what the program does, CLR might create more threads to perform special tasks.

Sometimes it is important to know what "special" threads would be created in CLR so we could understand better the implicit impact of our managed programs. Here is a list of most common special threads:

1. Finalizer thread. The thread is to run finalizers for "dead" objects. This thread is created when GC heap is initialized during EE start up. In Rotor, the thread proc for the thread is GCHeap::FinalizerThreadStart in vm\gcee.cpp. Because GC is undeterministic and finalizers are executed in a separate thread, you can't predict when exactly an object will be finalized. Because there is only one thread to run all finalizers, if one finalizer is blocked, no other finalizers could run. So it is discouraged to take any lock in finalizer. Also see Maoni Stephens's blog for details about finalizer thread.

2. Debugger helper thread. As its name suggests, this thread helps debuggers (mixed mode and managed debugger, but not pure unmanaged debug like windbg) to get information of the managed process and to execute certain debugging operations. The thread is created when EE initializes debugger during start up. In Rotor, the thread proc for this thread is DebuggerRCThread::ThreadProcStatic (debug\ee\Rcthread.cpp). Also see Mike Stall's blog about impact of this helper thread。

3. Concurrent GC thread (doesn't exist in Rotor). As explained in Maoni and Chris Lyon's blog, concurrent GC is a special GC mode which allows garbage to be collected while managed threads are running simultaneously. To achieve this goal, CLR creates a thread to perform GC concurrently with user threads. The thread is only created when CLR decides to do a concurrent GC (even when concurrent GC mode is on, not every GC is concurrent, read Maoni's blog for details) and will be recycled when there are no concurrent GC work to do.

4. Server GC threads (doesn't exist in Rotor). Maoni and Chris also explained Server GC mode where on multi-process machine CLR creates one GC heap for each CPU and one thread to do GC for each heap. When Server GC mode is enabled, server GC threads will be created at EE start up time when GC heaps are initialized.

5. App Domain unload helper thread. In CLR V1.X, when a thread requests to unload an App Domain and the thread is in that App Domain itself, it needs to create a worker thread to do the unloading work. The worker thread will be dead once the target AD is unloaded. In Rotor, the thread starts with UnloadThreadWorker.ThreadStart (bcl\system\Appdomain.cs). In Whidbey, all AD unload work is performed in a special thread regardless whether the requesting thread is in the unloading domain. The helper thread is created when first non-default App Domain is created (default domain is never unloaded) and will stay alive since then. Also see Chris Brumme's blog about details of AD unload.

6. Threadpool threads. Depends on how a program use CLR threadpool, CLR might create threads of a varieties of types. There is only one thread for some thread type. For other types, number of threads is related to number of CPUs, the work load, and some user configurable settings. The thread types including wait threads (threads to perform asynchronized wait, could be more than one); worker threads (threads to execute user work item, could be more than one); Completion port threads (threads wait for completion port IO in Windows, could be more than one, doesn't exist in Rotor); Gate thread (thread help to monitor status of completion port threads and worker threads, only one); Timer thread (thread manages timer queue, only one).

Thread safety of Timer callbacks

yunjin — Sun, 08 May 2005 11:12:00 GMT

I didn't realize I've stopped blogging for 1 year. What a shame! Fortunately I didn’t waste the time: we ship Whidbey Beta1 and Beta2 in the past year! Now with Beta2 out of door, I have more spare time for blogging. :)

Today I want to talk about some interesting facts about Timer in CLR. There is an example for how to use timer in MSDN: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/cpref/html/frlrfsystemthreadingtimerclasstopic.asp

This sample starts a timer and does certain things when the timer fires for certain times, like killing the timer. However, this sample has a bug which will cause trouble in stress scenario. To demonstrate the problem, I made a little change to the code:

using System;
using System.Threading;

class TimerExample
{
static void Main()
{
  AutoResetEvent autoEvent     = new AutoResetEvent(false);
  StatusChecker statusChecker = new StatusChecker(100);

  // Create the delegate that invokes methods for the timer.
  TimerCallback timerDelegate =
   new TimerCallback(statusChecker.CheckStatus);

  Console.WriteLine("{0} Creating timer.\n",
   DateTime.Now.ToString("h:mm:ss.fff"));
  Timer stateTimer =
   new Timer(timerDelegate, autoEvent, 0, 10);

  // start another thread to post work items to thread pool
  Thread t = new Thread (new ThreadStart (PostWortItem));
  t.Start ();

  // When autoEvent signals, dispose of
  // the timer.
  autoEvent.WaitOne();
  stateTimer.Dispose();
  Console.WriteLine("\nDestroying timer.");
}

// a Thread proc which keeps posting work items to thread pool
static void PostWortItem ()
{
  // Post some user work items to thread pool
  for (int i = 0; i < 1000; i++)
  {
   ThreadPool.QueueUserWorkItem (new WaitCallback (WorkItem));
   Thread.Sleep (10);
  }
}

// An nop work item for thread pool
static void WorkItem (object o)
{
  Thread.Sleep (500);
}
}

class StatusChecker
{
int invokeCount, maxCount;

public StatusChecker(int count)
{
  invokeCount = 0;
  maxCount = count;
}

// This method is called by the timer delegate.
public void CheckStatus(Object stateInfo)
{
  Console.WriteLine("Checking status " + (++invokeCount));

  if(invokeCount == maxCount)
  {
   //signal Main.
   AutoResetEvent autoEvent = (AutoResetEvent)stateInfo;
   autoEvent.Set();
  }
}
}

Basically I added another thread to keep posting work items to threadpool, but the rest part is still expected to behave the same: when the timer fires the 100^th time, it should set an event so the main thread would stop the timer.

In one of 5 runs in my machine, I got such output:

5:48:07.625 Creating timer.

Checking status 1
Checking status 2
Checking status 3
Checking status 4
…
Checking Status 93
Checking Status 94
Checking Status 95
Checking Status 96
Checking Status 97
Checking Status 98
Checking Status 102
Checking Status 99
Checking Status 103
Checking Status 104
Checking Status 105
…
Checking Status 698
Checking Status 700
Checking Status 701
Checking Status 703
Checking Status 703
Checking Status 704
Checking Status 705
…

^C

It seems that invokeCount never hits 100 thus the program doesn't stop and some other sequence in the output looks to be out of order.

How does this happen? First we need to understand how timer is implemented in CLR, who is executing the timer callbacks?

One simple idea would be putting all timers in a queue and having a dedicate thread doing something like this (pseudo code):

while (true)
{
    foreach (Timer t in timer queue)
    {
        if (t.TimeToFire ())
        {
            t.InvokeCallback ();
        }
    }
    Sleep(MinumInterval);
}

However with this logic one lengthy timer callback would block all other timers. In CLR, we do have a timer queue and a dedicate timer thread. However the only job of timer thread is to maintain the timer queue, when a timer needs to fire, timer thread queue a work item to threadpool, then a thread pool's worker thread will pick up the work item and invoke the timer callback.

In Rotor's source, the timer thread's logic is in vm/Win32threadpool.cpp, the thread proc is ThreadpoolMgr::TimerThreadStart and ThreadpoolMgr::FireTimers does most of interesting work. The pseudo code looks like:

while (true)
{
    foreach (Timer t in timer queue)
   {
       if (t.TimeToFire ())
      {
          // put a work item to thread pool
          // to call timer cal back on t once
          WorkItem work = CallTimerCallbackOnce (t);
          ThreadPool.QueueWorkItem (work);
      }
   }
   //MinumInterval is minum of next firing interval
   // for all timers in the queue
   Sleep(MinumInterval);
}

The timer thread only guarantees to put timer callback requests to a queue in thread pool (ThreadpoolMgr::QueueUserWorkItem) in order of timer firing. But timer callbacks are not called in a serialized way. If a timer fires twice and there are more than one worker thread in thread pool, there's no guarantee that the first callback will be finished before the next callback starts. Therefore, it's not thread safe for timer callbacks to access shared data without locking. That's why the example in MSDN breaks: when CheckStatus is executed in multiple threads, it's possible that "if(invokeCount == maxCount)" will never be satisfied. Changing the code to this would make it more robust:

public void CheckStatus(Object stateInfo)
{
  int count = Interlocked.Increment (ref invokeCount);
  Console.WriteLine("Checking status " + count);

  if(count == maxCount)
  …

Another interesting thing about timer implementation is that when a client thread creates a new timer, it doesn't insert the timer to timer queue directly. Instead, it queues a user APC to the timer thread (see ThreadpoolMgr::CreateTimerQueueTimer and InsertNewTimer). Similar thing is done for updating (ThreadpoolMgr::ChangeTimerQueueTimer and UpdateTimer) and deleting timer (ThreadpoolMgr:: DeleteTimerQueueTimer and DeregisterTimer). That way, client threads don't need to synchronize to access the shared timer queue. After all, the timer thread is sleeping (alertable) for most of time.

PS: to make the race happen more easily, I did more tweaks to the MSDN sample than the threadpool workitems, it should be obvious to you. ;)

This posting is provided "AS IS" with no warranties, and confers no rights.

FCall and GC hole - first post about Rotor

yunjin — Mon, 09 Feb 2004 09:34:00 GMT

An exsample of FCall

My friend Joel Pobar had a great post to demo how to add new code to Rotor which exposes more EE(Execution Engine) internal information to managed world. This is a very good example covers both BCL and EE, and how the two parts interact with each other. As showed in this example, BCL code could call into EE by a special type of method called “FCall“, like this:

FCIMPL1(MethodBody *, COMMember::GetMethodBody, MethodDesc **ppMethod)
{
   MethodDesc* pMethod = *ppMethod;
   METHODBODYREF MethodBodyObj = NULL;

   HELPER_METHOD_FRAME_BEGIN_RET_0();
   GCPROTECT_BEGIN(MethodBodyObj);

   TypeHandle thMethodBody(g_Mscorlib.FetchClass(CLASS__METHOD_BODY));
   MethodBodyObj = (METHODBODYREF)AllocateObject(thMethodBody.GetMethodTable());
   Module* pModule = pMethod->GetModule();

   COR_ILMETHOD_DECODER MethodILHeader(pMethod->GetILHeader(), pModule->GetMDImport(), TRUE);
   MethodBodyObj->maxStackSize = MethodILHeader.GetMaxStack();

   GCPROTECT_END();
   HELPER_METHOD_POLL();
   HELPER_METHOD_FRAME_END();

   return (MethodBody*)OBJECTREFToObject(MethodBodyObj);
}
FCIMPLEND

As I said before I'd try to explain some Rotor code in my blog, so let me start by analyzing a small problem in that piece of code. I won't call it a bug because it happens to be harmless here. But it does violate some CLR coding rules.

Some basic bricks of an FCall

First let's take a glance of those amazing macros:

FCIMPL1/FCIMPLEND: defined in vm/fcall.h. They just tweak calling convention between managed code (BCL) and unmanaged code (EE). You can see different flavors of FCIMPL, which serve calls with different argument numbers or types (to match the argument passing and enregistering rules in managed world).
HELPER_METHOD_FRAME_BEGIN_RET_0/HELPER_METHOD_FRAME_END: defined in vm/fcall.h. For operations like GC (Garbage Collection) and EH (Exception Handling) to work correctly, some frames have to be set up in the stack, especially at the boundary between managed part and unmanaged part. Frames are a topic could take a blog entry itself, so I won't cover it too much here. What you need to know now is that for performance reason, an FCall doesn't set up a frame by default. So when an FCall wants to throw an exception or allow a GC to happen, it has to set up a HelperMethodFrame (vm/frames.h) first. This job is done by HELPER_METHOD_FRAME_BEGIN* macro.HELPER_METHOD_FRAME_END is to tear down the frame from stack. All GC or exception throwing has to happen in the range guarded by this frame. In the code above, some operations could trigger a GC (at least AllocateObject could do so, not sure about others. It would be a hard job to trace into each code path to find out whether a GC could happen), so a HelperMethodFrame has to be established before that call.
GCPROTECT_BEGIN/GCPROTECT_END: defined in vm/frames.h. Similar to HELPER_METHOD_FRAME_BEGIN*, GCPROTECT_BEGIN is used to set up a GCFrame (vm/frames.h) and GCPROTECT_END pop the frame out. When a GC happens, it needs to find out all object references in stack to trace which objects in the managed heap are still alive, and when it moves the object (to compact the heap) it needs to update the references in stack with the new location of the objects. For managed code, JIT generates all information needed by GC. But for unmanaged part of CLR, the code whoever has references to managed objects is responsible to report all references itself. A GCFrame serves for this purpose. If an unmanaged method pushes a GCFrame to stack, the frame will report the protected reference (the argument to GCPROTECT_BEGIN) during GC. In our example, MethodBodyObj is an object reference so we set up a GCFrame for it.
HELPER_METHOD_POLL: defined in vm/fcall.h. This macro is meant to do a GC poll in range of a HelperMethodFrame. GC poll is another complicated thing I don't want to talk here. Basically it allows GC to happen in another thread, without a poll another thread that wants to perform a GC might be blocked, thus all managed threads in the application will be blocked.
OBJECTREFToObject: defined in vm/vars.hpp. It's used to take a pure object pointer out from an ObjectRef. An ObjectRef is a naked pointer in free build, but a wrapper with some very useful checking in debug build.

The problem

1. GC hole. In COMMember::GetMethodBody, a HELPER_METHOD_POLL is put after GCPROTECT_END. As I explained above, GCPROTECT_END will pop up the GC frame which is protecting the object reference, but HELPER_METHOD_POLL allows a GC to happen in another thread. So there are chances (although very small) that after the GC frame is popped up, another thread performs a GC. In such a GC, MethodBodyObj won't be reported. So GC might not know the object referenced by MethodBodyObj is still alive (if there's no other reference to the object) and collect it; or (if there are other references) GC might move the object but not update MethodBodyObj with the new address thus MethodBodyObj would hold a “stale” object pointer. Either case, COMMember::GetMethodBody will return a bogus object and the program might crash later in an unexpected way. We call this kind of errors “GC holes” in CLR. They are hard to detect because GC is non-deterministic. The funny thing is that nothing bad would happen in this method because HELPER_METHOD_POLL is actually defined as a no-op in this version:

// This is the fastest way to do a GC poll if you have already erected a HelperMethodFrame
// #define HELPER_METHOD_POLL() { __helperframe.Poll(); INDEBUG(__fCallCheck.SetDidPoll()); }

#define HELPER_METHOD_POLL() { }

I don't know why we use an empty macro for HELPER_METHOD_POLL but I'm sure it's supposed to be the version which is commented out. In later versions we may uncomment the above line to make HELPER_METHOD_POLL take effect. So although this FCall doesn't cause any trouble for now, it might later. The corrected version should be:

HELPER_METHOD_FRAME_BEGIN_RET_0();
GCPROTECT_BEGIN(MethodBodyObj);

...

HELPER_METHOD_POLL();
GCPROTECT_END();
HELPER_METHOD_FRAME_END();

2. If you dig deep into the code of HELPER_METHOD_FRAME_BEGIN*, you will find that those macros do a GC poll themselves. So unless the FCall does some very time consuming work, there's no need for another poll. Thus a refined version of our sample would be:

HELPER_METHOD_FRAME_BEGIN_RET_0();
GCPROTECT_BEGIN(MethodBodyObj);

...

GCPROTECT_END();
HELPER_METHOD_FRAME_END();

3. Because setting up HelperMethodFrame usually means the code wants to allow GC, for convenience we have versions of HELPER_METHOD_FRAME_BEGIN* to protect object references. Then a GCFrame is not needed. So the FCall could be written this way:

HELPER_METHOD_FRAME_BEGIN_RET_1(MethodBodyObj);

...

HELPER_METHOD_FRAME_END();

I hope you already got a taste how FCall works in CLR by reading this blog. Actually most thing I talked here can be found in comments at the beginning of vm/fcall.h. And if you want to see more examples of FCall, just search FCIMPL in vm directory, you will get plenty of them. Then you will see how CLR build a beautiful object-oriented world by the old fashion and kinda dirty way.

This posting is provided "AS IS" with no warranties, and confers no rights.