@TessFerrandez
I had an interesting case today where an asp.net app was stuck in a true deadlock. In other words two threads were both waiting for resources that the other thread owned.
The scenario in these cases usually goes something like this:
FunctionA (locks on ResourceA) and calls FunctionB where it needs ResourceB FunctionB (locks on ResourceB) and calls FunctionA where it needs ResourceA
And is typically pretty easy to spot, understand and fix…
However, in this particular case things were a bit different.
We had gotten memory dumps of the process and after loading them up in windbg, loding sos.dll, and running ~* e !clrstack to see what the threads were working on, we found that a lot of the threads were sitting in System.Threading.Monitor.Enter like below, indicating that they were waiting on a lock…
Note: I have given the methods in ClassA (and later on ClassB and ClassC) names that tell us what they are locking on so that it will be easier to follow along. If this were a real scenairo you would obviously have to look at the code for ClassA.ThisFunctionLocksOnStringA() to see what it was locking on:)
Finding the deadlock
I loaded up SOSEX.dll from http://www.stevestechspot.com/SOSEXUpdatedV11Available.aspx#ac51f1a46-0016-4f24-9ca7-61125105c6e1 and ran !dlk which looks for deadlocks.
1 deadlock detected.
This tells us that we have a deadlock between CLR Thread 26 and CLR Thread 55, where they both own locks on resources that the other thread is waiting for, so now we know that we have a deadlock and which threads are involved.
!dlk is really great for looking at deadlocks but it has one little snag and that is that it reports the CLR thread numbers which is different than the logical thread IDs that we are used to. However, we can “easily” map them to the logical thread IDs if we use the !threads output. The 2nd column in !threads shows the CLR thread ID in hex.
CLR Thread 26 (0x1a) = logical thread 44 CLR Thread 55 (0x37) = logical thread 73
We could also have avoided doing the math here by dumping out the active locks with !sos.syncblk
Syncblk here tells us that thread 44 owns the lock to syncblock 06e401d0 (so t44 is CLR Thread 26) and thread 73 owns the lock to syncblock 06ec6618 (so t73 is CLR Thread 55 in the !dlk output).
Ok, getting back off that tangent again:), now we know that 44 is waiting on 73 and 73 is waiting on 44, but why?
The stacks for thread 44 and thread 73 look like this
44 is waiting for the lock on StringA (where StringA="LockA") which 73 took in ClassA.ThisFunctionLocksOnStringA() before calling into ClassB.ThisFunctionLocksOnStringB(), so that is all well and good.
The question here is why does Thread 44 hold a lock on StringB?, there is no method on the stack that locked on StringB…
Why are we locking?
ClassB and ClassC look like this (with their own individual lock objects LockB and LockC):
public static void ThisFunctionLocksOnStringB() { lock (LockB) { //Do some stuff } } ... } public class ClassC { public static string LockC = "";
public static void ThisFunctionLocksOnStringC() { lock (LockC) { ClassA.ThisFunctionAlsoLocksOnStringA(); } }
... }
At first sight, LockB and LockC look like different objects, but in reality they are just pointers to the same string constant “”, so in reality this would be the same as saying lock(“”) in both cases. In fact if you did have a 3rd function saying lock(“”) this would also have to wait until the lock for LockB or LockC was released.
Summary
When using locks, best practice is to use a private static object so that you can control who/where code locks on this object.
There are some good guidelines for locks in general at: http://msdn.microsoft.com/en-us/library/c5kehkcz.aspx and http://www.guidanceshare.com/wiki/.NET_2.0_Performance_Guidelines_-_Locking_and_Synchronization
Laters, Tess
Here is an interesting blog post from Eric Lippert that explains a little bit about how string interning works and why it exists:
http://blogs.msdn.com/ericlippert/archive/2009/09/28/string-interning-and-string-empty.aspx
wait, but why were they both able to acquire the lock on the first place? isn't that the purpose of lock to block it so other threads can't enter at the same time and the other threads should wait before acquiring the lock?
Chris, thanks for the link, very interesting
Eber, the string LockA is different (it's actually set to "LockA", should have clarified that) but the strings LockB and LockC are both set to "", in this case 44 holds a lock on "" and 73 is waiting for that lock...
Hi Tess,
Great post. I've had a similar example in my WinDbg talks for a while, because this is one of the few areas where code in different AppDomains can affect each other.
The string interning mechanism is per process, so interned strings are shared across all AppDomains. From the perspective of using strings in the domains this is not a problem since strings are immutable. But when it comes to locking it may actually be a disaster, cause the interning mechanism will return the same reference for identical strings in different domains.
I was under the impression that two "" would point to two different objects in memory?
Like:
string a = string.Empty; //one object
string b = string.Empty; //same object as above
string c = ""; //different object
string d = ""; //yet another object
Bogdan, go look up information on string interning as suggested. The problem is that you do not get different objects when using the double quote syntax. The compiler will automatically reuse the same string object.
This happens, in a slightly different fashion, even with C/C++ compilers.