WCF enables throttling execution of operations but not their completions. This becomes and issue when a large number of outstanding operations complete almost simultaneously causing the callback on the client to be overwhelmed with completions.  Generally we don’t expect the client to issue of infinite number of pending operations but if you do end up with very high CPU usage and all suspect all your operations are stuck in the callback method which takes a lock then you need to throttle the callbacks yourself.

You could try Setting the minThreads but this affects the whole app domain. The issue is due to the large number of callbacks that come in concurrently. The sample attached throttles the callbacks to have only one thread execute completions while there are 20 threads starting the operations and all completing almost simultaneously. The idea is to wrap the AsyncResult  of your operation and complete only the required number of results in parallel and this would throttle the service operation Ends automatically. 

With xperf being more and more adopted and with rich stackwalking capabilities, its only natural to use it for finding out bottlenecks and cause for switch out.

Findout the ready thread information and what causes the threads to switch out and the associated stack that woke up when a thread switches back in is one way to determine what was the offending stack that causes other some other threads to switch out. This helps us identify potential hot locks or just really expensive locks or issues due to false data sharing.

You can run the following command to capture stack traces with ready thread information.

xperf –on base+cswitch+dispatcher –stackwalk cswitch+readythread

Incase you are not sure of how to debug managed code with with a crash/hang dump, then you most likely need to read this first. Once you have SOS and mscordacwks(.net 3.5 and up)  loaded you first dump the heap to find out if you have any services hosts at all.

1. We quickly find the method table entry for the ServiceHost type. We can get this from !dumpheap command as shown below. The following command helps to filter out just the types we are interested in.
   
   !dumpheap –stat –type System.Servicemodel.ServiceHost
2. The next step is to dump all the ServiceHost using the method table entry shown above. The foreach command in the debugger helps us to loop through all objects that below to the type.
   
   .foreach(x {!dumpheap -short -mt 000007fef39ba7f0}) {!do x}
   
3. Our next step would be to find out how many calls are pending on the service. For this we can examine the ServiceThrottle on calls Field the service host. You can either use the instance value of the service throttle or offset. The following shows the call throttle and the count value in the flow throttle object would show how many services are being executed. The FlowThrottle is the object that holds the counters for the calls for the service.
   
   In this case there is nothing executing so this dump was probably collected when there was no load. This is a quick way to identify if your service was actually executing any operations when collecting a process dump.