Debugging WCF Applications - What all I look for.

When I am debugging any WCF runtime bug, I classify it in two categories. One that needs IDE (simple logic bugs) and others that need WinDbg. Every one is familiar with the IDE debugging and will leave that topic for another day. Today I wanted to talk a little bit about how I debug WCF bugs that require Windbg. Usually bugs that involve dumps are varied but if I had to pick the common ones then it would the following ones.

• Server appears "hung" and is not processing messages.
• Client requests are timing out even when server is not loaded.

Note: Concrete classes are in bold and field names of classes are in italics.

Since I dont have too much information on what the contract is, and what modes (ConcurrencyMode/InstanceContextMode) the ServiceHost is configured, thats the first thing I find out. I usually look for ServiceHost objects and then dump the object.

0:006> !do 01273750
Name: System.ServiceModel.ServiceHost
MethodTable: 50e4279c
EEClass: 50e4271c
Size: 136(0x88) bytes
 (C:\WINDOWS\assembly\GAC_MSIL\System.ServiceModel\3.0.0.0__b77a5c561934e089\System.ServiceModel.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
79104f64  4000915       28       System.Boolean  0 instance        0 aborted
79104f64  4000916       29       System.Boolean  0 instance        0 closeCalled
50e51310  4000917        4 ...ct+ExceptionQueue  0 instance 00000000 exceptionQueue
790f9c18  4000918        8        System.Object  0 instance 0127391c mutex
79104f64  4000919       2a       System.Boolean  0 instance        0 onClosingCalled
79104f64  400091a       2b       System.Boolean  0 instance        0 onClosedCalled
79104f64  400091b       2c       System.Boolean  0 instance        1 onOpeningCalled
79104f64  400091c       2d       System.Boolean  0 instance        1 onOpenedCalled
79104f64  400091d       2e       System.Boolean  0 instance        0 raisedClosed
79104f64  400091e       2f       System.Boolean  0 instance        0 raisedClosing
79104f64  400091f       30       System.Boolean  0 instance        0 raisedFaulted
79104f64  4000920       31       System.Boolean  0 instance        0 traceOpenAndClose
790f9c18  4000921        c        System.Object  0 instance 01273750 eventSender
50e385f0  4000922       24         System.Int32  0 instance        2 state
7910d61c  4000923       10  System.EventHandler  0 instance 00000000 Closed
7910d61c  4000924       14  System.EventHandler  0 instance 00000000 Closing
7910d61c  4000925       18  System.EventHandler  0 instance 0128fae0 Faulted
7910d61c  4000926       1c  System.EventHandler  0 instance 00000000 Opened
7910d61c  4000927       20  System.EventHandler  0 instance 00000000 Opening
79104f64  4002bde       32       System.Boolean  0 instance        1 initializeDescriptionHasFinished
50dfecb0  4002bdf       34 ...meKeyedCollection  0 instance 01273928 baseAddresses
50e2068c  4002be0       38 ...patcherCollection  0 instance 012739b8 channelDispatchers
7910ca9c  4002be1       64      System.TimeSpan  1 instance 012737b4 closeTimeout
50df0118  4002be2       3c ...erviceDescription  0 instance 0128fc14 description
00000000  4002be3       40                       0 instance 012739f4 extensions
00000000  4002be4       44                       0 instance 00000000 externalBaseAddresses
00000000  4002be5       48                       0 instance 01293234 implementedContracts
50dfe788  4002be6       4c ...nceContextManager  0 instance 01273a30 instances
7910ca9c  4002be7       6c      System.TimeSpan  1 instance 012737bc openTimeout
50de4e78  4002be8       50 ...rformanceCounters  0 instance 00000000 servicePerformanceCounters
50dfe9f4  4002be9       54 ...r.ServiceThrottle  0 instance 01273a58 serviceThrottle
50e1ed28  4002bea       58 ...erviceCredentials  0 instance 00000000 readOnlyCredentials
50e0b9c0  4002beb       5c ...orizationBehavior  0 instance 012acc84 readOnlyAuthorization
00000000  4002bec       60                       0 instance 00000000 UnknownMessageReceived
7a746d44  4002bdd      74c           System.Uri  0   static 01273880 EmptyUri
790f9c18  4003075       74        System.Object  0 instance 01271bf4 singletonInstance
79101058  4003076       78          System.Type  0 instance 0128fb00 serviceType
50e4f74c  4003077       7c ...ontractCollection  0 instance 01290d20 reflectedContracts
7910c914  4003078       80   System.IDisposable  0 instance 00000000 disposableInstance

In this case you can see that the singletonInstance object is not null which means the InstanceContextMode is Single. You can also infer that if the disposableInstance object is not null. Other way of finding what the InstanceContextMode was to check for InstanceContextProviders and the name of InstanceContextProviders will provide you information on what kind of mode was used (0 = PerSession, 1=PerCall and 2=Single). The serviceType field will tell you the type of contract the host is hosting and with that information you can dig for the actual contract implementation class.

0:006> !dumpheap -type InstanceContextProvider
 Address       MT     Size
0129f780 50dff25c       20    
total 1 objects
Statistics:
      MT    Count    TotalSize Class Name
50dff25c        1           20 System.ServiceModel.Dispatcher.SingletonInstanceContextProvider

When you dump details of any IInstanceContextProvider, you can see that it contains the DispatchRuntime object associated with that provider. Dumping the DispatchRuntime gives you lot of information. For instance the DispatchRuntime object has a field called concurrencyMode that tells me what the ConcurrencyMode of the contract was (0-Single, 1-Multiple and 2-Reentrant).

0:006> !do 0129f780
Name: System.ServiceModel.Dispatcher.SingletonInstanceContextProvider
MethodTable: 50dff25c
EEClass: 50dff1e4
Size: 20(0x14) bytes
 (C:\WINDOWS\assembly\GAC_MSIL\System.ServiceModel\3.0.0.0__b77a5c561934e089\System.ServiceModel.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
50e30180  400328b        4 ...r.DispatchRuntime  0 instance 0129ed50 dispatchRuntime
50e183ac  4003469        8 ...l.InstanceContext  0 instance 0129f7a0 singleton
790f9c18  400346a        c        System.Object  0 instance 0129f794 thisLock
0:006> !do 0129ed50
Name: System.ServiceModel.Dispatcher.DispatchRuntime
MethodTable: 50e30180
EEClass: 50e30110
Size: 112(0x70) bytes
 (C:\WINDOWS\assembly\GAC_MSIL\System.ServiceModel\3.0.0.0__b77a5c561934e089\System.ServiceModel.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
50e2bf6c  4002fd1        4 ...horizationManager  0 instance 00000000 serviceAuthorizationManager
00000000  4002fd2        8                       0 instance 00000000 externalAuthorizationPolicies
50ddbab8  4002fd3       50         System.Int32  0 instance        0 securityAuditLogLocation
50e3ca64  4002fd4       54         System.Int32  0 instance        0 concurrencyMode
79104f64  4002fd5       64       System.Boolean  0 instance        1 suppressAuditFailure
50e0a434  4002fd6       58         System.Int32  0 instance        0 serviceAuthorizationAuditLevel
50e0a434  4002fd7       5c         System.Int32  0 instance        0 messageAuthenticationAuditLevel
79104f64  4002fd8       65       System.Boolean  0 instance        1 automaticInputSessionShutdown
50e1eae0  4002fd9        c ...ChannelDispatcher  0 instance 00000000 channelDispatcher
00000000  4002fda       10                       0 instance 0129ee1c inputSessionShutdownHandlers
50e38924  4002fdb       14 ...ndpointDispatcher  0 instance 0129ebd4 endpointDispatcher
50dfd8e8  4002fdc       18 ...IInstanceProvider  0 instance 00000000 instanceProvider
50e5c730  4002fdd       1c ...ceContextProvider  0 instance 0129f780 instanceContextProvider
50e183ac  4002fde       20 ...l.InstanceContext  0 instance 0129f7a0 singleton
79104f64  4002fdf       66       System.Boolean  0 instance        1 ignoreTransactionMessageProperty
00000000  4002fe0       24                       0 instance 0129ee58 messageInspectors
50e4b3ec  4002fe1       28 ...erationCollection  0 instance 0129edd0 operations
50e628c8  4002fe2       2c ...OperationSelector  0 instance 00000000 operationSelector
50dc9cbc  4002fe3       30 ...her.ClientRuntime  0 instance 012f0fc0 proxyRuntime
50dd8040  4002fe4       34 ...leDispatchRuntime  0 instance 012a881c runtime
00000000  4002fe5       38                       0 instance 0129ee94 instanceContextInitializers
79104f64  4002fe6       67       System.Boolean  0 instance        0 isExternalPoliciesSet
79104f64  4002fe7       68       System.Boolean  0 instance        0 isAuthorizationManagerSet
7910db30  4002fe8       3c ...ronizationContext  0 instance 00000000 synchronizationContext
50e421e0  4002fe9       60         System.Int32  0 instance        1 principalPermissionMode
790f7ab4  4002fea       40          System.Void  0 instance 00000000 roleProvider
79101058  4002feb       44          System.Type  0 instance 0128fb00 type
50e07e68  4002fec       48 ...DispatchOperation  0 instance 0129eed0 unhandled
79104f64  4002fed       69       System.Boolean  0 instance        0 transactionAutoCompleteOnSessionClose
79104f64  4002fee       6a       System.Boolean  0 instance        0 impersonateCallerForAllOperations
79104f64  4002fef       6b       System.Boolean  0 instance        1 releaseServiceInstanceOnTransactionComplete
50dc9e08  4002ff0       4c ...haredRuntimeState  0 instance 0129edc0 shared

Then I generally check the throttle limit sets on the serviceHost by dumping the serviceThrottle object. This also tells me how many throttles (Call/InstanceContext/Session) has been handed out.

0:006> !do 01273a58
Name: System.ServiceModel.Dispatcher.ServiceThrottle
MethodTable: 50dfe9f4
EEClass: 50dfe984
Size: 36(0x24) bytes
 (C:\WINDOWS\assembly\GAC_MSIL\System.ServiceModel\3.0.0.0__b77a5c561934e089\System.ServiceModel.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
50e37290  400349b        4 ...cher.FlowThrottle  0 instance 01273b54 calls ==>We will dump this as a sample
50e37290  400349c        8 ...cher.FlowThrottle  0 instance 01273c10 sessions
50e03a04  400349d        c ...her.QuotaThrottle  0 instance 00000000 dynamic
50e37290  400349e       10 ...cher.FlowThrottle  0 instance 00000000 instanceContexts
50e5e99c  400349f       14 ...l.ServiceHostBase  0 instance 01273750 host
79104f64  40034a0       1c       System.Boolean  0 instance        1 isActive
790f9c18  40034a1       18        System.Object  0 instance 01273af0 thisLock
0:006> !do 01273b54
Name: System.ServiceModel.Dispatcher.FlowThrottle
MethodTable: 50e37290
EEClass: 50e37220
Size: 36(0x24) bytes
 (C:\WINDOWS\assembly\GAC_MSIL\System.ServiceModel\3.0.0.0__b77a5c561934e089\System.ServiceModel.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
790fed1c  400327b       18         System.Int32  0 instance       16 capacity
790fed1c  400327c       1c         System.Int32  0 instance        0 count
790f9c18  400327d        4        System.Object  0 instance 01273b78 mutex
79113c8c  400327e        8 ...ding.WaitCallback  0 instance 01273b34 release
00000000  400327f        c                       0 instance 01273b84 waiters
790fa3e0  4003280       10        System.String  0 instance 01273ab8 propertyName
790fa3e0  4003281       14        System.String  0 instance 01273afc configName

For example you can see that in this case the Call throttle is set to 16 (default) and the count is 0 which means no throttles have been acquired. In cases when the count value is equal to capacity then you can check the waiters object to see which client is waiting for a call throttle. This applies to any FlowThrottle class.

If I wanted to see how many InstanceContext's have been issued by the runtime, then I check the instances property on ServiceHost which points the collection manager. You can dump the items array to see each individual InstanceContext.

0:006> !do 01273a30
Name: System.ServiceModel.Dispatcher.InstanceContextManager
MethodTable: 50e5e584
EEClass: 50e5e50c
Size: 40(0x28) bytes
 (C:\WINDOWS\assembly\GAC_MSIL\System.ServiceModel\3.0.0.0__b77a5c561934e089\System.ServiceModel.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
79104f64  40009df       18       System.Boolean  0 instance        0 aborted
790fed1c  40009e0        c         System.Int32  0 instance        1 busyCount
50e08e90  40009e1        4 ...mmunicationWaiter  0 instance 00000000 busyWaiter
790fed1c  40009e2       10         System.Int32  0 instance        0 busyWaiterCount
790f9c18  40009e3        8        System.Object  0 instance 0127391c mutex
50e26624  40009e4       14         System.Int32  0 instance        0 state
790fed1c  4003488       20         System.Int32  0 instance        2 firstFreeIndex
50e6e258  4003489       1c ...extManager+Item[]  0 instance 012f11d0 items

With the handle to InstanceContext, you can then check for the list of sessionful open channels that are associated with the InstanceContext at that time.

0:006> !do 0129f7a0
Name: System.ServiceModel.InstanceContext
MethodTable: 50e183ac
EEClass: 50e18334
Size: 112(0x70) bytes
 (C:\WINDOWS\assembly\GAC_MSIL\System.ServiceModel\3.0.0.0__b77a5c561934e089\System.ServiceModel.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
79104f64  4000915       28       System.Boolean  0 instance        0 aborted
79104f64  4000916       29       System.Boolean  0 instance        0 closeCalled
50e51310  4000917        4 ...ct+ExceptionQueue  0 instance 00000000 exceptionQueue
790f9c18  4000918        8        System.Object  0 instance 0129f81c mutex
79104f64  4000919       2a       System.Boolean  0 instance        0 onClosingCalled
79104f64  400091a       2b       System.Boolean  0 instance        0 onClosedCalled
79104f64  400091b       2c       System.Boolean  0 instance        1 onOpeningCalled
79104f64  400091c       2d       System.Boolean  0 instance        1 onOpenedCalled
79104f64  400091d       2e       System.Boolean  0 instance        0 raisedClosed
79104f64  400091e       2f       System.Boolean  0 instance        0 raisedClosing
79104f64  400091f       30       System.Boolean  0 instance        0 raisedFaulted
79104f64  4000920       31       System.Boolean  0 instance        0 traceOpenAndClose
790f9c18  4000921        c        System.Object  0 instance 0129f7a0 eventSender
50e385f0  4000922       24         System.Int32  0 instance        2 state
7910d61c  4000923       10  System.EventHandler  0 instance 00000000 Closed
7910d61c  4000924       14  System.EventHandler  0 instance 00000000 Closing
7910d61c  4000925       18  System.EventHandler  0 instance 00000000 Faulted
7910d61c  4000926       1c  System.EventHandler  0 instance 00000000 Opened
7910d61c  4000927       20  System.EventHandler  0 instance 00000000 Opening
79104f64  400346d       32       System.Boolean  0 instance        0 autoClose
50e20c74  400346e       34 ....InstanceBehavior  0 instance 012a88c8 behavior
50e5d498  400346f       38 ...iceChannelManager  0 instance 0129f828 channels
50e1b4f8  4003470       3c ...tanceContextFacet  0 instance 012f1308 concurrency
00000000  4003471       40                       0 instance 00000000 extensions
50e5e99c  4003472       44 ...l.ServiceHostBase  0 instance 01273750 host
50e03a04  4003473       48 ...her.QuotaThrottle  0 instance 00000000 quotaThrottle
50dfe9f4  4003474       4c ...r.ServiceThrottle  0 instance 00000000 serviceThrottle
790fed1c  4003475       64         System.Int32  0 instance        1 instanceContextManagerIndex
790f9c18  4003476       50        System.Object  0 instance 0129f810 serviceInstanceLock
7910db30  4003477       54 ...ronizationContext  0 instance 00000000 synchronizationContext
50e0b340  4003478       58 ...tanceContextFacet  0 instance 00000000 transaction
790f9c18  4003479       5c        System.Object  0 instance 01271bf4 userObject
79104f64  400347a       33       System.Boolean  0 instance        1 wellKnown
00000000  400347b       60                       0 instance 00000000 wmiChannels
79104f64  400347c       68       System.Boolean  0 instance        0 isUserCreated
50e6393c  400346b      8c4 ...textEmptyCallback  0   static 012f13bc NotifyEmptyCallback
50dde148  400346c      8c8 ...ntextIdleCallback  0   static 012f13dc NotifyIdleCallback
0:006> !do 0129f828
Name: System.ServiceModel.ServiceChannelManager
MethodTable: 50e5d498
EEClass: 50e5d420
Size: 64(0x40) bytes
 (C:\WINDOWS\assembly\GAC_MSIL\System.ServiceModel\3.0.0.0__b77a5c561934e089\System.ServiceModel.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
79104f64  40009df       18       System.Boolean  0 instance        0 aborted
790fed1c  40009e0        c         System.Int32  0 instance        0 busyCount
50e08e90  40009e1        4 ...mmunicationWaiter  0 instance 00000000 busyWaiter
790fed1c  40009e2       10         System.Int32  0 instance        0 busyWaiterCount
790f9c18  40009e3        8        System.Object  0 instance 0129f81c mutex
50e26624  40009e4       14         System.Int32  0 instance        0 state
790fed1c  400347f       34         System.Int32  0 instance        0 activityCount
50e08e90  4003480       1c ...mmunicationWaiter  0 instance 00000000 activityWaiter
790fed1c  4003481       38         System.Int32  0 instance        0 activityWaiterCount
50e6393c  4003482       20 ...textEmptyCallback  0 instance 00000000 emptyCallback
50dc8564  4003483       24 ...Channels.IChannel  0 instance 00000000 firstIncomingChannel
50e60abc  4003484       28 ...ChannelCollection  0 instance 00000000 incomingChannels
50e60abc  4003485       2c ...ChannelCollection  0 instance 012f3ca8 outgoingChannels
50e183ac  4003486       30 ...l.InstanceContext  0 instance 0129f7a0 instanceContext

The channels field in InstanceContext points to the Sessionful channels collection which when dumped will give you list of all channels when the incomingChannels field is accessed. If you determine the ConcurrencyMode to be Reentrant then its possible that some requests can be queued in the InstanceContext when its actively servicing a message. This information can be found by dumping the concurrency field in InstanceContext.

0:006> !do 012f1308
Name: System.ServiceModel.Dispatcher.ConcurrencyInstanceContextFacet
MethodTable: 50e1b4f8
EEClass: 50e1b488
Size: 20(0x14) bytes
 (C:\WINDOWS\assembly\GAC_MSIL\System.ServiceModel\3.0.0.0__b77a5c561934e089\System.ServiceModel.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
79104f64  4002fc0        c       System.Boolean  0 instance        0 Locked
00000000  4002fc1        4                       0 instance 00000000 calloutMessageQueue
00000000  4002fc2        8                       0 instance 00000000 newMessageQueue

calloutMessageQueue is the queue which holds the message which when processing made an outgoing call. The newMessageQueue denotes the new messages coming in that are waiting for the InstanceContext to finish processing the message.

Coming back to the channels collection in InstanceContext, you can pick any one channel from the list and when you dump it, it will be of type ServiceChannel. This is the internal representation of all WCF channels. Like the DispatchRuntime, the ServiceChannel also provides lot of information.

0:006> !do 012f0ebc
Name: System.ServiceModel.Channels.ServiceChannel
MethodTable: 50dcd674
EEClass: 50dcd5fc
Size: 152(0x98) bytes
 (C:\WINDOWS\assembly\GAC_MSIL\System.ServiceModel\3.0.0.0__b77a5c561934e089\System.ServiceModel.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
79104f64  4000915       28       System.Boolean  0 instance        0 aborted
79104f64  4000916       29       System.Boolean  0 instance        0 closeCalled
50e51310  4000917        4 ...ct+ExceptionQueue  0 instance 00000000 exceptionQueue
790f9c18  4000918        8        System.Object  0 instance 012f0f54 mutex
79104f64  4000919       2a       System.Boolean  0 instance        0 onClosingCalled
79104f64  400091a       2b       System.Boolean  0 instance        0 onClosedCalled
79104f64  400091b       2c       System.Boolean  0 instance        1 onOpeningCalled
79104f64  400091c       2d       System.Boolean  0 instance        1 onOpenedCalled
79104f64  400091d       2e       System.Boolean  0 instance        0 raisedClosed
79104f64  400091e       2f       System.Boolean  0 instance        0 raisedClosing
79104f64  400091f       30       System.Boolean  0 instance        0 raisedFaulted
79104f64  4000920       31       System.Boolean  0 instance        0 traceOpenAndClose
790f9c18  4000921        c        System.Object  0 instance 012f0ebc eventSender
50e385f0  4000922       24         System.Int32  0 instance        2 state
7910d61c  4000923       10  System.EventHandler  0 instance 012f11fc Closed
7910d61c  4000924       14  System.EventHandler  0 instance 00000000 Closing
7910d61c  4000925       18  System.EventHandler  0 instance 00000000 Faulted
7910d61c  4000926       1c  System.EventHandler  0 instance 00000000 Opened
7910d61c  4000927       20  System.EventHandler  0 instance 00000000 Opening
790fed1c  400310b       7c         System.Int32  0 instance        1 activityCount ==>Denotes how many activity is being actively processed. 0 indicates the Channel is ready to be closed.
79104f64  400310c       32       System.Boolean  0 instance        1 allowInitializationUI
79104f64  400310d       33       System.Boolean  0 instance        0 allowOutputBatching
79104f64  400310e       80       System.Boolean  0 instance        1 autoClose ==>Will autoclose when the other side initiates a Close.
50dd04c8  400310f       34 ...l+CallOnceManager  0 instance 00000000 autoDisplayUIManager
50dd04c8  4003110       38 ...l+CallOnceManager  0 instance 00000000 autoOpenManager
50dcd9f0  4003111       3c ...er.IChannelBinder  0 instance 012c8cd4 binder
50e1eae0  4003112       40 ...ChannelDispatcher  0 instance 0129e898 channelDispatcher
50dc9cbc  4003113       44 ...her.ClientRuntime  0 instance 012f0fc0 clientRuntime
79104f64  4003114       81       System.Boolean  0 instance        1 closeBinder
79104f64  4003115       82       System.Boolean  0 instance        0 closeFactory
79104f64  4003116       83       System.Boolean  0 instance        0 didIdleAbort ==>Did the idle timer expire
79104f64  4003117       84       System.Boolean  0 instance        0 didInteractiveInitialization
79104f64  4003118       85       System.Boolean  0 instance        0 doneReceiving ==>Usually means the Channel is done receiveing replies for all outstanding requests
50e38924  4003119       48 ...ndpointDispatcher  0 instance 0129ebd4 endpointDispatcher
79104f64  400311a       86       System.Boolean  0 instance        1 explicitlyOpened ==> Was it autoopened or explicitly opened
00000000  400311b       4c                       0 instance 00000000 extensions
50dc8618  400311c       50 ...iceChannelFactory  0 instance 00000000 factory
79104f64  400311d       87       System.Boolean  0 instance        1 hasSession
50e4331c  400311e       54 ...essionIdleManager  0 instance 012f1168 idleManager
50e183ac  400311f       58 ...l.InstanceContext  0 instance 0129f7a0 instanceContext
50dfe9f4  4003120       5c ...r.ServiceThrottle  0 instance 00000000 instanceContextServiceThrottle
79104f64  4003121       88       System.Boolean  0 instance        0 isPending
79104f64  4003122       89       System.Boolean  0 instance        0 isReplyChannel
50dbc7d0  4003123       60 ...l.EndpointAddress  0 instance 00000000 localAddress
50dc3874  4003124       64 ...ls.MessageVersion  0 instance 0129658c messageVersion
79104f64  4003125       8a       System.Boolean  0 instance        0 openBinder
7910ca9c  4003126       8c      System.TimeSpan  1 instance 012f0f48 operationTimeout
790f9c18  4003127       68        System.Object  0 instance 00000000 proxy
50dfe9f4  4003128       6c ...r.ServiceThrottle  0 instance 01273a58 serviceThrottle
790fa3e0  4003129       70        System.String  0 instance 00000000 terminatingOperationName
50e183ac  400312a       74 ...l.InstanceContext  0 instance 00000000 wmiInstanceContext
79104f64  400312b       8b       System.Boolean  0 instance        0 wasChannelAutoClosed
00000000  400312c       78                       0 instance 00000000 unknownMessageReceived

Digging through the binder field in ServiceChannel will give you the IChannelBinder which will hold a reference to the ChannelHandler object that is responsible for pumping messages from that Channel. If you feel that messages from a particular connected client is not being processed it could be that he ChannelHandler is not pumping messages. I say connected because only channels that have been accepted by the runtime will be having a ChannelHandler object. If there is no session throttles left then clients will timeout in their Open() call itself.

0:006> !do 012c8df0
Name: System.ServiceModel.Dispatcher.ChannelHandler
MethodTable: 50e07bd0
EEClass: 50e07b58
Size: 104(0x68) bytes
 (C:\WINDOWS\assembly\GAC_MSIL\System.ServiceModel\3.0.0.0__b77a5c561934e089\System.ServiceModel.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
50dec920  40031b6       4c ...ndler+RequestInfo  1 instance 012c8e3c requestInfo
50dcd9f0  40031b7        4 ...er.IChannelBinder  0 instance 012c8cd4 binder
50dcd674  40031b8        8 ...ls.ServiceChannel  0 instance 012f0ebc channel
79104f64  40031b9       3c       System.Boolean  0 instance        0 doneReceiving ==>Indicates all messages have been pumped from the Channel
50dcddf0  40031ba        c ...plexChannelBinder  0 instance 012c8cd4 duplexBinder
79104f64  40031bb       3d       System.Boolean  0 instance        1 hasRegisterBeenCalled
79104f64  40031bc       3e       System.Boolean  0 instance        1 hasSession
50e5e99c  40031bd       10 ...l.ServiceHostBase  0 instance 01273750 host
79104f64  40031be       3f       System.Boolean  0 instance        0 incrementedActivityCountInConstructor
79104f64  40031bf       40       System.Boolean  0 instance        0 isCallback
79104f64  40031c0       41       System.Boolean  0 instance        0 isPumpAcquired
79104f64  40031c1       42       System.Boolean  0 instance        0 isChannelTerminated
79104f64  40031c2       43       System.Boolean  0 instance        0 isConcurrent ==>0 indicates Single ConcurrencyMode
79104f64  40031c3       44       System.Boolean  0 instance        0 isManualAddressing
50e139b0  40031c4       14 ...r.ListenerHandler  0 instance 012acb8c listener ==>Listener off which Messages are received
50dc3874  40031c5       18 ...ls.MessageVersion  0 instance 0129658c messageVersion
50dfe854  40031cc       1c ...rHandlingReceiver  0 instance 012c8e70 receiver
79104f64  40031cd       45       System.Boolean  0 instance        0 receiveSynchronously ==>Indicated whether the pump is sync or async
79104f64  40031ce       46       System.Boolean  0 instance        0 receiveWithTransaction
50e65618  40031cf       20 ...ls.RequestContext  0 instance 00000000 replied
50e65618  40031d0       24 ...ls.RequestContext  0 instance 00000000 requestWaitingForThrottle
50e4bab4  40031d1       28 ...rappedTransaction  0 instance 00000000 acceptTransaction
50dfe9f4  40031d2       2c ...r.ServiceThrottle  0 instance 01273a58 throttle
79104f64  40031d3       47       System.Boolean  0 instance        1 wasChannelThrottled
50dfe9f4  40031d4       30 ...r.ServiceThrottle  0 instance 00000000 instanceContextThrottle
50e60c28  40031d5       34 ...actedBatchContext  0 instance 00000000 sharedTransactedBatchContext
00000000  40031d6       38 ...actedBatchContext  0 instance 00000000 transactedBatchContext
79104f64  40031d7       48       System.Boolean  0 instance        0 isMainTransactedBatchHandler
7910ca9c  40031b5      844      System.TimeSpan  1   static 012c8e80 CloseAfterFaultTimeout
79107d2c  40031c6      848 System.AsyncCallback  0   static 012c8ec0 onAsyncReceiveComplete
79113c8c  40031c7      84c ...ding.WaitCallback  0   static 012c8ee0 onContinueAsyncReceive
79113c8c  40031c8      850 ...ding.WaitCallback  0   static 012c8f00 onStartSyncMessagePump
79113c8c  40031c9      854 ...ding.WaitCallback  0   static 012c8f20 onStartAsyncMessagePump
79113c8c  40031ca      858 ...ding.WaitCallback  0   static 012c8f40 onStartSingleTransactedBatch
79113c8c  40031cb      85c ...ding.WaitCallback  0   static 012c8f60 openAndEnsurePump

Think I should stop at this for today. Some of the information here should be useful for all end user developers (like throttles). Due to our extensive testing, end users should never have to worry about debugging some classes (like ChannelDispatcher) unless they implement their own channel stack and want to know why the ChannelDispatcher is not pumping messages off their channel. Hopefully this will gives you enough information to go and dig around your server dumps and try to figure out where the bottle neck could be (Waiting for throttle/Waiting for reentrance/waiting for new message to be received etc).

Next time I will talk about how to debug the client side ServiceChannel (in case of timeouts) and talk a little bit more about input queues on the server.

Maheshwar Jayaraman