Troubleshooting a RemotingException by dumping out Remoting Channel Configuration - Distributed Matters - Site Home

This post is devoted to .NET Remoting, one of the technologies supported by our Distributed Services support team (though I should add that the volume of requests is not, and never has been, very high).

WCF is being more and more widely adopted as the communication infrastructure of .NET applications, but .NET Remoting still has its place in some scenarios. In particular, when platform interoperability is not required and marshal-by-reference semantics is needed.

The problem

At runtime, a .NET Remoting client receives a RemotingException exception when invoking a method on a remote object of type Server.Obj. The remoting object is accessed through a TCP channel.

This is the complete exception message:

Unhandled Exception: System.Runtime.Remoting.RemotingException: This remoting proxy has no channel sink which means either the server has no registered server channels that are listening, or this application has no suitable client channel to talk to the server.
   at System.Runtime.Remoting.Proxies.RemotingProxy.InternalInvoke(IMethodCallMessage reqMcmMsg, Boolean useDispatchMessage, Int32 callType)
   at System.Runtime.Remoting.Proxies.RemotingProxy.Invoke(IMessage reqMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at Itfs.IObj.DoNothing()
   at Client.Program.Main()

Note: the problem was originally experienced in the context of a complex application with a very large code base. We are using a much simpler repro here in order to focus on the issue at hand. Nonetheless, we’ll be using the same troubleshooting techniques that apply to complex and unknown solutions: instead of analyzing the code in search of a flaw, which is very time-consuming, we’ll start from the exception and from the analysis of the .NET Remoting data structures at runtime in order to find out where the problem is. Yes, you guessed right Smile : this means analyzing dumps with WinDbg.

The message of the RemotingException above explains quite a bit about the issue: the proxy does not have a channel sink. The message also reports 2 possible reasons: the server has no registered server channel, or the client has no suitable client channel. In order to fully understand what this means, a bit of background information on how .NET Remoting channels work is needed. Later, we’ll use this background information in order to troubleshoot the problem.

How .NET Remoting Channels work

.NET Remoting channels are classes that implement the IChannel interface. They are registered per-AppDomain through ChannelServices.RegisterChannel(). There are actually 2 types of channels: receiver channels implement the IChannelReceiver interface, sender channels implement the IChannelSender interface. They are also known as server-side and client-side channels, respectively. TcpServerChannel and HttpServerChannel are server-side channels, whereas TcpClientChannel and HttpClientChannel are client-side channels.
Channels are selected by .NET Remoting at runtime in this way: whenever a MarshalByRefObject instance is marshaled out of its AppDomain, information on the server channels for that AppDomain will flow with the ObjRef, in priority order (IChannel.ChannelPriority property).
When the ObjRef is deserialized on the client-side, each registered client-side channel for the current AppDomain is queried (again, in priority order) to check if it can handle the URL specified in the channelData in the marshaled packet. The first match is used.

The remoting infrastructure provides implicit TCP and HTTP client-side channels if they are not explicitly registered. Explicit client-channel registration overrides implicit registration.

The undocumented channel CrossAppDomainChannel is automatically registered. This makes it always possible to access remoting objects across AppDomain boundaries within the same process.

Last, the classes TcpChannel and HttpChannel are convenience channels that implement both IChannelReceiver and IChannelSender. As a consequence, they play the role of client-side AND server-side channels.

Troubleshooting the client

As we mentioned earlier, we are going to start from a dump file of the client application, which is where we receive the exception. When the problem is in an application that you do not know (this is always the case in our job), this is usually faster and more direct than analyzing the code.

There is another reason why starting from a dump (that is, from the data structures at runtime) is handy: in .NET, configuration can be done in 2 ways: imperatively in the code, or declaratively in configuration files. .NET Remoting configuration is no exception, so its channels, services and other configuration information can be both in code AND in config files. Therefore, it is not easy to figure out, from code and config files, which is the actual runtime configuration.

Note: some of the data structures that we are going to dump out are internal and, as such, they are subject to change in future versions of the .NET Framework. The analysis that follows applies to .NET Framework versions 2.0-3.5.

So let’s look at a dump of the client. In particular, let’s have a look at the registered channels. The client has only the default AppDomain, in addition to the ever-present System Domain and Shared Domain:

0:000> !dumpdomain
--------------------------------------
System Domain: 6e4ce1f8
LowFrequencyHeap: 6e4ce21c
HighFrequencyHeap: 6e4ce268
StubHeap: 6e4ce2b4
Stage: OPEN
Name: None
--------------------------------------
Shared Domain: 6e4cdb48
LowFrequencyHeap: 6e4cdb6c
HighFrequencyHeap: 6e4cdbb8
StubHeap: 6e4cdc04
Stage: OPEN
Name: None
Assembly: 002e72d0
--------------------------------------
Domain 1: 002a1df8
LowFrequencyHeap: 002a1e1c
HighFrequencyHeap: 002a1e68
StubHeap: 002a1eb4
Stage: OPEN
SecurityDescriptor: 002a3120
Name: Client.exe
Assembly: 002e72d0 [C:\Windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll]
ClassLoader: 002e7350
SecurityDescriptor: 002e3900
  Module Name
6d481000 C:\Windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll
00272358 C:\Windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\sortkey.nlp
00272010 C:\Windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\sorttbls.nlp

...

The list of registered channels for the default AppDomain can be found by checking at RegisteredChannelList objects rooted in domain 002a1df8:

0:000> !dumpheap -type RegisteredChannelList
 Address       MT     Size
01f448f0 6d6cc1bc       12     
01f449a4 6d6cc1bc       12     
01f44ad0 6d6cc1bc       12     
total 3 objects
Statistics:
      MT    Count    TotalSize Class Name
6d6cc1bc        3           36 System.Runtime.Remoting.Channels.RegisteredChannelList
Total 3 objects
0:000> !gcroot 01f448f0
Note: Roots found on stacks may be false positives. Run "!help gcroot" for
more info.
Scan Thread 0 OSTHread 1164
Scan Thread 2 OSTHread 267c
Scan Thread 3 OSTHread 1f88
Scan Thread 7 OSTHread 235c
0:000> !gcroot 01f449a4
Note: Roots found on stacks may be false positives. Run "!help gcroot" for
more info.
Scan Thread 0 OSTHread 1164
Scan Thread 2 OSTHread 267c
Scan Thread 3 OSTHread 1f88
Scan Thread 7 OSTHread 235c
0:000> !gcroot 01f44ad0
Note: Roots found on stacks may be false positives. Run "!help gcroot" for
more info.
Scan Thread 0 OSTHread 1164
Scan Thread 2 OSTHread 267c
Scan Thread 3 OSTHread 1f88
Scan Thread 7 OSTHread 235c
DOMAIN(002A1DF8):HANDLE(Pinned):1313fc:Root:02eb1010(System.Object[])->
01f44ad0(System.Runtime.Remoting.Channels.RegisteredChannelList)

The RegisteredChannelList for the default domain is 01f44ad0, let’s check its contents:

0:000> !do 01f44ad0
Name: System.Runtime.Remoting.Channels.RegisteredChannelList
MethodTable: 6d6cc1bc
EEClass: 6d4a2ee4
Size: 12(0xc) bytes
 (C:\Windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
6d6c4eec  4001f1d        4      System.Object[]  0 instance 01f44aa8 _channels
0:000> !da 01f44aa8 
Name: System.Runtime.Remoting.Channels.RegisteredChannel[]
MethodTable: 6d6c4eec
EEClass: 6d4aa8a0
Size: 24(0x18) bytes
Array: Rank 1, Number of elements 2, Type CLASS
Element Methodtable: 6d6cc204
[0] 01f44ac0
[1] 01f44994
0:000> !do 01f44ac0
Name: System.Runtime.Remoting.Channels.RegisteredChannel
MethodTable: 6d6cc204
EEClass: 6d507784
Size: 16(0x10) bytes
 (C:\Windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
6d6cc248  4001f1b        4 ...Channels.IChannel  0 instance 01f44a4c channel
6d6eb3e0  4001f1c        8          System.Byte  1 instance        3 flags
0:000> !do 01f44a4c 
Name: System.Runtime.Remoting.Channels.CrossAppDomainChannel
MethodTable: 6d6cc0e4
EEClass: 6d507720
Size: 12(0xc) bytes
 (C:\Windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
6d6e84dc  4001f7e      664        System.Object  0   shared   static staticSyncObject
    >> Domain:Value  002a1df8:01f44a58 <<
6d6ea584  4001f7f      668 ...ity.PermissionSet  0   shared   static s_fullTrust
    >> Domain:Value  002a1df8:01f44a64 <<
0:000> !do 01f44994
Name: System.Runtime.Remoting.Channels.RegisteredChannel
MethodTable: 6d6cc204
EEClass: 6d507784
Size: 16(0x10) bytes
 (C:\Windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
6d6cc248  4001f1b        4 ...Channels.IChannel  0 instance 01f4487c channel
6d6eb3e0  4001f1c        8          System.Byte  1 instance        3 flags
0:000> !do 01f4487c 
Name: System.Runtime.Remoting.Channels.Tcp.TcpChannel
MethodTable: 61f476f4
EEClass: 61eb5bec
Size: 24(0x18) bytes
 (C:\Windows\assembly\GAC_MSIL\System.Runtime.Remoting\2.0.0.0__b77a5c561934e089\System.Runtime.Remoting.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
61f47794  4000160        4 ....TcpClientChannel  0 instance 01f448ac _clientChannel
61f4781c  4000161        8 ....TcpServerChannel  0 instance 00000000 _serverChannel
6d6eab0c  4000162       10         System.Int32  1 instance        1 _channelPriority
6d6e88c0  4000163        c        System.String  0 instance 01f44894 _channelName

So the registered channels include the CrossAppDomainChannel and the TcpChannel, and both are send AND receive channels (RegisteredChannel.flags = 3). Also, the TcpChannel has a valid _clientChannel and does not have a _serverChannel.

By now we know that this is the expected runtime configuration for a remoting client using the TCP channel. In other words, the analysis above shows that there is nothing wrong with the channel configuration of the client. As you may have guessed Smile

, it is time to look into the server.

Troubleshooting the server

We know that the remoting object which is being used by the client is an instance of the Server.Obj class. Let’s determine which AppDomain hosts this object:

0:013> !dumpheap -type Server.Obj
 Address       MT     Size
01d79960 03d035a8       12     
total 1 objects
Statistics:
      MT    Count    TotalSize Class Name
03d035a8        1           12 Server.Obj
Total 1 objects
0:013> !dumpmt 03d035a8
EEClass: 03d012f0
Module: 03d02d68
Name: Server.Obj
mdToken: 02000004  (d:\MSDNBlog\RemotingExceptionWithChannels\Repro\Server\bin\Debug\Server.exe)
BaseSize: 0xc
ComponentSize: 0x0
Number of IFaces in IFaceMap: 1
Slots in VTable: 9
0:013> !dumpmodule 03d02d68
Name: d:\MSDNBlog\RemotingExceptionWithChannels\Repro\Server\bin\Debug\Server.exe
Attributes: PEFile 
Assembly: 00490bb0
LoaderHeap: 00000000
TypeDefToMethodTableMap: 03d00038
TypeRefToMethodTableMap: 03d0004c
MethodDefToDescMap: 03d000d0
FieldDefToDescMap: 03d000ec
MemberRefToDescMap: 03d000f4
FileReferencesMap: 03d00178
AssemblyReferencesMap: 03d0017c
MetaData start address: 010f211c (2380 bytes)
0:013> !dumpassembly 00490bb0
Parent Domain: 004d4880
Name: d:\MSDNBlog\RemotingExceptionWithChannels\Repro\Server\bin\Debug\Server.exe
ClassLoader: 00490c20
SecurityDescriptor: 004f4600
  Module Name
03d02d68 d:\MSDNBlog\RemotingExceptionWithChannels\Repro\Server\bin\Debug\Server.exe

Note that we went through the object’s Method Table, containing Module and Containing Assembly in order to determine the object’s AppDomain, which is at address 0x004d4880.

Let’s dump out the RegisteredChannelList objects for each AppDomain in the server. We know by now how to do this, since we did exactly the same thing for the client:

0:013> !dumpheap -type RegisteredChannelList
 Address       MT     Size
01d30ffc 6d6cc1bc       12     
01d3422c 6d6cc1bc       12     
01d4727c 6d6cc1bc       12     
01d4b554 6d6cc1bc       12     
01d71758 6d6cc1bc       12     
01d71798 6d6cc1bc       12     
total 6 objects
Statistics:
      MT    Count    TotalSize Class Name
6d6cc1bc        6           72 System.Runtime.Remoting.Channels.RegisteredChannelList
Total 6 objects
0:013> !gcroot 01d30ffc
Note: Roots found on stacks may be false positives. Run "!help gcroot" for
more info.
Scan Thread 0 OSTHread 1254
Scan Thread 2 OSTHread 11f4
Scan Thread 6 OSTHread 2380
Scan Thread 7 OSTHread 1904
Scan Thread 9 OSTHread 1130
Scan Thread 10 OSTHread 221c
Scan Thread 12 OSTHread 2424
Scan Thread 11 OSTHread 1594
DOMAIN(0044A330):HANDLE(Pinned):1713fc:Root:02d01010(System.Object[])->
01d30ffc(System.Runtime.Remoting.Channels.RegisteredChannelList)
0:013> !gcroot 01d3422c
Note: Roots found on stacks may be false positives. Run "!help gcroot" for
more info.
Scan Thread 0 OSTHread 1254
Scan Thread 2 OSTHread 11f4
Scan Thread 6 OSTHread 2380
Scan Thread 7 OSTHread 1904
Scan Thread 9 OSTHread 1130
Scan Thread 10 OSTHread 221c
Scan Thread 12 OSTHread 2424
Scan Thread 11 OSTHread 1594
DOMAIN(004A4908):HANDLE(Pinned):8112fc:Root:02d06280(System.Object[])->
01d3422c(System.Runtime.Remoting.Channels.RegisteredChannelList)
0:013> !gcroot 01d4727c 
Note: Roots found on stacks may be false positives. Run "!help gcroot" for
more info.
Scan Thread 0 OSTHread 1254
Scan Thread 2 OSTHread 11f4
Scan Thread 6 OSTHread 2380
Scan Thread 7 OSTHread 1904
Scan Thread 9 OSTHread 1130
Scan Thread 10 OSTHread 221c
Scan Thread 12 OSTHread 2424
Scan Thread 11 OSTHread 1594
DOMAIN(004BB1D0):HANDLE(Pinned):9412fc:Root:02d086a0(System.Object[])->
01d4727c(System.Runtime.Remoting.Channels.RegisteredChannelList)
0:013> !gcroot 01d4b554 
Note: Roots found on stacks may be false positives. Run "!help gcroot" for
more info.
Scan Thread 0 OSTHread 1254
Scan Thread 2 OSTHread 11f4
Scan Thread 6 OSTHread 2380
Scan Thread 7 OSTHread 1904
Scan Thread 9 OSTHread 1130
Scan Thread 10 OSTHread 221c
Scan Thread 12 OSTHread 2424
Scan Thread 11 OSTHread 1594
DOMAIN(004C3D18):HANDLE(Pinned):db12fc:Root:02d096b0(System.Object[])->
01d4b554(System.Runtime.Remoting.Channels.RegisteredChannelList)
0:013> !gcroot 01d71758 
Note: Roots found on stacks may be false positives. Run "!help gcroot" for
more info.
Scan Thread 0 OSTHread 1254
Scan Thread 2 OSTHread 11f4
Scan Thread 6 OSTHread 2380
Scan Thread 7 OSTHread 1904
Scan Thread 9 OSTHread 1130
Scan Thread 10 OSTHread 221c
Scan Thread 12 OSTHread 2424
Scan Thread 11 OSTHread 1594
0:013> !gcroot 01d71798 
Note: Roots found on stacks may be false positives. Run "!help gcroot" for
more info.
Scan Thread 0 OSTHread 1254
Scan Thread 2 OSTHread 11f4
Scan Thread 6 OSTHread 2380
Scan Thread 7 OSTHread 1904
Scan Thread 9 OSTHread 1130
Scan Thread 10 OSTHread 221c
Scan Thread 12 OSTHread 2424
Scan Thread 11 OSTHread 1594
DOMAIN(004D4880):HANDLE(Pinned):3d112fc:Root:02d0a6c0(System.Object[])->
01d71798(System.Runtime.Remoting.Channels.RegisteredChannelList)

5 out of 6 RegisteredChannelList instances are rooted in some AppDomain, but none is rooted in the AppDomain 004d4880, which contains the remoting object. This means that there is a problem with the channel configuration of the server: since no channel is registered in the domain of the Server.Obj remoting object, the object cannot be accessed by a remote client. This is what causes the RemotingException on the client.
Let’s dump out the AppDomains of the server:

0:013> !dumpdomain
--------------------------------------
System Domain: 6e4ce1f8
LowFrequencyHeap: 6e4ce21c
HighFrequencyHeap: 6e4ce268
StubHeap: 6e4ce2b4
Stage: OPEN
Name: None
--------------------------------------
Shared Domain: 6e4cdb48
LowFrequencyHeap: 6e4cdb6c
HighFrequencyHeap: 6e4cdbb8
StubHeap: 6e4cdc04
Stage: OPEN
Name: None
Assembly: 0045f0a8
--------------------------------------
Domain 1: 0044a330
LowFrequencyHeap: 0044a354
HighFrequencyHeap: 0044a3a0
StubHeap: 0044a3ec
Stage: OPEN
SecurityDescriptor: 0044b018
Name: Server.exe
Assembly: 0045f0a8 [C:\Windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll]
ClassLoader: 0045f118
SecurityDescriptor: 0045c3c0
  Module Name
6d481000 C:\Windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll
001b2358 C:\Windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\sortkey.nlp
001b2010 C:\Windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\sorttbls.nlp
...

--------------------------------------
Domain 2: 004a4908
LowFrequencyHeap: 004a492c
HighFrequencyHeap: 004a4978
StubHeap: 004a49c4
Stage: OPEN
SecurityDescriptor: 00461e70
Name: a52aab56-8c58-4b32-b298-d22914734328
Assembly: 0045f0a8 [C:\Windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll]
ClassLoader: 0045f118
SecurityDescriptor: 004944b0
  Module Name
6d481000 C:\Windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll
001b2358 C:\Windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\sortkey.nlp
001b2010 C:\Windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\sorttbls.nlp
...

--------------------------------------
Domain 3: 004bb1d0
LowFrequencyHeap: 004bb1f4
HighFrequencyHeap: 004bb240
StubHeap: 004bb28c
Stage: OPEN
SecurityDescriptor: 00462130
Name: 471b56c5-5b8c-44aa-a3e6-5d14fe3c4d91
Assembly: 0045f0a8 [C:\Windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll]
ClassLoader: 0045f118
SecurityDescriptor: 004948f0
  Module Name
6d481000 C:\Windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll
001b2358 C:\Windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\sortkey.nlp
001b2010 C:\Windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\sorttbls.nlp
...

--------------------------------------
Domain 4: 004c3d18
LowFrequencyHeap: 004c3d3c
HighFrequencyHeap: 004c3d88
StubHeap: 004c3dd4
Stage: OPEN
SecurityDescriptor: 004b70d0
Name: 970ee547-afb3-491b-b159-359af030978a
Assembly: 0045f0a8 [C:\Windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll]
ClassLoader: 0045f118
SecurityDescriptor: 00494b98
  Module Name
6d481000 C:\Windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll
001b2358 C:\Windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\sortkey.nlp
001b2010 C:\Windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\sorttbls.nlp
...

--------------------------------------
Domain 5: 004d4880
LowFrequencyHeap: 004d48a4
HighFrequencyHeap: 004d48f0
StubHeap: 004d493c
Stage: OPEN
SecurityDescriptor: 00461ec8
Name: b343ebfd-7820-45c5-8a25-1fa0738d59bb
Assembly: 0045f0a8 [C:\Windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll]
ClassLoader: 0045f118
SecurityDescriptor: 00494db8
  Module Name
6d481000 C:\Windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll
001b2358 C:\Windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\sortkey.nlp
001b2010 C:\Windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\sorttbls.nlp
...

The default domain is 0x0044a330 and its channel list, as shown in the previous !gcroot commands, is 01d30ffc. Let’s dump out its contents (again, this is similar to what we did previously in the client dump):

0:013> !do 01d30ffc
Name: System.Runtime.Remoting.Channels.RegisteredChannelList
MethodTable: 6d6cc1bc
EEClass: 6d4a2ee4
Size: 12(0xc) bytes
 (C:\Windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
6d6c4eec  4001f1d        4      System.Object[]  0 instance 01d30fd4 _channels
0:013> !da 01d30fd4 
Name: System.Runtime.Remoting.Channels.RegisteredChannel[]
MethodTable: 6d6c4eec
EEClass: 6d4aa8a0
Size: 24(0x18) bytes
Array: Rank 1, Number of elements 2, Type CLASS
Element Methodtable: 6d6cc204
[0] 01d30fec
[1] 01d30ecc
0:013> !do 01d30fec
Name: System.Runtime.Remoting.Channels.RegisteredChannel
MethodTable: 6d6cc204
EEClass: 6d507784
Size: 16(0x10) bytes
 (C:\Windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
6d6cc248  4001f1b        4 ...Channels.IChannel  0 instance 01d30f78 channel
6d6eb3e0  4001f1c        8          System.Byte  1 instance        3 flags
0:013> !do 01d30f78 
Name: System.Runtime.Remoting.Channels.CrossAppDomainChannel
MethodTable: 6d6cc0e4
EEClass: 6d507720
Size: 12(0xc) bytes
 (C:\Windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
6d6e84dc  4001f7e      664        System.Object  0   shared   static staticSyncObject
    >> Domain:Value  0044a330:01d30f84 004a4908:01d341cc 004bb1d0:01d4721c 004c3d18:01d4b4f4 004d4880:01d7171c <<
6d6ea584  4001f7f      668 ...ity.PermissionSet  0   shared   static s_fullTrust
    >> Domain:Value  0044a330:01d30f90 004a4908:01d341d8 004bb1d0:01d47228 004c3d18:01d4b500 004d4880:01d71728 <<
0:013> !do 01d30ecc
Name: System.Runtime.Remoting.Channels.RegisteredChannel
MethodTable: 6d6cc204
EEClass: 6d507784
Size: 16(0x10) bytes
 (C:\Windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
6d6cc248  4001f1b        4 ...Channels.IChannel  0 instance 01d19c9c channel
6d6eb3e0  4001f1c        8          System.Byte  1 instance        3 flags
0:013> !do 01d19c9c 
Name: System.Runtime.Remoting.Channels.Tcp.TcpChannel
MethodTable: 61f476f4
EEClass: 61eb5bec
Size: 24(0x18) bytes
 (C:\Windows\assembly\GAC_MSIL\System.Runtime.Remoting\2.0.0.0__b77a5c561934e089\System.Runtime.Remoting.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
61f47794  4000160        4 ....TcpClientChannel  0 instance 01d19ccc _clientChannel
61f4781c  4000161        8 ....TcpServerChannel  0 instance 01d19d04 _serverChannel
6d6eab0c  4000162       10         System.Int32  1 instance        1 _channelPriority
6d6e88c0  4000163        c        System.String  0 instance 01d19cb4 _channelName
0:013> !do 01d19d04 
Name: System.Runtime.Remoting.Channels.Tcp.TcpServerChannel
MethodTable: 61f4781c
EEClass: 61eb5cb4
Size: 68(0x44) bytes
 (C:\Windows\assembly\GAC_MSIL\System.Runtime.Remoting\2.0.0.0__b77a5c561934e089\System.Runtime.Remoting.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
6d6eab0c  4000192       2c         System.Int32  1 instance        1 _channelPriority
6d6e88c0  4000193        4        System.String  0 instance 01d19cb4 _channelName
6d6e88c0  4000194        8        System.String  0 instance 01d30900 _machineName
6d6eab0c  4000195       30         System.Int32  1 instance     5555 _port
6d6e73e4  4000196        c ....ChannelDataStore  0 instance 01d3092c _channelData
6d6e88c0  4000197       10        System.String  0 instance 00000000 _forcedMachineName
6d6eeadc  4000198       38       System.Boolean  1 instance        1 _bUseIpAddress
6ce5e13c  4000199       14 System.Net.IPAddress  0 instance 01d2d9e4 _bindToAddr
6d6eeadc  400019a       39       System.Boolean  1 instance        0 _bSuppressChannelData
6d6eeadc  400019b       3a       System.Boolean  1 instance        0 _impersonate
6ce622b8  400019c       34         System.Int32  1 instance        2 _protectionLevel
6d6eeadc  400019d       3b       System.Boolean  1 instance        0 _secure
6d6cfd5c  400019e       18 System.AsyncCallback  0 instance 01d30a78 _acceptSocketCallback
61f45ac8  400019f       1c ...emotingConnection  0 instance 00000000 _authorizeRemotingConnection
6d6eeadc  40001a0       3c       System.Boolean  1 instance        0 authSet
6d6e7434  40001a1       20 ...annelSinkProvider  0 instance 01d309c8 _sinkProvider
61f478e0  40001a2       24 ...rverTransportSink  0 instance 01d30a68 _transportSink
61f46478  40001a3       28 ...lusiveTcpListener  0 instance 01d30a98 _tcpListener
6d6eeadc  40001a4       3d       System.Boolean  1 instance        1 _bExclusiveAddressUse
6d6eeadc  40001a5       3e       System.Boolean  1 instance        1 _bListening

With respect to the dump of the client channel configuration, the thing that stands out here is that we have a tcp server channel, listening on port 5555. This is expected, since this is the .NET Remoting server.

In summary, the channel configuration for the default AppDomain looks good, but the channel configuration of the AppDomain hosting the remoting object Server.Obj does not.

We have enough information now to go and look at the code.

Finding the problem in the code

Since the configuration looks good in the default AppDomain and it does not in the AppDomain hosting the remoted object, we look at how the server creates the remoted object:

public IObj GetObj(string sName)
{
    return m_ObjsAppDomain.CreateInstanceAndUnwrap(Assembly.GetExecutingAssembly().FullName, typeof(Obj).ToString()) as IObj;
}

So, as we already knew from previous analysis, the object is created in a new AppDomain. GetObj() is a method of another remoted object, which lives in the default AppDomain and, therefore, works fine.

This is how the server registers the channels at start-up:

static void Main()
{
    TcpChannel tcpCh = new TcpChannel(Config.TcpPort);
    ChannelServices.RegisterChannel(tcpCh, false);
    ... 
}

This clarifies the reason of the issue: the code above registers the tcp channel in the current domain, which at start-up is the default AppDomain. When the AppDomain hosting the new remoting object is created, there is no registration of channels in that AppDomain.

You may wonder why the problem does not happen when the remoted object is created, but instead when the client calls one of its methods. This is because it is only at this stage that the channel is set up between the client and the server.

With this information, solving the problem is straightforward and can be done in one of 2 ways:

Always create remoting objects in the default AppDomain, so as to use the channel configuration for the default AppDomain
Do explicit channel configuration for each new AppDomain hosting remoting objects

Note: keep in mind that declarative channel configuration through config files only applies to the default AppDomain.

Summary

Here are some quick take-aways from this post:

In order to troubleshoot .NET Remoting-related exceptions like for example System.Runtime.Remoting.RemotingException you do not necessarily need to know the code of the application. A good knowledge of the runtime data structures used by .NET Remoting, along with dump analysis, can be enough
Channel registration is always per-AppDomain. If you are creating an AppDomain, you are responsible for registering server-side channels with it.
Imperative configuration in code applies to the current AppDomain. Declarative configuration in config files apply to the default AppDomain.
Start from System.Runtime.Remoting.Channels.RegisteredChannelList to look at registered channels for a given AppDomain at runtime.