Mahjayar's WebLog. : .Net Remoting

Implications of using IPCServerChannel when connection from remote clients

Mahjayar — Mon, 24 Sep 2007 07:06:00 GMT

In .Net 2.0 (Whidbey), Remoting introduced support for IPC channels. This was a quick way for local machine processes to communicate with objects hosted locally. Now there is a small behavior that users might not be aware of when they are unmarshalling an ObjRef from an remote appdomain that has an IpcChannelServer registered. As you might know when an object that implements MarshalbyRefObject is passed across appdomains (via an argument to a function call or as a return value) Remoting sends a ObjRef instead of the actual object. This ObjRef contains information on the unique Uri that the object is identified in the remote server and the various channels via which the remote object can be reached (in addition to other information). Now imagine the scenario where the Server has registered IpcServerChannel in addition to other channels (http and tcp) to enable local clients to communicate with the server. When a remote client gets any ObjRef from this Server it will unmarshal the ObjRef and as part of the process will create a Sink that the TransparentProxy for the ObjRef will use. The ObjRef contains a field called ChannelInfo that contains information on all server channels registered on the Server. The client iterates over the channel info list it received from the server over the list of channels registered in its local appdomain to see which client channel it can connect.

This ChannelInfo field always stores the list of available channels in descending order based on their priority. The priority is of a channel denotes how "preferred" is that channel as compared to the other registered channels. If two channels have the same priority then they are sorted based on the order they were registered. In Remoting, HttpServerChannel and TcpServerChannel both have a default priority of 1 while IpcServerChannel has a priority of 20. I am going to exclude the CrossAppDomainChannel, that is registered by default and has the highest priority of 64, from this discussion as that channel is automatically not considered when an Objref is located on a remote machine. So if the server has registered the following channels TcpServerChannel, HttpServerChannel and IpcServerChannel (in that specific order), the ChannelInfo data on that ObjRef will contain the following information

ChannelInfo[]

IpcChannelData

TcpChannelData

HttpChannelData

The sink creation process for an ObjRef works in the following way.

Iterate over list of received ChannelInfo data over list of registered client channels (sorted in descending order of their prioirty just like server channels) to see if any one of the channel can connect to the Uri pointed by the RemoteChannelData object.
If none of the registered channels can create a sink then Remoting will iterate over the list again to see if it can use one of the default client channels (by default Remoting will always register TcpClient, HttpClient and IpcClient if the user has not explicitly registered one).

Lets see how the sink creation process works on the client side for couple of scenarios.

Scenario 1:

Client has explicitly registered a TcpClientChannel and HttpServerChannel. It first passes Ipc channel Uri to TcpClientChannel/HttpClientChannel and will see that both cant connect to Ipc Uri. Then it will pick the next ChannelInfo data (Tcp in this case) and pass it to TcpClientChannel. TcpClientChannel will see that it can indeed connect to the Tcp Uri and create the sink and things works fine. Once Remoting finds a Sink then it will not continue to see if other channels can also be used. It will *always* use the first channel that recognizes the Uri.

Scenario 2:

Client has explicitly registered TcpClientChannel and IpcClientChannel (order doesn't matter as Ipc will always be ahead of Tcp because of its default priority) . It first passes the Ipc channel Uri to see if any client can connect to the Uri and it will return a Ipc sink as the IpcClientChannel will successfully create the sink. Remoting will quit looking for other Sinks and use the Ipc channel sink for all future communication. When you try to make any call on the TransparentProxy you will get the following error message.

Unhandled Exception: System.Runtime.Remoting.RemotingException: Failed to connect to an IPC Port: The system cannot find the file specified.
Server stack trace:
   at System.Runtime.Remoting.Channels.Ipc.IpcPort.Connect(String portName, Boolean secure, TokenImpersonationLevel impersonationLevel, Int32 timeout)
   at System.Runtime.Remoting.Channels.Ipc.ConnectionCache.GetConnection(String portName, Boolean secure, TokenImpersonationLevel level, Int32 timeout)
   at System.Runtime.Remoting.Channels.Ipc.IpcClientTransportSink.ProcessMessage(IMessage msg, ITransportHeaders requestHeaders, Stream requestStream, ITransportHeaders& responseHeaders, Stream& responseStream)
   at System.Runtime.Remoting.Channels.BinaryClientFormatterSink.SyncProcessMessage(IMessage msg)
Exception rethrown at [0]:
Reason being that Remoting Ipc channels cannot communicate with names pipes on remote machines. So you see that having an IpcClientChannel registered and unmarshalling an ObjRef containing an IpcServer Channel info is almost bound to fail if the ObjRef is from a remote machine.

One more associated problem with the server sending list of registered channels to the client will show up when you have a custom ServerChannel registered that is not known to the clients. In cases when a client tries to unmarshall the ObjRef from such a server it will fail to deserialize the ChannelIno as the custom ServerChannels ChannelDataStore will not be known to the client and you will get a serialization error such as

Unhandled Exception: System.Runtime.Serialization.SerializationException: Type 'Server.MyCustomData' in Assembly 'Server, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' is not marked as serializable.

Server stack trace:
   at System.Runtime.Serialization.FormatterServices.InternalGetSerializableMembers(RuntimeType type)
   at System.Runtime.Serialization.FormatterServices.GetSerializableMembers(Type type, StreamingContext context)
   at System.Runtime.Serialization.Formatters.Binary.WriteObjectInfo.InitMemberInfo()
   at System.Runtime.Serialization.Formatters.Binary.WriteObjectInfo.InitSerialize(Object obj, ISurrogateSelector surrogateSelector, StreamingContext context, SerObjectInfoInit serObjectInfoInit, IFormatterConverter converter, ObjectWriter objectWriter)
   at System.Runtime.Serialization.Formatters.Binary.WriteObjectInfo.Serialize(Object obj, ISurrogateSelector surrogateSelector, StreamingContext context, SerObjectInfoInit serObjectInfoInit, IFormatterConverter converter, ObjectWriter objectWriter)
   at System.Runtime.Serialization.Formatters.Binary.ObjectWriter.Serialize(Object graph, Header[] inHeaders, __BinaryWriter serWriter, Boolean fCheck)
   at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Serialize(Stream serializationStream, Object graph, Header[] headers, Boolean fCheck)
   at System.Runtime.Remoting.Channels.BinaryServerFormatterSink.SerializeResponse(IServerResponseChannelSinkStack sinkStack, IMessage msg, ITransportHeaders& headers, Stream& stream)
   at System.Runtime.Remoting.Channels.BinaryServerFormatterSink.ProcessMessage(IServerChannelSinkStack sinkStack, IMessage requestMsg, ITransportHeaders requestHeaders, Stream requestStream, IMessage& responseMsg, ITransportHeaders& responseHeaders, Stream& responseStream)

Solutions:

There are no straight forward way for solving the above mentioned problem. Some possible solutions that I can suggest are

To avoid your client from picking the Ipc channel you can increase the priority of your TcpClientChannels and HttpServerChannel to something more than 20 (so they are placed above the IpcChannel).
Or you can try the reverse of earlier suggestion and lower the priority of IpcClientchannel to 0.
Implement ITrackingHandler on the server and in your implementation of public void MarshaledObject(object obj, ObjRef or) {} you can loop over the ChannelInfo array on ObjRef and remove any ChannelInfo you dont want to send across. One draw back to this process is the fact that Remoting does not give you any way of determining the reason for marshalling the object. What I mean is there is no way of finding out if the marshalling is happening because of a call made by a remote client or a client residing on the local machine.

Custom pick a Remoting channel based on Remote object Uri

Mahjayar — Sun, 16 Sep 2007 02:18:00 GMT

One common question I see on Remoting forum is how to configure a proxy to use a particular channel from the set of registered channels for communicating when creating a proxy. For instance consider an Remoting application that exposes both a secure and non secure TcpChannel and expects clients for local box to use the unsecured channel and clients from other machine to use the secured one. Usually each channel has a priority based on which they are sorted in the app domain. When you register multiple channels with the same priority then they are added in the same order the user configured them. When a sink is to be prepared for a proxy request, Remoting will walk the registered list in descending order and find the first one that can serve the Uri scheme. So if you have multiple channels registered for the same transport medium you really don't have an option of choosing between them.

I thought about this and decided to write a small wrapper channel that will help me in achieving this. The solution is to register all your usual list of channels with this channel wrapper instead of registering them with ChannelServices. Then you register this wrapper channel with ChannelServices and so each time a Sink is to be prepared Remoting will call in to our channel wrapper and it will pick the channel based on user preference.

Lets take a look at this class which I call RemotingCompositeChannel.

public class RemotingCompositeChannel : IChannelSender, ITrackingHandler

{

//Implements IChannelSender to provide IMessageSink when requested

//Implements ITrackingHandler so we can wire up when an CAO proxy is unmarshalled

public RemotingCompositeChannel()

: this(UriComparisonMode.Exact)

{

}

public RemotingCompositeChannel(UriComparisonMode mode)

{}

CompositeChannel can be created in two Uri comparison modes, Exact and WildcardMatch and their function is to match a Uri exactly or do pattern matches. You first create an instance of the composite channel and register all of your regular Remoting client channels to this composite channel. Finally register the CompositeChannel with ChannelServices.

RemotingCompositeChannel compositeChannel = new RemotingCompositeChannel(UriComparisonMode.WildcardMatch);

HttpClientChannel httpChannel = new HttpClientChannel("http", null);

TcpClientChannel tcpChannel = new TcpClientChannel("tcp", null);

IpcClientChannel ipcChannel = new IpcClientChannel("ipc", null);

TcpClientChannel tcpSecuredChannel = new TcpClientChannel("tcpSecured", null);

tcpSecuredChannel.IsSecured = true;

//Wildcard pattern match mode

compositeChannel.Add(@"http://.*", httpChannel);

compositeChannel.Add("tcp://localhost:8889.*", tcpChannel);

compositeChannel.Add("tcp://localhost:8891.*", tcpSecuredChannel);

compositeChannel.Add("ipc://.*", ipcChannel);

//Register that single CompositeChannel

Console.WriteLine("Registered a compositechannel that contains multiple channels");

ChannelServices.RegisterChannel(compositeChannel, false);

RemotingCompositeChannel is set between IPCChannel Priority and CrossAppdomianChannel's priority so it sits on top of all other registered cross process/machine channels. Then in your client do your regular CAO/SAO activation stuff and you should see the right channel being picked.

Lets see a sample MarshalByRefObject definition

public interface IRemoteService

{

string SayHello(String s);

string SayAuthenticatedHello(String s);

}

public class MyRemoteService : MarshalByRefObject, IRemoteService { }

public class MyAuthenticatedRemoteService : MyRemoteService { }

public virtual string SayAuthenticatedHello(string s)

{

IPrincipal principal = Thread.CurrentPrincipal;

Console.WriteLine("Is User authenticated: " + principal.Identity.IsAuthenticated);

if (principal.Identity.IsAuthenticated)

{

Console.WriteLine("Authenticated user name: " + principal.Identity.Name);

Console.WriteLine("Authentication type: " + principal.Identity.AuthenticationType);

return "AuthenticatedSayHello: " + s;

}

else

{

return "NotAuthenticated";

}

SayAuthenticatedHello will print the authenticated username of the incoming connection if any.

Sample client usage:

//Register that single CompositeChannel

Console.WriteLine("Registered a compositechannel that contains multiple channels");

ChannelServices.RegisterChannel(compositeChannel, false);

Console.WriteLine("Point MyRemoteService activation to port 8889 which is unsecured.");

RemotingConfiguration.RegisterActivatedClientType(typeof(MyRemoteService), "tcp://localhost:8889/MyServer");

Console.WriteLine("Point MyAuthenticatedRemoteService activation to port 8891 which is secured.");

RemotingConfiguration.RegisterActivatedClientType(typeof(MyAuthenticatedRemoteService), "tcp://localhost:8891/MyServer");

MyRemoteService server = new MyRemoteService();

Console.WriteLine("Type of proxy:" + server);

Console.WriteLine("Calling anonymous method");

Console.WriteLine(server.SayHello("Hello"));

try

{

      Console.WriteLine("Calling authenticated method");

      Console.WriteLine(server.SayAuthenticatedHello("Hello"));

}

catch (Exception e)

{

      Console.WriteLine("Exception calling authenticated method." + e);

}

MyAuthenticatedRemoteService server1 = new MyAuthenticatedRemoteService();

Console.WriteLine("Type of proxy:" + server1);

Console.WriteLine("Calling anonymous method");

Console.WriteLine(server1.SayHello("Hello"));

try

{

      Console.WriteLine("Calling authenticated method");

      Console.WriteLine(server1.SayAuthenticatedHello("Hello"));

}

catch (Exception e)

{

      Console.WriteLine("Exception calling authenticated method." + e);

}

So ideally we want server object use the unauthenticated channel and the server1 object use the authenticated channel.

Output from the server:

Server started. Press any key to terminate
AnonymousSayHello : Hello
Is User authenticated: False
AnonymousSayHello : Hello
Is User authenticated: True
Authenticated user name: [MyDomainRemovedForSecuritySake]\mahjayar ==>Success. Used the authenticated channel
Authentication type: NTLM

Output From Client:

Registered a compositechannel that contains multiple channels
Point MyRemoteService activation to port 8889 which is unsecured.
Point MyAuthenticatedRemoteService activation to port 8891 which is secured.
Unmarshalled object Server.MyRemoteService
IS transparent Proxy
Type of proxy:Server.MyRemoteService
Calling anonymous method
Hello
Calling authenticated method
NotAuthenticated
Unmarshalled object Server.MyAuthenticatedRemoteService
IS transparent Proxy
Type of proxy:Server.MyAuthenticatedRemoteService
Calling anonymous method
Hello
Calling authenticated method
AuthenticatedSayHello: Hello

As you can see the first client's call to AuthenticatedSayHello comes back as unauthorized and the second client (of type MyAuthenticatedRemoteService) uses the authorized channel.

PS: The usual disclaimer applies the RemotingCompositeChannel.cs class attached to the post. Its my own code and not tested by Microsoft or clarifiers for support from Microsoft. Use them in your code as you wish and if you find any bugs or have any suggestions for improvement then please post back. The code has enough comments to explain the reasoning for each method. This should work for both CAO and SAO objects. I will leave the SAO object verification up to the user.

Update: Added a link to the file from my Skydrive account.

http://cid-1020651b1389d347.skydrive.live.com/self.aspx/Public/RemotingCompositeChannel.cs

Visual Studio 2005 SP1 Released

Mahjayar — Sun, 17 Dec 2006 22:14:00 GMT

Update: Turns out this SP was only for the Visual Studio tools part and not for the framework. Which means the QFE wont be in this. Sorry about that (Will confirm before posting the next time).

~~Over the weekend, Visual Studio team released SP1 for~~ ~~Whidbey. Usually when we release a SP, it will contain fixes for all the QFE's released till that point. This means that the fix for the Remoting bug that I talked~~ ~~about~~ ~~here~~ ~~should be available in the SP. Give the SP a try~~.

Maheshwar Jayaraman

Debugging Whidbey Remoting AccessViolation Problem

Mahjayar — Sat, 14 Oct 2006 21:12:38 GMT

There has been some cases where users have reported an AccessViolation when upgrading their Remoting app's to Whidbey. Some users found that the problem reprod only when they had some anti virus software (Nod32 in particular) installed and the AV went away when they configured the anti-virus not to scan the problematic exe's. In addition to working on WCF, I am also the dev servicing Remoting for all versions of the framework. We had never seen this problem in our internal testing and since there were enough cases out there to indicate that its not due to one single machine/configuration, we decided to see if there really was a problem.

Reproing the problem.

First we had to repro the problem and since most users said it was reproducible reliably with Nod32 anti virus installed, we installed a trial version of the software and wrote a simple remoting app that just echoes the value sent by the client. The client was able to connect to the server and have its input echoed but when the client shutdown, it caused the server process to AV.

Unhandled Exception: System.AccessViolationException: Attempted to read or write protected memory. This is often an indi
cation that other memory is corrupt.
at System.Net.UnsafeNclNativeMethods.OSSOCK.WSAGetOverlappedResult(SafeCloseSocket socketHandle, IntPtr overlapped, U
Int32& bytesTransferred, Boolean wait, IntPtr ignored)
at System.Net.Sockets.BaseOverlappedAsyncResult.CompletionPortCallback(UInt32 errorCode, UInt32 numBytes, NativeOverl
apped* nativeOverlapped)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverla
pped* pOVERLAP)

Debugging the problem:

Now that the issue was reproducible, we ran the server process in windbg and ran the repro again. With windbg attached the AV is caught as a first chance exception.

First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
eax=00000000 ebx=001872f0 ecx=011efa74 edx=00000000 esi=012a2498 edi=011efc48
eip=20b0cd29 esp=011ef9cc ebp=011efa00 iopl=0 nv up ei ng nz na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00010286
imon!NOD32Ioctl+0xaec9:
20b0cd29 8910 mov dword ptr [eax],edx ds:0023:00000000=????????

So Nod32 tried to save value of EBX register in EAX which means that the system expects the register to hold a pointer reference and not value 0. Just to be sure that the heap isnt corrupted, ran the !verifyheap command.

0:005> !verifyheap
-verify will only produce output if there are errors in the heap

VerifyHeap command reports a clean heap. Lets take a look at the stacktrace and unwind the stack.

0:005> !clrstack
OS Thread Id: 0x89c (5)
ESP EIP
011efa50 20b0cd29 [NDirectMethodFrameStandaloneCleanup: 011efa50] System.Net.UnsafeNclNativeMethods+OSSOCK.WSAGetOverlappedResult(System.Net.SafeCloseSocket, IntPtr, UInt32 ByRef, Boolean, IntPtr)
011efa6c 7a60e1ce System.Net.Sockets.BaseOverlappedAsyncResult.CompletionPortCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)
011efaa4 793d6ac4 System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)
011efc48 79e88f63 [GCFrame: 011efc48]
0:005> kb 100
ChildEBP RetAddr Args to Child
WARNING: Stack unwind information not available. Following frames may be wrong.
011efa00 00fc0a97 000005b4 012a2498 011efa74 imon!NOD32Ioctl+0xaec9
011efa38 7a60e1ce 012a2084 00000040 012a2498 CLRStub[StubLinkStub]@fc0a97
011efa98 793d6ac4 00000000 00000000 00000000 System_ni!System.Net.Sockets.BaseOverlappedAsyncResult.CompletionPortCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)+0xb2 [f:\RTM\ndp\fx\src\Net\System\Net\Sockets\_BaseOverlappedAsyncResult.cs @ 360]
011efab8 79e88f63 00000000 00000000 00000000 mscorlib_ni!System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)+0x68 [f:\RTM\ndp\clr\src\BCL\System\Threading\Overlapped.cs @ 85]
011efacc 79e88ee4 011efc80 00000001 011efc74 mscorwks!CallDescrWorker+0x33 [f:\rtm\ndp\clr\src\vm\i386\asmhelpers.asm @ 936]
011efb4c 79f20212 011efc80 00000001 011efc74 mscorwks!CallDescrWorkerWithHandler+0xa3 [f:\rtm\ndp\clr\src\vm\class.cpp @ 13568]
011efb6c 79f201bc 011efc7c 00000001 011efc74 mscorwks!DispatchCallBody+0x1e [f:\rtm\ndp\clr\src\vm\crossdomaincalls.cpp @ 504]
011efbd0 79f2024b 011efc7c 00000001 011efc74 mscorwks!DispatchCallDebuggerWrapper+0x3d [f:\rtm\ndp\clr\src\vm\crossdomaincalls.cpp @ 528]
011efc04 7a07bebf 011efc7c 00000001 011efc74 mscorwks!DispatchCallNoEH+0x51 [f:\rtm\ndp\clr\src\vm\crossdomaincalls.cpp @ 620]
011efcac 79ecb4a4 012a246c 001872f0 00000001 mscorwks!BindIoCompletionCallBack_Worker+0x123 [f:\rtm\ndp\clr\src\vm\comthreadpool.cpp @ 702]
011efcbc 79ecb442 011efd98 011efd44 79f93fe6 mscorwks!ManagedThreadBase_DispatchInner+0x4d [f:\rtm\ndp\clr\src\vm\threads.cpp @ 16923]
011efd50 79ecb364 011efd98 bdde0c59 001872f0 mscorwks!ManagedThreadBase_DispatchMiddle+0xb0 [f:\rtm\ndp\clr\src\vm\threads.cpp @ 16974]
011efd8c 7a0e1b7e 011efd98 00000001 00000000 mscorwks!ManagedThreadBase_DispatchOuter+0x6d [f:\rtm\ndp\clr\src\vm\threads.cpp @ 17085]
011efdb4 7a0e1bab 00000001 7a07bdfe 011efdec mscorwks!ManagedThreadBase_FullTransitionWithAD+0x25 [f:\rtm\ndp\clr\src\vm\threads.cpp @ 17107]
011efdc8 7a07c031 00000001 7a07bdfe 011efdec mscorwks!ManagedThreadBase::ThreadPool+0x13 [f:\rtm\ndp\clr\src\vm\threads.cpp @ 17149]
011efe1c 7a07c063 00000040 00000000 012a2498 mscorwks!BindIoCompletionCallbackStubEx+0x8c [f:\rtm\ndp\clr\src\vm\comthreadpool.cpp @ 777]
011efe30 79f2f3b0 00000040 00000000 012a2498 mscorwks!BindIoCompletionCallbackStub+0x13 [f:\rtm\ndp\clr\src\vm\comthreadpool.cpp @ 797]
011efe94 79ecb00b 00000000 badb0d00 00000000 mscorwks!ThreadpoolMgr::CompletionPortThreadStart+0x406 [f:\rtm\ndp\clr\src\vm\win32threadpool.cpp @ 4060]
011effb4 7c80b683 00183ff0 7c919a9c 7c800000 mscorwks!Thread::intermediateThreadProc+0x49 [f:\rtm\ndp\clr\src\vm\threads.cpp @ 3001]
011effec 00000000 79ecafc5 00183ff0 00000000 KERNEL32!BaseThreadStart+0x37 [d:\nt\base\win32\client\support.c @ 532]

Looks like Network Class Libraries (NCL) called in to unmanaged code WSAGetOverlappedResult and the call to that AV'd. Since the unmanaged call goes straight to Nod32 it means that its an LSP which sits on top of TCP. To confirm it, ran a query to see the list of installed LSP's on the system.

C:\Program Files\Microsoft Platform SDK\Samples\NetDS\WinSock\LSP>nonifslsp\XP32_RETAIL\instlsp.exe -p
Winsock 32-bit Catalog:
=======================
1019 - NOD32 protected [MSAFD Tcpip [TCP/IP]]
1020 - NOD32 protected [MSAFD Tcpip [UDP/IP]]
1021 - NOD32 protected [MSAFD Tcpip [RAW/IP]]
1022 - NOD32 protected [RSVP UDP Service Provider]
1023 - NOD32 protected [RSVP TCP Service Provider]
1001 - MSAFD Tcpip [TCP/IP]
1002 - MSAFD Tcpip [UDP/IP]
1003 - MSAFD Tcpip [RAW/IP]
1004 - RSVP UDP Service Provider
1005 - RSVP TCP Service Provider
1006 - MSAFD NetBIOS [\Device\NetBT_Tcpip_{EF980A36-42D5-4F46-8C5E-B9D289A5A7AE}] SEQPACKET 0
1007 - MSAFD NetBIOS [\Device\NetBT_Tcpip_{EF980A36-42D5-4F46-8C5E-B9D289A5A7AE}] DATAGRAM 0
1008 - MSAFD NetBIOS [\Device\NetBT_Tcpip_{14C32293-28C3-40B2-A4FF-FD3B9147252F}] SEQPACKET 1
1009 - MSAFD NetBIOS [\Device\NetBT_Tcpip_{14C32293-28C3-40B2-A4FF-FD3B9147252F}] DATAGRAM 1
1010 - MSAFD NetBIOS [\Device\NetBT_Tcpip_{D7376ECB-770A-4B34-8AFB-E553ADEA4B9A}] SEQPACKET 2
1011 - MSAFD NetBIOS [\Device\NetBT_Tcpip_{D7376ECB-770A-4B34-8AFB-E553ADEA4B9A}] DATAGRAM 2
1018 - NOD32

So Nod32 does indeed sit on top of Microsoft's Tcp provider and is routing calls. When the client disconnects then the server receives that socket shutdown and as part of that NCL tries to get the overlapped result on that socket. If you look at the disassembly at the place the AV happened you will find that its all in Nod32 code which is proprietary.

0:005> u 20b0cd29-10 20b0cd29+10
imon!NOD32Ioctl+0xaeb9:
20b0cd19 4c dec esp
20b0cd1a 241c and al,1Ch
20b0cd1c 8b4604 mov eax,dword ptr [esi+4]
20b0cd1f 5f pop edi
20b0cd20 8901 mov dword ptr [ecx],eax
20b0cd22 8b442420 mov eax,dword ptr [esp+20h]
20b0cd26 8b560c mov edx,dword ptr [esi+0Ch]
20b0cd29 8910 mov dword ptr [eax],edx ==> line where exception was thrown.
20b0cd2b 8b542424 mov edx,dword ptr [esp+24h]
20b0cd2f 8b4e08 mov ecx,dword ptr [esi+8]
20b0cd32 33c0 xor eax,eax
20b0cd34 890a mov dword ptr [edx],ecx
20b0cd36 8b4e08 mov ecx,dword ptr [esi+8]

So disassembling that code will not be possible and so lets take a step back and try to debug the next code in the stack before Nod32. Lets take a look at the arguments passed in to the WSAGetOverlappedResult call. The signature is defined in NCL as follows.

[DllImport(WS2_32, SetLastError=true)]
internal static extern bool WSAGetOverlappedResult(
[In] SafeCloseSocket socketHandle,
[In] IntPtr overlapped,
[Out] out uint bytesTransferred,
[In] bool wait,
[In] IntPtr ignored
);

Lets take a look at the invocation of this function in BaseOverlappedAsyncResult.CompletionPortCallback().

bool success = UnsafeNclNativeMethods.OSSOCK.WSAGetOverlappedResult(
socket.SafeHandle,
(IntPtr)nativeOverlapped,
out numBytes,
false,
IntPtr.Zero);

From the list of arguments lets try to rule out potentially problematic arguments. Socket.SafeHandle isnt null or NCL would have NullRef'd in managed code itself. In the debugger it was ensured that the nativeOverlapped structure wasn't null. numBytes is passed by reference and the the value is passed in to this function by _IOCompletionCallback.PerformIOCompletionCallback. The bool is passed by value and so that leaves the the only other variable IntPtr.Zero that could be the problematic argument. But this part of code hasn't been changed from V1 which means that if this is the issue then it should repro in V1 and V1.1 as well. Since its a Whidbey only issue may be something changed in IntPtr implementation from V1.1 and 2.0. Lets take a look at the struct IntPtr.Zero implementation in Whidbey.

unsafe private void* m_value;
public static readonly IntPtr Zero;

So we have memory allocated for the struct but there is no memory allocated for the the void pointer! This means that if someone tries to reference/dereference the value pointed by this IntPtr struct it will either throw NullRef in managed code or throw AV in unmanaged code. So this explains the AV in this case. So does this mean that the implementation has changed in V2.0 only? Lets take a look at the implementation of IntPtr.Zero in V1.1

public static readonly IntPtr Zero = new IntPtr(0);

Sure enough the implementation has changed. In v1.1 the struct was initialized with a pointer pointing to zero and hence when Nod32 tried to dereference the pointer it works fine. So next question is should Nod32 be dereferencing this pointer? For that lets see what MSDN documents about the WSAGetOverlappedResult method.

BOOL WSAAPI WSAGetOverlappedResult(
  SOCKET s,
  LPWSAOVERLAPPED lpOverlapped,
  LPDWORD lpcbTransfer,
  BOOL fWait,
  LPDWORD lpdwFlags
);
Parameters

s
[in] A descriptor identifying the socket. This is the same socket that was specified when the overlapped operation was started by a call to WSARecv, WSARecvFrom, WSASend, WSASendTo, or WSAIoctl.
lpOverlapped
[in] A pointer to a WSAOVERLAPPED structure that was specified when the overlapped operation was started. This parameter must not be a NULL pointer.
lpcbTransfer
[out] A pointer to a 32-bit variable that receives the number of bytes that were actually transferred by a send or receive operation, or by WSAIoctl. This parameter must not be a NULL pointer.
fWait
[in] A flag that specifies whether the function should wait for the pending overlapped operation to complete. If TRUE, the function does not return until the operation has been completed. If FALSE and the operation is still pending, the function returns FALSE and the WSAGetLastError function returns WSA_IO_INCOMPLETE. The fWait parameter may be set to TRUE only if the overlapped operation selected the event-based completion notification.
lpdwFlags
[out] A pointer to a 32-bit variable that will receive one or more flags that supplement the completion status. If the overlapped operation was initiated through WSARecv or WSARecvFrom, this parameter will contain the results value for lpFlags parameter. This parameter must not be a NULL pointer.

Turns out the flag cannot be a null pointer which means that NCL was passing the wrong argument for this. So the fix is to pass a valid pointer to the unmanaged API.

Now for users that want to try this out without having to install Nod32 (because you already have anti-virus installed) then you can use the LSP route. Windows SDK ships a LSP sample that you can install on your box on top of TCP and then run any Remoting app and sure enough you can find odd behaviors. I will leave that as an exercise for readers.

How to get the QFE:

We have issued an QFE for this and since it hasnt been through the requirements for a public KB yet this is only available by calling Microsoft product support. Dont worry this should be a free call. To get this QFE call Microsoft Product Support Services (http://support.microsoft.com) and ask for the Hotfix for KB number 923028 or mention that you need the fix for "Visual Studio Update QFE 4333" (Thats the internal bug # of this).

Maheshwar Jayaraman [WCF]

Configuring Kerberos delegation in Remoting

Mahjayar — Tue, 11 Jul 2006 23:00:00 GMT

One of the common question we have in remoting forums is how to configure Kerberos delegation with .Net 2.0. Rick Rainey has an excellent detailed post on how to configure this over at the remoting blog. Check it out!. Thanks Rick.

Maheshwar Jayaraman

Announcing the .Net Remoting team blog

Mahjayar — Fri, 07 Apr 2006 23:26:00 GMT

I am also part of the team that supports all versions of .Net Remoting. We found that lot of customers had simialr questions/concerns in migrating from .Net 1.1 to .Net 2.0 and we thought it would be great if we started a team blog for Remoting. We have our first post that talks about how to setup eventing in remoting. This has been a common question from customers and we thought this will be a good starting post. Blog is set up at http://blogs.msdn.com/dotnetremoting and the first post can be found here. Email capability is enabled and it comes directly to our team. We have a pretty quick turnaround time for answering (most cases same day) so fire those questions our way. If you have any ideas on what you want to see blogged about remoting then let us know.

Mahesh