Why .NET Compact Framework fails to call some HTTPS web servers

A bug was discovered recently in the .NET Compact Framework version 2.0 (SP2 and earlier) and 3.5 that causes HttpWebRequest's directed at some HTTPS servers to fail with this error: (for web services)

System.Net.WebException: Unable to read data from the transport connection. ---> System.Net.Sockets.SocketException: Unknown error (0x0).
   at System.Net.HttpWebRequest.fillBuffer(HttpWebRequest request, Connection connection, CoreResponseData data)
   at System.Net.HttpWebRequest.getLine(HttpWebRequest request, Connection connection, CoreResponseData data)
   at System.Net.HttpWebRequest.parseResponse(HttpWebRequest request, Connection connection, Boolean defaultKeepAlive)
   at System.Net.HttpWebRequest.startReceiving(Connection connection)
   at System.Net.Connection.startReceiving(Object ignored)
   at System.Threading.ThreadPool.WorkItem.doWork(Object o)
   at System.Threading.Timer.ring()

   at System.Net.HttpWebRequest.finishGetResponse()
   at System.Net.HttpWebRequest.GetResponse()
   // ...
   at Program.Main(String[] args)

For standard HttpWebRequest calls from your app the stack trace would be different, although the exception message and cause would be the same.

While there are multiple reasons this exception can be produced, one thing that can cause it is if the server sends an empty encryption packet to the NetCF client.  Here is an illustration of the encryption process that occurs on the server for HTTPS:

First, a memory buffer is initialized with unencrypted data that the server wants to send to the client over the network.

Next, the server makes a call to EncryptMessage and the unencrypted data is encrypted in place.  A header and footer is added to the buffer:

This header if 5 bytes long, and the footer is several more bytes.  This entire packet, which is necessarily longer than the original unencrypted data, is then sent down the wire where the process is reversed using a call to DecryptMessage.

The trouble with NetCF's SSL stack is when the server encrypts a buffer of zero length and sends it to the client:

Although the original data had a length of zero, the encrypted version of it actually has a non-zero length.  When this packet is sent to a NetCF client, current versions of NetCF decrypt the packet and return a zero length buffer to the caller.  The semantics of a network Read method is traditionally that it blocks until some data is received, and if a zero length buffer is returned it is the sign that the socket was closed.  Because NetCF will return an empty buffer after decrypting an "empty" encryption packet, the caller may misinterpret this as a sign of a disconnected socket and terminate the connection.

In fact this is exactly what happens in NetCF's web services code when calling services over SSL that respond with empty encryption packets.  As a result the connection fails before the response is fully received and an exception is thrown.

What causes the server to send these empty encryption packets?  Technically it's legally allowed for these empty packets to be sent (although they are pointless), so depending on your server and its configuration this may never happen or it may happen regularly.

Unfortunately there is no way to get NetCF to ignore these packets, so the only workarounds are these, in order of simplicity:

1. Don't use SSL to make your request (security implications here of course by sending your request and its response in clear text)
2. Reconfigure your existing server in some way to get it to avoid generating these empty encryption packets.
3. Build a new web server that will forward your device's requests on a separate connection to the ultimate target server and forward the responses back to the device.  This would work for web sites and web services.  But this new front-end server (which serves something of the role of a proxy server) would have to be configured to not generate these empty encryption packets.
4. [Added 1/2/08] Build a device-side web proxy that calls into the SSPI functions itself (native would probably be easier than managed for this).  Then have your managed app call into the proxy.  This proxy would be responsible for consuming the empty packets and re-encrypting everything (if necessary to secure IPC) for the app on the same device without any empty SSL packets.
5. Wait until a future version of NetCF that fixes this bug.
6. Write your own HTTPS client using native code or P/Invoke'ing from NetCF (difficult to get right).

So how do you know for sure if the HTTPS server is sending empty encryption packets that are causing your NetCF clients to fail?  I wrote a native C++ app that calls a web server on an HTTPS connection and detects these packets.  (No, I'm afraid it can't filter them out for NetCF clients) By running this app against your web server the app will tell you whether your server generates these empty packets.  At least then you know what's going on.

[Updated Feb 1, 2008] You can download the source code from its resource page on the MSDN Code Gallery.