July, 2006

Block ciphers are more popular than stream ciphers, with several either in active use or recent enough to require supporting for legacy interoperability. I'll talk about the RC2 and DES algorithms today and cover the newer AES algorithm tomorrow.

DES was a widely-used and analyzed cipher algorithm developed for use by the US government. DES is a 64-bit block cipher with a 64-bit key. However, 8 bits of the key are parity bits that are discarded by the algorithm so the effective key strength is 56 bits. DES works by expanding the contents of an input block by duplicating some of the bits, mixing the expanded input block with key bits derived from the encryption key, applying a non-linear transformation to the mixed bits, and finally permuting the output bits of the transformation. This process is called a round and the overall algorithm uses 16 rounds on each block.

Although there's no trivial way of breaking DES, the small key space means that brute forcing a message doesn't require that much computer power. A variant called triple-DES consists of the same algorithm applied with three different keys in succession. Due to some cryptoanalytic properties, triple-DES only doubles the number of effective key strength bits to 112. However, this is still a considerable leap over normal DES. All variants of DES are now deprecated in favor of using AES.

RC2 is another algorithm by Ron Rivest (there's an entire product line of these things) that intends to be very similar in application to DES. RC2 also is a 64-bit block cipher but has a variable key size of up to 128 bits. The variable key size allowed RC2 to be used in a degraded mode that met the US encryption restrictions at the time. RC2 is faster than triple-DES and includes a key salt to prevent precomputing large tables of encryption keys. RC2 has a similar round system to DES but uses two different types of rounds called mix and mash. The details of these rounds were published as RFC 2268. Both RC2 and RC4 were secret algorithms until their source code was anonymously leaked in the mid-1990s.

Next time: Advanced Encryption Standard

The list of commonly used stream ciphers is very short because there's really only one. RC4, developed by Ron Rivest, is essentially the only stream cipher that has been widely deployed. RC4 is very fast and found in wireless networking devices, as part of Wired Equivalent Privacy (WEP) and WiFi Protected Access (WPA), among other places. WEP does not offer much protection because a wireless network with lots of WEP traffic will reuse portions of the key stream if you listen patiently for a while. This is exactly what you weren't supposed to do with stream ciphers. This is more a poor use of RC4 rather than a flaw in the algorithm itself. WPA is a replacement for WEP that provides better security.

The RC4 key stream is generated from a 256 byte state array using a pseudorandom number generator. Each time random bits are generated, the state array gets slightly scrambled so that future output bears very little resemblance to past output. The state array is initialized by the encryption key. Encryption keys can be between 1 and 256 bytes long, with shorter keys essentially repeated to pad out the length. Initially, the state array has a fixed value. The algorithm is then initialized by looping over the repeated encryption key and scrambling the state array according to the key bit values. The encrypted stream is generated by taking the exclusive-or of the bits of the input stream with the randomly-generated bits.

Next time: More Symmetric Cipher Suites

Consider the following simple test server that uses no WCF code at all.

using System;
using System.Net;

class Server
{
   static void Main(string[] args)
   {
      HttpListener listener = new HttpListener();
      listener.Prefixes.Add("http://localhost:1000/");
      listener.Start();
      HttpListenerContext context = listener.GetContext();
      Console.Out.WriteLine(context.Request.Headers);
      context.Response.Close();
      listener.Stop();
      Console.ReadLine();
   }
}

Consider the following simple test client that uses no WCF code at all.

using System;
using System.Net;

class Client
{
   static void Main(string[] args)
   {
      HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create("http://localhost:1000");
      request.Accept = "*/*";
      request.UserAgent = "fooagent";
      request.Headers[HttpRequestHeader.MaxForwards] = "10";
      Console.Out.WriteLine(request.Headers);
      request.GetResponse().Close();
      Console.ReadLine();
   }
}

A funny thing happens on the way from the client to the server. If you look at the headers collection on the client, it contains all of the headers that we set. If you look at the headers collection on the server, it contains all of the headers that we set except for the user agent header. It turns out that HttpListener will always drop the user agent header (and only that header) from the header collection. Since our self-hosted HTTP transport uses the HttpListener header collection to populate the HTTP headers, that means you'll never see a user agent header even if a client sets one. You should be fine when running a service hosted in IIS because that has an entirely different mechanism for listening for HTTP requests.

Of course, we're not entirely free of header foibles of our own. For fun, try setting the Referer header on a WCF client.

Expect these things to just work when WCF is released.

Has this error message happened to you? It's because the lifetime of a message only lasts for one use. Once you've looked at the contents of a message, or copied the contents somewhere, you can't read the message again. This is a common problem encountered when people are trying to write a message inspector. Since you're expected to pass the message along after you're done inspecting it, it's quite likely that you'll need to make a new copy of the message. If you don't make a copy of the message, then the next person will have nothing to read.

This means that your message inspector code should look something like this:

public void AfterReceiveReply(ref Message reply, object correlationState)
{
   MessageBuffer buffer = reply.CreateBufferedCopy(MaxMessageSize);

   // Do something with the copied message

   reply = buffer.CreateMessage();
   buffer.Close();
}

When using symmetric encryption, repetition is the enemy of security. For the basic stream cipher and block cipher algorithms, an attacker can exploit repetition in either the input or key to gain information about the protected message.

Stream ciphers have a straightforward attack if you encrypt two messages with the same portion of a key stream. Due to the symmetry of encryption and decryption, you can apply the encrypted stream of one message with the encrypted stream of the other message and produce a predictable combination of the input streams. Depending on what you know about the structure of the messages, this may be enough to allow you to untangle the plaintext of each message. Of course, if the attacker controls one of the messages, then they can simply decrypt using the message text to directly obtain the encryption key. The solution to this problem is to never reuse a key stream and regenerate keys when the stream is about to repeat.

Block ciphers have a similar attack when a message contains the same block more than once. Since a particular block value in one message will always encrypt to the same value, it's possible to analyze the repetitions and learn something about the input text. This type of repetition is defeated by chaining the value of the current block of input text with the value of the previous encrypted block using some function. Someone that knows the encryption key can easily undo this process but it's otherwise very difficult to find repetitions because the same encrypted text can be decoded multiple ways depending on the previous blocks. There's nothing with which to chain the first block so many algorithms create a random salt to use for initialization. This salt value is then included in the message for decoding.

Even with these protections, it's still possible for two encrypted blocks to have the same value. The next pair of blocks is then susceptible to analysis because the input value being chained is the same in both cases. Fortunately, this is a rare occurrence with a large enough block size. It just so happens that we've already figured out how likely this is to occur.

Next time: Symmetric Cipher Suites

I still get a lot of hits for the series of documentation articles I did back in April and May. All of these articles are now in the Windows SDK, which is where you should go if you want to get the latest version with updates. I don't make any changes to the old entries. If you need the updated versions, you should get the SDK or check out the linked articles online. [Update] I should also mention that the community site has the absolute latest versions of these articles available as Documentation CTPs.

Making Sense of Transport Quotas. This was the first article I did about configuring quota settings and it continues to be the most popular in the series. It's now available as Transport Quotas.

How to: Enabling Streaming. After some revision this ended up just being a topic about streaming and dropped the "How to" portion. It's now available as Enabling Streaming. There's a separate placeholder for someone to eventually write the "How to" of it.

Configuring HTTP. The second most popular article in the series, this is really more of an overview of configuring the http.sys driver. It's now available as Configuring HTTP and HTTPS.

Configuring WCF for NATs and Firewalls. This started as a series of articles about NATs and firewalls that got merged together. It's now available as Working with NATs and Firewalls.

Using Impersonation with Transport Security. The first of two topics about transport security, it's survived essentially unchanged.

Understanding HTTP Authentication. The second of the transport security topics, it also is essentially unchanged.

Choosing a Transport. This was sitting around in several pieces until Clemens Vasters needed some material for a talk. I put together the article and liked it enough to add it to the SDK.

Yesterday I kicked off the topic of symmetric encryption by talking about block ciphers. Stream ciphers are another common pattern for symmetric encryption algorithm. Unlike block ciphers that operate on chunks of input text, a stream cipher operates on a byte-at-a-time basis using an input stream.  Actually, a stream cipher works using two data streams. The first data stream is the stream of input text. The second data stream is the stream of key data. The key data stream is generated by a function whose seed is the encryption key.

Encryption works by taking a byte (or similarly sized piece) from the input stream and a byte from the key stream and combining them using some function. Typically, that function is a really simple one, such as exclusive-or. Security is provided by making the key stream hard to guess rather than making the algorithm complex. Decryption works in reverse by taking a byte from the encrypted stream and a byte from the same key stream to return the byte from the input stream. This works because the generated key stream is always the same for a particular encryption key and the encryption function has a simple reverse operation (itself).

Stream ciphers are very simple but have more obvious weaknesses than the block ciphers we looked at yesterday. Stream ciphers are susceptible to targeted tampering of the message because changes to the text are narrowly scoped. Flipping a single bit in the encrypted text won't affect any of the surrounding text when the message is decoded. Flipping that same bit with a block cipher affects all of the bits in that block.

There are also several attack methods if you reuse portions of the key stream for multiple messages. Since an encryption key can only generate a limited amount of key stream, safely encrypting more text requires exchanging more encryption key bits. Most of the commonly used algorithms for symmetric encryption are block ciphers.

Next time: Symmetric Encryption Algorithm Design Issues

Back in May I gave a brief introduction to encryption and decryption. The next few posts are a short series on symmetric encryption algorithms, which use a shared secret key for both encryption and decryption. I've got a little bit of coverage about the general algorithms and issues with this type of cryptography before giving some examples of the actual algorithms that are used. The encryption algorithms I'll talk about come in two varieties, called block ciphers and stream ciphers.

Block ciphers split the input text into fixed-size chunks called blocks. Each block of the input is exactly the same length as all of the other blocks, for example 16 bytes. Since most input texts are not going to be an exact multiple of the block size, the encryption algorithm needs to pad the input text with some additional bytes to fill out the last block. The decryption algorithm needs to remove those pad bytes before returning the resulting text. Typically, the number of pad bytes is added to the message for this purpose, meaning that every message encrypted by a block cipher is between 1 and the size of the block bytes larger than before encryption. Small messages can be inefficient to transmit using block ciphers.

The operation of a block cipher is to take a block of input text and a block of key to produce a block of output text. You can think of this action as a table that contains all of the possible input text blocks as rows and all of the possible key blocks as columns, giving a value for every row-column combination. This table defines the function for encryption. There's then another table that defines the function for decryption. If you've got an input block M and a key block K, then the encryption function gives you an output block O. If you then look at row O, column K of the decryption function table, then the value of that entry will be M. As long as the two sides know the key blocks, they can exchange messages.

The weakness of this simple method is that if we use a fixed key block and parts of the message repeat, then the corresponding parts of the encrypted text will also repeat. When we look at symmetric encryption algorithm issues, this is one of the ones we'll try to address.

Next time: How Stream Ciphers Work

I thought I'd share the binding inspector program I wrote a few months ago to explore the different settings for the standard bindings series of articles.

You can access all of the articles in the series here:

• BasicHttp
• NetTcp
• NetNamedPipe
• WSHttp
• WSDualHttp
• WSFederationHttp
• NetMsmq
• NetPeerTcp

using System;
using System.Collections;
using System.Collections.Generic;
using System.IO;
using System.Reflection;
using System.ServiceModel;
using System.ServiceModel.Channels;
using System.Text;

namespace BindingInspector
{
   class PropertyNameComparer : IComparer<PropertyInfo>
   {
      public int Compare(PropertyInfo x, PropertyInfo y)
      {
         return String.Compare(x.Name, y.Name);
      }
   }

   class BindingInspector
   {
      static string NEVER = "***";
      static string NULL = "NULL";

      static bool freezeSeenProperties;
      static List<string> expandedProperties = new List<string>();
      static Dictionary<string,string> seenProperties = new Dictionary<string,string>();
      static List<Type> hiddenElements = new List<Type>();
      static StreamWriter writer;

      static void Main(string[] args)
      {
         expandedProperties.Add("ReaderQuotas");
         expandedProperties.Add("ReliableSession");
         expandedProperties.Add("MsmqTransportSecurity");

         hiddenElements.Add(typeof(AsymmetricSecurityBindingElement));
         hiddenElements.Add(typeof(TransportSecurityBindingElement));
         hiddenElements.Add(typeof(SymmetricSecurityBindingElement));

         writer = File.CreateText("basichttp.html");
         BasicHttpMain();
         writer.Close();
         writer = File.CreateText("nettcp.html");
         NetTcpMain();
         writer.Close();
         writer = File.CreateText("netnamedpipe.html");
         NetNamedPipeMain();
         writer.Close();
         writer = File.CreateText("wshttp.html");
         WSHttpMain();
         writer.Close();
         writer = File.CreateText("wsdualhttp.html");
         WSDualHttpMain();
         writer.Close();
         writer = File.CreateText("wsfederationhttp.html");
         WSFederationHttpMain();
         writer.Close();
         writer = File.CreateText("netmsmq.html");
         NetMsmq();
         writer.Close();
         writer = File.CreateText("netpeertcp.html");
         NetPeerTcp();
         writer.Close();
      }

      static void BasicHttpMain()
      {
         BasicHttpBinding binding1 = new BasicHttpBinding();
         binding1.Security.Mode = BasicHttpSecurityMode.None;

         BasicHttpBinding binding2 = new BasicHttpBinding();
         binding2.Security.Mode = BasicHttpSecurityMode.Transport;

         BasicHttpBinding binding3 = new BasicHttpBinding();
         binding3.Security.Message.ClientCredentialType = BasicHttpMessageCredentialType.Certificate;
         binding3.Security.Mode = BasicHttpSecurityMode.Message;

         BasicHttpBinding binding4 = new BasicHttpBinding();
         binding4.Security.Mode = BasicHttpSecurityMode.TransportWithMessageCredential;

         BasicHttpBinding binding5 = new BasicHttpBinding();
         binding5.Security.Mode = BasicHttpSecurityMode.None;
         binding5.MessageEncoding = WSMessageEncoding.Mtom;

         AnalyzeBindings(binding1, binding2, binding3, binding4, binding5);
      }

      static void NetTcpMain()
      {
         NetTcpBinding binding1 = new NetTcpBinding();
         binding1.Security.Mode = SecurityMode.None;

         NetTcpBinding binding2 = new NetTcpBinding();
         binding2.Security.Mode = SecurityMode.Transport;

         NetTcpBinding binding3 = new NetTcpBinding();
         binding3.Security.Mode = SecurityMode.Message;

         NetTcpBinding binding4 = new NetTcpBinding();
         binding4.Security.Mode = SecurityMode.TransportWithMessageCredential;

         NetTcpBinding binding5 = new NetTcpBinding();
         binding5.Security.Mode = SecurityMode.None;
         binding5.ReliableSession.Enabled = true;

         AnalyzeBindings(binding1, binding2, binding3, binding4, binding5);
      }

      static void NetNamedPipeMain()
      {
         NetNamedPipeBinding binding1 = new NetNamedPipeBinding();
         binding1.Security.Mode = NetNamedPipeSecurityMode.None;

         NetNamedPipeBinding binding2 = new NetNamedPipeBinding();
         binding2.Security.Mode = NetNamedPipeSecurityMode.Transport;

         AnalyzeBindings(binding1, binding2);
      }

      static void WSHttpMain()
      {
         WSHttpBinding binding1 = new WSHttpBinding();
         binding1.Security.Mode = SecurityMode.None;

         WSHttpBinding binding2 = new WSHttpBinding();
         binding2.Security.Mode = SecurityMode.Transport;

         WSHttpBinding binding3 = new WSHttpBinding();
         binding3.Security.Mode = SecurityMode.Message;

         WSHttpBinding binding4 = new WSHttpBinding();
         binding4.Security.Mode = SecurityMode.TransportWithMessageCredential;

         WSHttpBinding binding5 = new WSHttpBinding();
         binding5.Security.Mode = SecurityMode.None;
         binding5.ReliableSession.Enabled = true;

         WSHttpBinding binding6 = new WSHttpBinding();
         binding6.Security.Mode = SecurityMode.None;
         binding6.MessageEncoding = WSMessageEncoding.Mtom;

         AnalyzeBindings(binding1, binding2, binding3, binding4, binding5, binding6);
      }

      static void WSDualHttpMain()
      {
         WSDualHttpBinding binding1 = new WSDualHttpBinding();
         binding1.Security.Mode = WSDualHttpSecurityMode.None;

         WSDualHttpBinding binding2 = new WSDualHttpBinding();
         binding2.Security.Mode = WSDualHttpSecurityMode.Message;

         WSDualHttpBinding binding3 = new WSDualHttpBinding();
         binding3.Security.Mode = WSDualHttpSecurityMode.None;
         binding3.MessageEncoding = WSMessageEncoding.Mtom;

         AnalyzeBindings(binding1, binding2, binding3);
      }

      static void WSFederationHttpMain()
      {
         WSFederationHttpBinding binding1 = new WSFederationHttpBinding();
         binding1.Security.Mode = WSFederationHttpSecurityMode.None;

         WSFederationHttpBinding binding2 = new WSFederationHttpBinding();
         binding2.Security.Mode = WSFederationHttpSecurityMode.Message;

         WSFederationHttpBinding binding3 = new WSFederationHttpBinding();
         binding3.Security.Mode = WSFederationHttpSecurityMode.TransportWithMessageCredential;

         WSFederationHttpBinding binding4 = new WSFederationHttpBinding();
         binding4.Security.Mode = WSFederationHttpSecurityMode.None;
         binding4.MessageEncoding = WSMessageEncoding.Mtom;

         AnalyzeBindings(binding1, binding2, binding3, binding4);
      }

      static void NetMsmq()
      {
         NetMsmqBinding binding1 = new NetMsmqBinding();
         binding1.Security.Mode = NetMsmqSecurityMode.None;

         NetMsmqBinding binding2 = new NetMsmqBinding();
         binding2.Security.Mode = NetMsmqSecurityMode.Transport;

         NetMsmqBinding binding3 = new NetMsmqBinding();
         binding3.Security.Mode = NetMsmqSecurityMode.Message;

         NetMsmqBinding binding4 = new NetMsmqBinding();
         binding4.Security.Mode = NetMsmqSecurityMode.Both;

         AnalyzeBindings(binding1, binding2, binding3, binding4);
      }

      static void NetPeerTcp()
      {
         NetPeerTcpBinding binding1 = new NetPeerTcpBinding();
         binding1.Security.Mode = SecurityMode.None;

         NetPeerTcpBinding binding2 = new NetPeerTcpBinding();
         binding2.Security.Mode = SecurityMode.Transport;

         NetPeerTcpBinding binding3 = new NetPeerTcpBinding();
         binding3.Security.Mode = SecurityMode.Message;

         NetPeerTcpBinding binding4 = new NetPeerTcpBinding();
         binding4.Security.Mode = SecurityMode.TransportWithMessageCredential;

         AnalyzeBindings(binding1, binding2, binding3, binding4);
      }

      static void ResetSeenProperties()
      {
         seenProperties.Clear();
         freezeSeenProperties = false;
         seenProperties.Add("Max", NEVER);
         seenProperties.Add("Default", NEVER);
         seenProperties.Add("IdentityVerifier", NEVER);
         seenProperties.Add("MessageEncoding", NEVER);
         seenProperties.Add("MessageVersion", NEVER);
         seenProperties.Add("Name", NEVER);
         seenProperties.Add("Security", NEVER);
         seenProperties.Add("WriteEncoding", NEVER);
      }

      static void AnalyzeBindings(params Binding[] bindings)
      {
         ResetSeenProperties();
         StreamWriter tempWriter = writer;
         writer = StreamWriter.Null;
         foreach (Binding binding in bindings)
         {
            AnalyzeBinding(binding);
         }
         freezeSeenProperties = true;
         writer = tempWriter;

         Type bindingType = bindings[0].GetType();
         writer.WriteLine("<P><B>{0}</B></P>", bindingType.ToString());
         ReflectObject(bindings[0], "");
         ResetSeenProperties();
         foreach (Binding binding in bindings)
         {
            AnalyzeBinding(binding);
            freezeSeenProperties = true;
         }
      }

      static void AnalyzeBinding(Binding binding)
      {
         writer.WriteLine("<OL>");
         foreach (BindingElement element in binding.CreateBindingElements())
         {
            Type bindingElementType = element.GetType();
            writer.WriteLine("<LI>" + bindingElementType.ToString());
            if (!hiddenElements.Contains(bindingElementType))
            {
               ReflectObject(element, "");
            }
            writer.WriteLine("</LI>");
         }
         writer.WriteLine("</OL>");
      }

      static void ReflectObject(Object o, string indent)
      {
         bool writePreBlock = indent == "";
         bool firstProperty = true;
         Type type = o.GetType();
         PropertyInfo[] properties = type.GetProperties();
         Array.Sort(properties, new PropertyNameComparer());
         foreach (PropertyInfo info in properties)
         {
            object summary = info.GetValue(o, null);
            string lastPropertyText;
            seenProperties.TryGetValue(info.Name, out lastPropertyText);
            if (lastPropertyText == NEVER)
            {
               continue;
            }
            if (summary == null && lastPropertyText == NULL)
            {
               continue;
            }
            if (summary != null && summary.ToString() == lastPropertyText)
            {
               continue;
            }
            if (!freezeSeenProperties)
            {
               if (summary == null)
               {
                  seenProperties[info.Name] = NULL;
               }
               else
               {
                  seenProperties[info.Name] = summary.ToString();
               }
            }
            if (firstProperty && writePreBlock)
            {
               writer.WriteLine("<PRE>");
               firstProperty = false;
            }
            writer.Write("{0}: ", indent + info.Name);
            if (expandedProperties.Contains(info.Name))
            {
               writer.WriteLine();
               ReflectObject(summary, indent + "  ");
               continue;
            }
            if (summary == null)
            {
               writer.WriteLine("<NULL>");
            }
            else
            {
               writer.WriteLine(summary.ToString().Replace("\n", "\n" + indent + "  "));
            }
         }
         if (!firstProperty && writePreBlock)
         {
            writer.WriteLine("</PRE>");
         }
      }
   }
}

Index for bindings in this series:

• BasicHttp
• WSHttp
• WSDualHttp
• WSFederationHttp
  
• NetTcp
• NetPeerTcp
  
• NetNamedPipe
• NetMsmq
  

The last standard binding in the series is the NetPeerTcp binding for the PeerChannel transport. The peer transport is a multiparty network transport that sends messages across a mesh of peers. The NetPeerTcp binding requires installing the Peer-to-Peer networking service in Windows. Without this service installed, you cannot create an instance of the binding because it attempts to contact the peer name resolver.

Although the NetPeerTcp binding supports the same security settings, None, Transport, Message, and TransportWithMessageCredential, as the NetTcp binding, I'm going to break the usual pattern by not going over the modes individually. As far as I can tell, changing the security mode doesn't actually rely on changing the channel stack of the peer transport for configuration.

1. System.ServiceModel.Channels.PnrpPeerResolverBindingElement

   ReferralPolicy: Service
   

2. System.ServiceModel.Channels.BinaryMessageEncodingBindingElement

   MaxReadPoolSize: 64
   MaxSessionSize: 2048
   MaxWritePoolSize: 16
   ReaderQuotas: 
     MaxArrayLength: 16384
     MaxBytesPerRead: 4096
     MaxDepth: 32
     MaxNameTableCharCount: 16384
     MaxStringContentLength: 8192
   

3. System.ServiceModel.Channels.PeerTransportBindingElement

   ListenIPAddress: 
   ManualAddressing: False
   MaxBufferPoolSize: 524288
   MaxReceivedMessageSize: 65536
   Port: 0
   Scheme: net.p2p
   

The two parts of this binding are the transport element and the peer resolver element. The peer resolver is the portion that translates an ID for the peer mesh to a collection of endpoint addresses in the mesh. The remaining configuration settings on the binding are two related to the peer resolver and the standard ones common to all bindings.

CloseTimeout: 00:01:00
EnvelopeVersion: Soap12 (http://www.w3.org/2003/05/soap-envelope)
IsPnrpAvailable: True
Namespace: http://tempuri.org/
OpenTimeout: 00:01:00
ReceiveTimeout: 00:10:00
Resolver: System.ServiceModel.PeerResolvers.PeerResolverSettings
SendTimeout: 00:01:00

Next time: How Block Ciphers Work

The July CTP .NET Framework 3.0 and Windows SDK are now available for download. This release is primarily for people that want to track the bug fixes that have happened over the last month. I won't have a post detailing the exciting new features of this release because unlike the June CTP there aren't any. Tom Archer has the details about compatibility between the various releases of Vista, Visual Studio, and .NET.

Index for bindings in this series:

• BasicHttp
• WSHttp
• WSDualHttp
• WSFederationHttp
  
• NetTcp
• NetPeerTcp
  
• NetNamedPipe
• NetMsmq
  

It's been a while since the last article in the series on standard bindings, but there are only two left to go so I wanted to finish the series off. These last two bindings are a bit different than the others due to their specialization. Today is the standard binding for MSMQ and tomorrow is the standard binding for PeerChannel.

Microsoft Message Queuing (MSMQ) is a durable queue service that comes with Windows. MSMQ is the only transport included by default in WCF that is durable and that has asynchronous delivery of messages. You'll need to install message queuing support to actually operate your own queue, although you can create an instance of the binding even if the components are not present.

Standard binding standard disclaimer:

I've cut down on the number of properties presented by eliminating duplicates between the binding settings and binding element settings. For instance, the XML reader quotas can be set on either the binding or the message encoder binding element, but I'm only going to show them on the message encoder. I've also omitted most of the security credential settings because they're very messy and you hopefully won't need to change them much.

MSMQ supports both transport and message security and has the additional capability of being able to turn both on at the same time. I'll start though by showing what the binding looks like with security disabled.

1. System.ServiceModel.Channels.BinaryMessageEncodingBindingElement

   MaxReadPoolSize: 64
   MaxSessionSize: 2048
   MaxWritePoolSize: 16
   ReaderQuotas: 
     MaxArrayLength: 16384
     MaxBytesPerRead: 4096
     MaxDepth: 32
     MaxNameTableCharCount: 16384
     MaxStringContentLength: 8192
   

2. System.ServiceModel.Channels.MsmqTransportBindingElement

   CustomDeadLetterQueue: 
   DeadLetterQueue: System
   Durable: True
   ExactlyOnce: True
   ManualAddressing: False
   MaxBufferPoolSize: 524288
   MaxPoolSize: 8
   MaxReceivedMessageSize: 65536
   MaxRetryCycles: 2
   MsmqTransportSecurity: 
     MsmqAuthenticationMode: None
     MsmqEncryptionAlgorithm: RC4Stream
     MsmqProtectionLevel: None
     MsmqSecureHashAlgorithm: Sha1
   QueueTransferProtocol: Native
   ReceiveErrorHandling: Fault
   ReceiveRetryCount: 5
   RetryCycleDelay: 00:30:00
   Scheme: net.msmq
   TimeToLive: 1.00:00:00
   TransactedReceiveEnabled: True
   UseActiveDirectory: False
   UseMsmqTracing: False
   UseSourceJournal: False
   

This binding contains very few entries and all of the queuing configuration settings are available directly off of the transport binding element. There are no additional settings provided at the binding level besides the typical ones you see on all bindings.

CloseTimeout: 00:01:00
EnvelopeVersion: Soap12 (http://www.w3.org/2003/05/soap-envelope)
Namespace: http://tempuri.org/
OpenTimeout: 00:01:00
ReceiveTimeout: 00:10:00
SendTimeout: 00:01:00

The MSMQ transport channel provides transport-level security itself so there is no change to the channel stack when transport security is enabled.

1. System.ServiceModel.Channels.BinaryMessageEncodingBindingElement
2. System.ServiceModel.Channels.MsmqTransportBindingElement

Adding message security does put a new element in the channel stack, but since there's no change when transport security is enabled, the channel stack looks the same whether the security mode is Message or Both.

1. System.ServiceModel.Channels.SymmetricSecurityBindingElement
2. System.ServiceModel.Channels.BinaryMessageEncodingBindingElement
3. System.ServiceModel.Channels.MsmqTransportBindingElement

Next time: Inside the Standard Bindings: NetPeerTcp

Here's a cheat sheet for the different parts of the messaging framework and their responsibilities.

1. Hosting layer
   1. Hosting environment. Provides a process for the service to live inside. Standard choices are console applications, NT services, and IIS.
   2. Activation. The service is either always running or has a listener inside the hosting environment that activates the service when a message arrives. Activation solves the problem of restarting the service after it fails but adds the new problem of maintaining the service state between invocations.
2. Service layer
   1. Service contract. Defines a collection of logical operations that the service can perform. Each operation has a test that determines whether an incoming message can be interpreted for use with that operation.
   2. Proxy. Used by clients to map between the calls against the interface contract of the service and messages exchanged with the service.
   3. Dispatcher. Used by the service to map between calls against the interface contract of the service and messages exchanged with the service. The dispatcher is responsible for selecting the correct service operation to invoke when a message arrives.
   4. Message pump. Moderates the flow of messages from the channel layer to the service. The message pump needs to continuously accept new client connections and messages but throttle itself back when the service becomes overloaded.
   5. Service endpoints. Projects the service out to clients by providing an instance of the channel layer. Each endpoint project an independent instance allowing the same service to be associated with multiple channel stacks and networking configurations.
3. Channel layer
   1. Channel stack. A composable pipeline through which messages flow between the network and application. The channel stack can send its own protocol and infrastructure messages to hide the network configuration from the application.
4. Messaging layer
   1. Message encoder. Translates between the logical message format and a physical stream of bytes that can be transmitted across the network.
   2. Stream upgrades. An optional, client-initiated transformation of the physical stream of bytes before it is sent to the network transport.
   3. Transport. Creates or accepts client connections to the service. Sends or receives data across the network.

Next time: Inside the Standard Bindings: NetMsmq

On Tuesday, I left off the discussion of the messaging framework with the messaging layer, which sits beneath the hosting layer and service layer. The messaging layer is responsible for actually moving messages around. As you may have guessed from the name, the messaging layer is the part of the system that is message-oriented rather than service-oriented. The messaging layer can send and receive messages without regard to the contract or operations that your service has defined. The translation between messages and services happens inside the service layer. In WCF, the messaging layer is conceptually subdivided into a channel layer and a transport layer.

At the channel layer, messages are represented by an XML InfoSet container called Message. Channels have a top and bottom side that take an instance of Message and connect towards the application and network, respectively. When a message arrives at one side of a channel, the logic inside the channel, called the protocol, decides what gets emitted from the other side of the channel. The channel can emit 0, 1, or 37 messages (other values also permissible) in response to a single input message. Channels are selected for use with a service based on the protocol that they perform. We have channels for tasks such as security, reliable messaging, transactions, logging, message interception, and message chunking.

Collectively, all of the channels that make up an endpoint at the service layer are referred to as a channel stack. The top of the channel stack is the client or service and the bottom the channel stack is the transport layer. There are no specific responsibilities at the channel layer although a channel can modify the behavior of every service operation if it so chooses. This means that the channel layer has a lot of power but no preconceived notions about what to do with that power.

At the transport layer, messages are taken from the XML InfoSet format, converted into some byte representation, and transmitted to another party. On the other side of the transport, there is a receiver that captures the byte representation and transforms those bytes back into an XML InfoSet message. Performing the transformation between messages and bytes is handled by the message encoder while the transmission of bytes is handled by the transport channel.

Next time: Pieces of the Messaging Framework, Part 4

There are three transports that I tend to refer to as the "standard" transports of WCF: HTTP, TCP/IP, and named pipes. These transports receive a lot more attention than the other transports we've written, MSMQ and PeerChannel, although it's tough to get excited about named pipes.

Ignoring WCF for a moment, almost everyone that writes web applications uses HTTP as their transport. Almost everyone that doesn't use HTTP uses TCP/IP instead. There's a very long tail of other formats, ranging from tens of users to tens of thousands of users.

Now, let's stop ignoring WCF and start ignoring existing web applications. If you were writing an application that used any network protocol you wanted, what would you want to have support for? Well, it depends on what purpose you're building the application for. Let's say I want to write an application that streams some content to clients. Here's my list of five protocols I'd think about:

1. BitTorrent
2. Some instant messaging protocol
3. TCP/IP
4. Some cloud data storage service protocol
5. UDP (with multicasting)

Most of these protocols have good reach and connectivity. There's either a naturally low bar for establishing connections or someone else has a vested interest in providing a low bar. Very few of these protocols has a natural programming model that makes writing applications fun and easy.  HTTP doesn't even make the top five.

Assuming that we can solve the programming model problem, which protocols are the right ones to put in the toolbox?

Beneath the hosting layer sits the service layer. The hosting layer took care of mundane chores like reading configuration files and starting processes. The hosting layer ensures that the service has a good home to live in. Inside the service layer is a collection of clever tricks that give the illusion that building a distributed system does not require using clever tricks. There are 3, 4, or 5 clever tricks that make up the service layer depending on your minimal threshold for cleverness.

The service layer has a contract that describes how two parties interpret each other's messages. A service contract consists of a collection of operations. Each operation has its own operation contract. An operation contract is a test that determines whether the contents of a message are suitable for use with a particular logical operation. Contracts are used to get around the problem that everyone disagrees about how to authoritatively unify all of the world's type systems. The two sides don't have to agree on types as long as they both agree that any messages meet their interpretation of the contract.

The service layer has a collection of endpoints that describe various ways in which the service projects out into the world. The service contract of course does not actually exist outside of whatever program is using it. The sender and receiver can be using entirely different contracts as long as the operation tests defined by those contracts happen to have the same outcome for every message that the two sides try to exchange. What actually requires agreement is that the two sides have some way of sending each other messages over a networking mechanism. An endpoint specifies how a message moves from Point A to Point B without trying to explain what A meant when it sent the message to B.

The service layer has a proxy and dispatcher that makes it seem like the service contract actually exists. A proxy hides the fact that messages need to be exchanged when the client wants to perform one of the operations in the service contract. A dispatcher hides the fact that messages need to be exchanged when the server wants to implement one of the operations in the service contract. The proxy and dispatcher sides do not have to agree on the programmatic model for the operation as long as they agree on the outcome of the operation. The operation can have different names on each side. The operation can have different types on each side. The operation can have different invocation patterns on each side. There are many contracts that can mutually validate a particular sequence of messages.

The service layer has a pumping mechanism that hides the exchange of messages behind the contract. Something has to actually move the messages from the proxy through the endpoint and something has to equally move the messages through the endpoint to the dispatcher. The part that impels operations is the message pump. The conduit that the messages move through is the messaging layer, which is tomorrow's topic.

Next time: Pieces of the Messaging Framework, Part 3

There's no easy way to describe the division of responsibilities between layers of the framework in a WCF service. Although we encourage a particular structure for handling messages, we don't have enforcement to back that up. I'll talk for a few days about a four-layer model consisting of a hosting layer, a service layer, a channel layer, and a transport layer. These layers are both subdividable and collapsible. For example, you can partition the transport layer as finely as you like by dividing the responsibilities of that layer among any number of nested components. Likewise, you can collapse the service layer by directly reading untyped messages from the channel and performing your own dispatching. If you want to do such things, then you'll have to adjust the following description to your monstrous creation.

The hosting layer contains whatever magic necessary to get your service launched and running. Inside the service, you typically don't care at all how the service got started. That doesn't mean that you can pair any hosting environment and service together though. A service imposes requirements for activation and state management on the hosting environment. The service must start no later than the first message destined for the service. The service state must end no sooner than the last message using that state.

IIS 6 can listen for HTTP traffic and use the arrival of a message to start running a service. By carefully configuring the hosting environment, you can create a service that starts just in time to receive its first message. When the service stops running, IIS will start a new copy the next time it receives a message destined for that service. This method works fine when your service speaks HTTP and keeps no state around between messages.

If the service gets its first message over a named pipe, then this method wouldn't work. IIS will not be listening on the named pipe when the message arrives and so the service will never start. By putting your web service inside of an NT service instead, you would sidestep the problem of activating on the first message. An NT service starts immediately before any messages arrive. Your web service can listen for the first message using any mechanism it wants.

If the service needs to hold any state between invocations, then the method of directly hosting in IIS also wouldn't work. There's no guarantee that you're talking to the same service instance from moment to moment because IIS can decide to stop the service and later start a new copy. The hosting environment needs to expand to include some form of durable storage. Find somewhere else to store that data, such as in ASP.NET session state.

We'll now put aside the problem of process management and instead look at the problem of defining what the service does.

Next time: Pieces of the Messaging Framework, Part 2

The autoposter ate today's regular post so I decided to fill this space with a quick Q & A about the future.

Q: Now that the June CTP is out, what we can expect in the next big release for WCF?

A: Here's the timeline for the RC1 release: Vista RC1 is expected in late August and WCF will include its RC1 version in that release as well. However, you shouldn't expect to see any drastic changes past the June CTP. The June CTP was a snapshot of the RC1 release when it was already pretty far along. This is very similar to the relationship between the February CTP and the Beta 2 release. Over all of WCF, I would expect that there will be a small amount of API changes and a large amount of bug fixing. The majority of the effort on the messaging layer is going to be targeted at fixing clogs in the processing pipeline.

The semi-digested post will appear on Monday as if all of this never happened.

Continuing on the themes of changes and quotas, today's post is about how the connection pool quotas and settings moved for the connection-oriented transports. The last time I talked about connection pool quotas, these knobs lived on the binding elements of the transports. All of these knobs are still available in some form but there now is one level of indirection from the binding element.

We found that the names of the connection pool quota knobs were confusing people, especially the IdleTimeout quota that many people thought was a timeout for the transport rather than the connection pool. At the same time, there weren't any good names for these settings because any name that was clear enough to indicate that the setting was for the connection pool was going to be a really, really long name. We compromised by keeping the names of the connection pool settings short but moving those settings to a subobject of the binding element. The ConnectionPoolSettings property collects all of these settings in one place for each of the transports with a connection pool.

public sealed class NamedPipeConnectionPoolSettings
{
   public string GroupName { get; set; }
   public TimeSpan IdleTimeout { get; set; }
   public int MaxOutboundConnectionsPerEndpoint { get; set; }
}

public sealed class TcpConnectionPoolSettings
{
   public string GroupName { get; set; }
   public TimeSpan IdleTimeout { get; set; }
   public TimeSpan LeaseTimeout { get; set; }
   public int MaxOutboundConnectionsPerEndpoint { get; set; }
}

The old ConnectionLeaseTimeout is now called LeaseTimeout and the names of the other settings are unchanged.

Next time: Pieces of the Messaging Framework

A few weeks ago we spent some time thinking about the right way to think about quotas. This metathinking was triggered by a fix for a bug in the service host. The WCF service model layer, including the service host, sits on top of the messaging layer and provides a set of quotas that are similar but not at all the same as our quotas. The bug was that one of the service host quotas for a maximum number of sessions was not being enforced. Fixing that bug caused an ugly interaction with one of the transport quotas for a maximum number of connections. Sessions and connections are not the same thing but the two are linked in such a way that changing the quota for the number of sessions doesn't make a lot of sense unless you also pull along the quota for the number of connections.

The interaction occurred when people hit the quota for the maximum number of sessions. Under the covers, the service host is running a loop to handle as many connections as there are bubbling up from the messaging layer. When the session quota is exceeded, the loop stops and both the number of sessions and the number of handled connections stops growing. People with lots of processing power want to handle more sessions than our default quota values permit. If you were to increase the session quota though, you'd find that it didn't really help. That was because the number of connections also had a quota and even though the service host was trying to run its loop, there simply were no more connections coming up from the transport to create sessions from. This meant that you also had to change the quota for the number of connections to make all of this works, which kind of sucks. You have to fiddle with two knobs to solve one problem.

This lead to a slight tweak in the philosophy for providing quotas. There shouldn't be a quota when the movement of data is purely voluntary. The service host needs a quota because it would otherwise be running a continuous loop that moves more and more data without any intervention in your code. The transport doesn't need a quota because handling a connection is an active process. You have to call a method each time you want another connection. If you want to stop getting new connections, then you can just stop calling the method.

We took a look at other places with these loops and tried to make other cases that were similar have the same user experience.

• We removed the MaxInboundConnections quota for connection-oriented transports. The number of handled connections is something you control through your actions.
• We added a MaxPendingConnections quota for these transports. Connections pend without you taking any action so the number of pending connections needs a quota.
• We renamed the MaxConnectionsPendingDispatch quota on SMSvcHost to MaxPendingConnections for consistency.
• We added a MaxAcceptedChannels quota to the one-way channel I talked about the other day. This channel has a loop similar to the service host starting sessions so we need some way to decide when the loop should stop running.

Next time: Controlling Connection Pooling for TCP and Named Pipes

We spent a lot of time before the last release looking for parts of the framework that either didn't work or were unnecessary to get the job done. An amazing amount of cruft can build up in a design over time. A great idea or new feature is often accompanied by several supporting pieces of infrastructure to make the feature work. When that feature is later changed or removed, those supporting pieces often mutate into some other purpose rather than going away. Cleaning out those extra bits now means that we won't have to support them for years to come.

Here are ten bits of the channel framework you may or may not have noticed disappeared in June.

1. The InnerChannelListener property was removed from ChannelListenerBase. Use GetProperty instead to run queries on the channel stack at run time.
2. The Identity property was removed from IChannelListener. Use GetProperty<EndpointIdentity> to access this instead.
3. The MessageVersion property was removed from IChannelManager. Use GetProperty<MessageVersion> to access this instead.
4. The Scheme property was removed from IChannelManager. There were a few implementations that needed this property, and those specific channels still have a Scheme property on their factory and listener.
5. The Manager property of IChannel was removed. If you use the ChannelBase class, your implementation will still have a reference to the manager, but the property is now protected by default.
6. We got rid of everything on the IChannelManager interface so we just pitched the whole thing. There's still a ChannelManagerBase class for you to extend but that's purely for convenience.
7. The ChannelManagerBase class was slimmed down by removing the AbortChannels, CloseChannels, BeginCloseChannels, and EndCloseChannels methods. All of these methods were part of a channel lifecycle management system that clashed with our guidance for controlling channel resources. We've tried to not tie the lifetime of a channel up with the lifetime of its factory or listener.
8. The EventSender property and constructor were removed from CommunicationObject. This was a very infrequently used feature that was causing problems for some channel implementations.
9. The UnhandledBindingElements collection was removed from BindingContext. Add any information to the BindingParameters collection instead on the BindingContext and use Find or FindAll to search through the parameters later. This means there's only one place to search now instead of two.
10. Each binding element had a BindingElementConverter class for importing and exporting policy. Some time ago we moved policy exporting to the binding elements but never removed anything from the converter classes. All of the BindingElementConverter classes are now named BindingElementImporter and just support policy import.

There's one last piece that wasn't technically removed but somewhat fits in with this list. The Fault member of CommunicationObject was changed from public to protected. It was totally inadvertent for anyone but the implementer to have access to this.

Next time: Consistency for Timeouts and Quotas