(A small break from the current WCF extensibility series, as I faced this issue helping a customer this week and I thought it would be worth sharing)
In order to support transferring of large messages (e.g., uploading or downloading a large file), WCF added support (since its first version) to streaming message transfer – unlike the default behavior, which is to buffer the entire message prior to sending it to the wire (or delivering it to the application layer, in the receiving end). Buffering has lots of advantages – it is faster, it allows for operations such as signing the body (which needs the whole content), reliable messaging (which may need to retransmit a message, and streams can often be read only once). However, there are situations, such as uploading a 1GB file to a server, when buffering the message would be too memory-intensive, if at all possible. For these cases streaming can be used.
Enabling streaming is simply a matter of setting the appropriate transfer mode in the binding. BasicHttpBinding, NetTcpBinding and NetNamedPipeBinding expose that property directly. For other binding types, you’ll need to convert them to a custom binding and set that property on the transport binding element itself. But to really get the benefits of streamed transfer, you need to use some data types which don’t need to buffer all the information prior to being serialized. Using a Message object directly (untyped message programming) certainly can be done, as you control the whole message layout, and it can be created based on classes which can emit the message parts on the fly (such as a XmlReader or a BodyWriter), but that’s too low-level for most applications (you need essentially to created the request / parse the response almost “from scratch” – just a little above dealing with raw bytes).
Another parameter type which is naturally “suited” for streaming scenarios are types which implement IXmlSerializable. The contract for such types is that they essentially control their whole serialization / deserialization. On serialization, the class WriteXml method is called, receiving a XmlWriter positioned at the wrapping element. At that point, the class can write as much information as needed, without needing to buffer anything in memory. On the ReadXml method, the class receives a XmlReader positioned again at the wrapping element, and the class can read as much information can consume it without having to buffer it all (but it should only read information pertaining to itself). Below is a simple example of an IXmlSerializable type which produces / consumes 10000 elements in the message, without having to buffer it.
IXmlSerializable types aren’t very friendly for simple operations, as the user still has to write code to handle all the serialization. For simple scenarios such as uploading / downloading files, for example, it would be cumbersome to have to write the code to read from the stream / write to the stream and convert it into XML. To make those scenarios easier to implement, WCF exposes on the service model a few capabilities to help with streaming. For operation or message contracts, if you define a single body parameter of type System.IO.Stream, WCF will map it to the whole message body, and the operation can be defined fairly simply, like in the example below. As far as a WCF operation is concerned, a Stream type is equivalent to a byte operation (they both map to the XML schema type xs:base64Binary directly). Notice that the type needs to be Stream, not any of its subclasses (MemoryStream, FileStream, etc). When writing the message to the wire, WCF will read the stream passed by the user, and write its bytes out. When reading the message, WCF will create its own read-only stream, and pass it to the user.
Notice the restriction of one single body parameter, of type Stream – that’s when WCF will map the entire message body to the stream parameter. For most of the cases, this can be resolved by simply changing the operation signature and moving some parameters from the operation signature to the message header, which is the case of the UploadFile above – the operation would normally be defined as “void UploadFile(string fileName, Stream fileContents)”, but at that point both there are two parameters mapped to the body and the default (easy) mapping falls apart. But changing the operation to a MessageContract-based one is quite straightforward, and it solves most of the problems.
But the recommendation of changing the operation contract to make it WCF-happy many times doesn’t work with interoperability scenarios. In such cases, we want to consume an existing service provided by a 3rd party which we have no control over. At that point, we’re stuck with creating a message in the exact format as requested by the service. If the service expects the file contents (or any large binary data, for that matter) to be sent as part of a data contract graph, the proxy generated by svcutil / add service reference will contain such data contracts or types decorated with XML attributes (e.g., XmlType, XmlRoot, etc) and one of those types will contain a byte member. Byte arrays are buffered by nature (the array is stored in memory), and switching it to a Stream type won’t work – in the middle of the data contracts the control is handled to the serializers, and they don’t know how to handle Stream parameters like the WCF service model.
So we’re back to the no-Stream support case. Again, we can handcraft the message ourselves, but that usually requires a lot of effort (creating the XML “by hand”). The other alternative, IXmlSerializable is recognized by the serializers (both the DataContractSerializer, and the XmlSerializer, the two “default” serializers for WCF), so that’s where we can go. The main advantage of going the IXmlSerializable way is that we only need to change the class which contains the byte field, while the others (any wrapping types) can be left untouched. The task now is to implement the WriteXml and ReadXml methods in a way that they reflect the same contract as the “default” behavior for those classes.
To explain how to change one such class, I’ll define a simple service which contains a binary (byte) member nested in a data contract. I’ll use the XmlSerializer ([XmlSerializerFormat]) since that’s the serializer mostly used in interop scenarios, but the steps should be the same for DataContractSerializer types as well. Here’s the service:
After running the service, I can point add service reference or svcutil to it to generate my proxy code:
This generates the following proxy code:
Now, to the conversion. Unfortunately, there’s no automated way (AFAIK) to do the conversion, so we have to do it “by hand”. For me, the easiest way is to actually call the method once (using a small byte array value which shouldn’t have any memory issues) while monitoring the request on Fiddler (my favorite HTTP monitoring tool), and see what WCF actually wrote. By doing that we’ll see what the service expects:
And that request shown in fiddler is the following:
Now that we know what the request looks like, we can start converting the Content class to IXmlSerializable. The generated proxy classes are marked as partial, which will actually help us isolate the IXmlSerializable code in a separate file (it’s a matter of personal preference, but I prefer to keep the generated proxy as unchanged as possible, in case I need to regenerate it). Let’s focus first on write (sending byte / Stream parameters to the server). Most data types are written to XmlWriter using its WriteValue method (it has many overloads), while binary data is written using the WriteBase64 method. Here’s a first attempt of implementing IXmlSerializable on that type:
It builds fine. But when I try to run it, it fails with the exception below:
System.InvalidOperationException: There was an error reflecting type 'RequestClass'. ---> System.InvalidOperationException: There was an error reflecting property 'content'. ---> System.InvalidOperationException: There was an error reflecting type 'Content'. ---> System.InvalidOperationException: Only XmlRoot attribute may be specified for the type Content. Please use XmlSchemaProviderAttribute to specify schema type.
The problem is that the attribute [XmlType] applied to the type Content in the generated file is conflicting with the IXmlSerializable implementation. XmlType is used to control the XML schema of a type, and IXmlSerializable has its own way of providing schema. This can be fixed by commenting out the attribute on the generated file:
Now it builds fine, it runs fine. Looking at the request in Fiddler, it is exactly the same as the previous one, so we’re in the right track. It’s possible that it had some differences (mostly likely due to things such as XML namespaces), in this case we’d have to go back and update the writing logic. We haven’t changed anything yet, but now we can move on to adding a stream parameter. First, let’s define a Stream property which will be used instead of the byte one:
Now we can change the WriteXml implementation. We can still use a small buffer to avoid writing bytes 1-1, but it will certainly be smaller than many MBs of large files.
And the Write part is done.
Before we move on to the Read part, let me just spend some time discussing about MTOM. MTOM is a W3C recommendation which defines an optimization for transmitting binary data in SOAP messages. Typically, binary data is base64-encoded and written inline in the SOAP body. This is fine for many scenarios, but it has the drawback that base64-encoding increases the data size by ~33%. MTOM optimizes that by sending large binary data not inlined in the message, but as MIME attachments instead. They have a small overhead for some MIME headers, but the binary data is sent as-is, without any encoding, so the message doesn’t have the 33% size penalty.
In WCF, MTOM is implemented by an implementation of the XmlWriter abstract class (the internal class XmlMtomWriter). Whenever the user (or WCF code itself) writes binary data (WriteBase64), the MTOM writer will hold on to the data written and emit it as attachments after the whole message body has been written. By holding on to the data, the writer is essentially buffering it (since it needs to do it, as it needs to finish writing the message prior to writing the attachments), so the code above for WriteXml won’t do what we want in MTOM – it will still buffer the stream contents!
The solution for this problem is to hand over the Stream itself to the MTOM writer, and let it hold on to it until it’s ready to consume it. This is done via the IStreamProvider interface, and the XmlDictionaryWriter (a class derived from XmlWriter which is used internally in WCF) adds a new overload to WriteValue which takes an IStreamProvider parameter. In normal (i.e., non-MTOM) writers, this implementation simply copies the stream to the writer by calling WriteBase64 (as we did in our initial WriteXml implementation). The MTOM writer, however, holds on to the stream and will only consume it when it needs to.
This is the MTOM-aware version of the WriteXml method now:
And now the Write part is really done.
In order to test the reading part I’ll update the test server with an extra operation which returns a byte field nested inside a data contract graph.
Updating the client to use it: re-generated the proxy with svcutil/ASR, comment out the [XmlType] attribute on the Content class, and update the Main method:
And it fails when run:
System.ServiceModel.CommunicationException: Error in deserializing body of reply message for operation 'GetResponse'. ---> System.InvalidOperationException: There is an error in XML document (1, 363). ---> System.NotImplementedException: The method or operation is not implemented. at Content.ReadXml(XmlReader reader)
Now we need to implement ReadXml. The idea is similar to the WriteXml: look at the response on fiddler
And start reading from the XmlReader into the class. Notice that the reader is positioned at the wrapping element (<content>), so to access the fields we need first to read past it. After that we can start reading the data fields themselves:
This works fine, but at that point we’re still buffering the response. Unfortunately we can’t avoid it – since it’s in the middle of the body (not the whole body), WCF needs to read all of the content data to be able to finish reading the rest of the message. If this is a problem, however, we can shift the buffering from memory to another location – such as disk – which can hold on to very large structures a lot easier. Here’s the modified code. Notice that I didn’t expose the file stream as a property, but as a method instead, to make it clearer that it’s not just retrieving the value of a field, but it’s doing something else instead.
Hope this helps!
[Code for this post]