Update to HttpClient and Automatic Decompression

Update to HttpClient and Automatic Decompression

Rate This
  • Comments 13

Two weeks ago, we published a beta of HttpClient that provides support for automatic decompression. Since then, we’ve received a fair amount of feedback, especially around the fact that it relies on a native implementation of compression that isn’t available on the same platforms as HttpClient.

Today, we’ve published another beta of HttpClient that will address those concerns.

HttpClient no longer depends on Microsoft.Bcl.Compression

In the last post, we explained why Microsoft.Bcl.Compression depends on native code. To recap, we are using the same implementation of DeflateStream that we shipped in the .NET Framework 4.5. This implementation uses the de facto standard zlib, which is a native library. Unfortunately, Windows Phone 7.1 and Silverlight don't support native code (Windows Phone 8 does), which means that we couldn’t support automatic decompression on those two platforms.

To enable automatic decompression in HttpClient, we had three options:

  1. Add a dependency on Microsoft.Bcl.Compression that doesn’t use native code
  2. Add a dependency on Microsoft.Bcl.Compression that uses native code
  3. Add the (managed) decompression code directly into HttpClient

A main benefit of using the native zlib library is the quality of compression that it provides. It’s not that we couldn’t implement a good compression algorithm in managed code – the problem is that we simply don’t have one and writing one would take considerable resources and a lot more time. So we decided early on not to pursue this option for now, which leaves us with options 2 and 3.

Decompression is still implemented in managed code (both in the .NET Framework 4.5 and in the Microsoft.Bcl.Compression package). Since HttpClient only needs to decompress data, it doesn’t actually depend on anything that is implemented in native code, which makes option 3 an interesting candidate.

However, from an engineering perspective option 2 is cleaner, as there is no code duplication but it comes at the expense of dropping support for platforms that don’t support native code. Previously, we believed it would be better to invest in a good long term solution than to provide a less ideal implementation.

Based on the feedback we received, we decided to revisit that decision and after careful consideration, we decided that the best approach is to remove the dependency on Microsoft.Bcl.Compression and add an internal copy of the managed decompression algorithms to Microsoft.Net.Http. This way, we can support automatic decompression on all platforms.

This doesn’t change Microsoft.Bcl.Compression in any way – it still requires native code.

Using automatic decompression on Silverlight

Before we explain how Silverlight supports automatic decompression let’s recap how automatic decompression works generally in HTTP: The client adds an HTTP request header called Accept-Encoding that specifies which compression algorithm it supports. In the response, the server can either choose to ignore that information and return an uncompressed response or compress the data in one of the algorithms the client requested. The server indicates which algorithm it used via the Content-Encoding response header. For an example with more details, see our earlier blog post on automatic decompression.

Silverlight has two networking stacks: the browser networking stack and the client networking stack. The browser networking stack supports automatic decompression. It always sets the Accept-Encoding header on each request and automatically decompresses the response.

The client networking stack doesn’t have support for setting the HTTP Accept-Encoding header, but throws an exception if you try to do this manually. If you control the server, you can define your own way of activating compression, for example, by using a different HTTP header, by using a query parameter, or by simply compressing all responses. As long as the server sets the Content-Encoding header, HttpClient can automatically decompress the contents.

Now let’s have a look at what that means for the automatic decompression APIs in Silverlight. The SupportsAutomaticDecompression property will return true. The AutomaticDecompression property is set as follows:

  • Browser stack. The value is ignored. All requests will have set the Accept-Encoding header to indicate they accept both gzip as well as deflate. Responses are automatically decompressed using the appropriate algorithm.
  • Client stack. The value is ignored when making requests, i.e., the Accept-Encoding header will not be set. However, the Content-Encoding header will be honored when reading responses, i.e., responses are automatically decompressed using the algorithm specified by the Content-Encoding response header.

The route to RTM

Our previously announced, but tentative, schedule looked like the following. We’ve since added about 2 weeks to these dates to accommodate the changes that I’ve described above.

  1. End of June Early July: RC of HttpClient 2.2 with automatic decompression
  2. Mid July Mid July (depending on feedback): RTM of HttpClient 2.2 with automatic decompression

This schedule was based on the assumption that we wouldn’t make major changes to the features of HttpClient. Since we changed the way compression is handled, we decided to ship another beta before an RC release.

This doesn’t change the schedule drastically but we expect to delay both the RC as well as the RTM by about two weeks. We want to give you as well as our partners enough time to absorb the changes and provide feedback.

Summary

The new version of HttpClient no longer depends on Microsoft.Bcl.Compression. Automatic decompression is now supported on all platforms and also allows the application to remain in the AnyCPU architecture.

Microsoft.Bcl.Compression is unaffected by this change; it still isn’t supported on Windows Phone 7.1 or on Silverlight because it requires support for native code.

Please let us know what you think about this update.

Leave a Comment
  • Please add 8 and 4 and type the answer here:
  • Post
  • Magnificent. Great work guys.

  • I like this, I like this a lot.  Thanks for listening.

  • Great work with delivering a real the portable implementation with compression!

  • "The client networking stack doesn’t have support for setting the HTTP Accept-Encoding header, but throws an exception if you try to do this manually"

    I don't get it. How is SharpGIS.GZipWebClient able to set the header then?

    github.com/.../GZipWebClient.cs

    Or maybe it's something that would work only for Windows Phone and not for the desktop Silverlight version?

  • "Silverlight has two networking stacks: the browser networking stack and the client networking stack. The browser networking stack [..] always sets the Accept-Encoding header on each request and automatically decompresses the response. The client networking stack doesn’t have support for setting the HTTP Accept-Encoding header [..]"

    Forgive me if this has been explained before, but why is this so? What's the engineering rationale for having two stacks that have diametrically opposite behavior with regards to compression?

  • Guys,

    When will you add support for Mono for Android e MonoTouch for Bcl.Build?

    "Could not install package 'Microsoft.Bcl.Build 1.0.7'. You are trying to install this package into a project that targets 'portable-win+net45+MonoAndroid16+MonoTouch40+sl40+wp71', but the package does not contain any assembly references or content files that are compatible with that framework. For more information, contact the package author."

  • Thanks for accommodating older tech.

  • @Richard, @Paul, @Daniel, @fourbadcats: Thanks for the kind words, very much appreciated!

    @KooKiz: The restriction we mentioned applies to "real" Silverlight, not the phone version of Silverlight.

    @Jeroen: That's an excellent question. Quoted from this blog post (blogs.msdn.com/.../new-networking-stack-in-silverlight-3.aspx):

    In previous versions of Silverlight, the browser handled all the HTTP communication for the Silverlight plug-in. The browser HTTP stack works well but has some limitations including the HTTP methods you can use and the response codes that are accepted. In Silverlight 3, you can opt-in to have the Silverlight client perform the HTTP handling. By opt-in, I mean, if you don't specifically choose the client to perform HTTP processing, it will be done by the browser, just like it always has been. However, if you decide you need to call REST services, you can easily do this with Silverlight3 by opting in to the client HTTP stack.

    @Mário: At this point we don't have any plans to make our binaries available to Mono for Android or MonoTouch. In fact, our license doesn't even allow usage at this point. However, we are working with Xamarin to clarify how Portable Class Libraries are interoperating with the Mono stacks.

  • And what about using the native decompression on platforms, where it's available, and managed decompression on other platforms? That could surely be achieved, no?

  • Great, excellent work!

  • Is there a posibility that in future versions allow full access to all cookies in a request? I need access to httponly cookies and CookieContainer "black box" is really hurting me in my project, because I have to deal with non rfc compliant servers and current HttpRequest inmplementantion ignores these cookies.

  • visualstudio.uservoice.com/.../13995623-visual-studio-team

  • How to implement auto decompression and compression for POST method in .Net 4.0?

Page 1 of 1 (13 items)