Portable Compression and HttpClient Working Together

Portable Compression and HttpClient Working Together

Rate This
  • Comments 31

Today we’re happy to announce that we released two NuGet packages:

Before we go into the details, let’s first take a look at why compression is particularly interesting for HttpClient.

Compression and HttpClient

We live in a world where we are permanently surrounded by devices, particularly smart phones. Apps that run on these devices are often not super useful in isolation; they require services to provide data and enhance their features.

As a result many developers use the HttpClient class to access web resources such as REST services. Most service providers want to minimize the data that is being transmitted between a client and server. This is particularly helpful when services are accessed by apps on mobile devices that use metered Internet connections. The cost to use the app is directly affected by the amount of data being transmitted, and less data means lower cost to the end-user.

In HTTP, headers provide a way to negotiate capabilities between the client and server. The Accept-Encoding header allows clients to tell the server that they support reading the response in a compressed format (e.g., gzip or Deflate), and is a logical step when using HTTP from a mobile client. The server may choose to ignore the header or reply in another format the client indicates support for.

Not surprisingly, after we announced a portable HttpClient, you asked us to support the AutomaticDecompression property on HttpClientHandler, particularly for Windows Phone apps. Unfortunately, at the time, we didn’t have a portable compression library at our disposal.

In this blog post, I’d like to show you our new portable compression library and explain how we leverage it to support AutomaticDecompression in HttpClient.

Enter Microsoft.Bcl.Compression

Today we released a beta of Microsoft.Bcl.Compression, a NuGet package that provides compression APIs in a portable class library. The compression support covers the following aspects:

We also released an update to our HttpClient NuGet package that takes a dependency on Microsoft.Bcl.Compression to support automatic decompression.

The Microsoft.Bcl.Compression NuGet package is supported on the following platforms:

  • .NET Framework 4.5
  • Windows Phone 8
  • .NET for Windows Store apps

This package also includes portable class libraries that target any of those platforms.

The platform list is different from the platforms supported by HttpClient. In particular, HttpClient supports the .NET Framework 4 and Windows Phone 7.5, but the Microsoft.Bcl.Compression package isn’t supported on those platforms.

Using Microsoft.Bcl.Compression directly

Of course, a portable compression library has plenty of uses other than just reducing the footprint of networking requests. That's why Microsoft.Bcl.Compression is completely independent from HttpClient and can be used directly. Let’s have a look at how you would use it.

Consider a Windows Phone app that shows a random poem from a collection of poems. The poems are represented as text files. In order to keep the app small, we compressed all poems using gzip. During application startup, a random poem is selected and read from the application data. In order to decompress the contents of the poem, we use the GZipStream class.

        private static string ReadRandomPoem()
        {
            int randomPoemNumber = 42; // Super random
            string poemName = string.Format("Poem{0}.txt.gz", randomPoemNumber);
            using (Stream stream = OpenApplicationFile(poemName))
            using (Stream decompressed = new GZipStream(stream, CompressionMode.Decompress))
            using (StreamReader reader = new StreamReader(decompressed))
            {
                string text = reader.ReadToEnd();
                return text;
            }
        }

Now let’s say that in the next version of the app we allow the user to select a form of poem, such as sonnet or haiku. To implement this, we decide to store all poems that share the same form in the same ZIP archive. To read a ZIP archive we use the ZipArchive class.

        private static string ReadRandomPoem()
        {
            int randomPoemNumber = 42; // Still super random
            string poemName = string.Format("Poem{0}.txt", randomPoemNumber);
            using (Stream archiveStream = OpenApplicationFile("haikus.zip"))
            using (ZipArchive archive = new ZipArchive(archiveStream, ZipArchiveMode.Read))
            {
                ZipArchiveEntry zipArchiveEntry = archive.GetEntry(poemName);
                using (Stream stream = zipArchiveEntry.Open())
                using (StreamReader reader = new StreamReader(stream))
                {
                    string text = reader.ReadToEnd();
                    return text;
                }
            }
        }

Using automatic decompression with HttpClient

Now let’s have look at how you would use the decompression APIs with HttpClient.

First, automatic decompression is not enabled by default for HttpClient. To use it, you need to set the AutomaticDecompression property on HttpClientHandler to GZip or Deflate. This API follows the optional feature pattern; meaning it exposes an API that indicates whether a particular feature is supported. Automatic decompression support is indicated with the SupportsAutomaticDecompression property.

If an implementation of HttpClient doesn’t support automatic decompression, it returns false from the SupportsAutomaticDecompression property and throws a NotSupportedException from the setter and getter for the AutomaticDecompression property. This was how our previous release of the HttpClient NuGet package behaved.

With this pattern in mind, using automatic decompression looks like this:

        var handler = new HttpClientHandler();
        if (handler.SupportsAutomaticDecompression)
        {
            handler.AutomaticDecompression = DecompressionMethods.GZip |
                                             DecompressionMethods.Deflate;
        }
        var httpClient = new HttpClient(handler);
        var str = await httpClient.GetStringAsync("http://en.wikipedia.org/wiki/Gzip");   

The request then provides the additional Accept-Encoding header:

        GET http://en.wikipedia.org/wiki/Gzip HTTP/1.1
        Host: en.wikipedia.org
        Accept-Encoding: gzip, deflate
        Connection: Keep-Alive

Servers that choose to support it will respond and indicate the compression algorithm they used. In this example, the server compressed the body using gzip:

        HTTP/1.1 200 OK
        Server: nginx/1.1.19
        Date: Wed, 13 Mar 2013 14:04:24 GMT
        Content-Type: text/html; charset=UTF-8
        Content-Length: 17765
        Connection: keep-alive
        X-Content-Type-Options: nosniff
        Content-Language: en
        Last-Modified: Tue, 05 Mar 2013 03:38:51 GMT
        Content-Encoding: gzip
        Expires: Thu, 01 Jan 1970 00:00:00 GMT
        Cache-Control: private, s-maxage=0, max-age=0, must-revalidate
        Vary: Accept-Encoding,Cookie
        Age: 179
    

The response stream that HttpClient returns to you will automatically decompress the result using the appropriate algorithm. On my machine I got the following results:

This yields to a reduction by 77%. Needless to say, the actual results depend on the request and how well it compresses. So your results will naturally vary.

The CPU architecture matters

In the .NET Framework 4.5 we changed our DeflateStream implementation to use the popular zlib library. As a result, DeflateStream provides a better implementation of the deflate algorithm and, in most cases, a better compression than in earlier versions of the .NET Framework.

Microsoft.Bcl.Compression also uses the deflate algorithm, which is implemented in native code. This means that Microsoft.Bcl.Compression requires CPU-specific binaries, and each project that consumes it has to be CPU specific. Given that some of the platforms the Microsoft.Bcl.Compression package supports already provide built-in support for compression, only Windows Phone projects are affected by this requirement.

However, we’ve found a way to avoid making each project CPU-specific:

  • Class library projects that consume Microsoft.Bcl.Compression can remain Any CPU. This makes them consumable by other Any CPU projects as well as CPU-specific projects.
  • Windows Phone application projects that use Microsoft.Bcl.Compression (either directly or indirectly via another class library) must add a NuGet reference to Microsoft.Bcl.Compression, and specify either ARM or x86 architecture, for phone or emulator respectively. This ensures we deploy binaries that match the CPU architecture of the application.

We understand that this can be tricky to get right. Therefore, our NuGet package will guide you through the process. Let’s assume that you have a solution with the following two projects:

  • Windows Phone App (configured as Any CPU)
  • Portable class library (configured as Any CPU)

Let’s say the portable class library depends on Microsoft.Bcl.Compression, whereas the phone app doesn’t depend on Microsoft.Bcl.Compression but does depend on the portable class library.

Building the solution will yield the following error message:

Phone Application: Project must install NuGet package Microsoft.Bcl.Compression.

After adding the reference to Microsoft.Bcl.Compression and rebuilding your app, a new MSBuild task (which comes with the NuGet package) will deploy CPU-specific binaries as dependencies of your app. If your app is configured as Any CPU, the build will fail again with this error message:

Phone Application: Microsoft.Bcl.Compression does not support the currently selected platform of 'AnyCPU'. The supported platforms are 'x86' and 'ARM'.

The project should build successfully after you change the platform target to X86 or ARM.

We’d love to hear your feedback around this!

Summary

We've released a beta version of the new Microsoft.Bcl.Compression package that provides portable compression support for your apps. When you're using portable compression, don’t forget to set your CPU architecture for Windows Phone projects. And as you saw earlier: even if you forget it, our package will take care of reminding you.

We've also released a new beta for the existing Microsoft.Net.Http package in version 2.2. It adds support for AutomaticDecompression. Since this added substantial functionality to HttpClient, we decided to ship a beta instead of updating the stable release we shipped a week ago.

Here is our tentative release schedule for turning Microsoft.Net.Http 2.2 into a stable release:

  1. End of June: RC of HttpClient 2.2 with automatic decompression
  2. Around July (depending on feedback): RTM of HttpClient 2.2 with automatic decompression

This timeline depends on your feedback. So please let us know what you think of Microsoft.Bcl.Compression, and how the new version of HttpClient works for you!

Leave a Comment
  • Please add 1 and 1 and type the answer here:
  • Post
  • If we build an app using this, can we submit it to the store, or do we have to wait until you RTM?

  • @Nate: We do have an RTM license for HttpClient without compression (2.1). This version has a license that allows usage in production. The version 2.2 we announced in this post is a beta and doesn't come with a go live license. Right now, Microsoft.Bcl.Compression is only available as a beta and thus can't be used in production at all. Please let me know if you have more questions.

  • So it looks like, if you want to build a PCL that supports the widest range possible, you'll want to use HttpClient.Compression instead, as it has a PCL implementation of Zlib and works on .NE 4 and Windows Phone 7.5. Incidentally, why did you decide to go that route and use native when a PCL implementation of Zlib already exists?

  • Does it have range requests figured out?

    forum.nginx.org/read.php

  • Any Chance to have also a silverlight and .NEt 4.0 Support? We Need Support older OS.

  • The lack of support for Windows Phone 7.5 is a deal breaker for me. It's *way* too early to ditch this platform, as there's still a significant user-base on it.

  • @Robert McLaws: The primary reason we went down the native route is because that's the code we already have. The goal of our NuGet package isn't necessarily to get the broadest possible reach but to evolve our binaries into PCL bits while preserving the existing functionality.

    @Ladislav and KooKiz: We don't plans on making Microsoft.Bcl.Compression available to more platforms but to be clear: HttpClient still supports all those platforms. AutomaticDecompression will not work on Silverlight nor Windows Phone 7.5 but it does work on .NET 4 because we use the in-box version of compression.

  • Why did you write the deflate algorithm in native code? Did you guys at least try unsafe C# code before resorting to native? Were the speed savings really that significant that it was necessary?

  • Please ignore my last post. You answered the question while I was reading the page and writing it! Thus the need for some AJAX on these blog sites!

    I hope you guys decide to reimplement it in C# at some point. The extra CPU caveats are definitely something people could do without.

  • @Immo Landwerth

    Well yes, HttpClient works on WP7, and that's a really good thing. Still, I'm facing the problem of updating my existing apps: some of them would really benefit from a rewriting of core components using tasks + httpclient, as it would clean up the code quite a bit and make it that much easier to add new features. But the apps depend heavily on the network, and are already using compression. Therefore, if I take that route, WP7 users will notice a significant drop in performance.

    That said, it's not a dead-end for me. I can still use third-party libraries that add compression support to httpclient. I was just hoping to see an official version.

  • Why do you abandon Silverlight.

    Petty

    :(

  • @KooKiz and Yiannis: Thanks a lot for your feedback. The reason we don't support Silverlight nor the WP7 platform for Microsoft.Bcl.Compression has to do with our dependency on native code. Neither Silverlight nor Windows Phone support regular P/Invoke. For Windows Phone 8 we were able to wrap the native code as a WinRT component. The other platforms (.NET 4.5 and Windows Store) basically came "for free" as we are simply using the in-box version of compression. Of course, we could probably re-write the code in unsafe C# but this would be a major work item. At this point we aren't convinced that this is justified.

    @KooKiz: It seems you are primarily interested in automatic decompression on WP7 and not full support for Microsoft.Bcl.Compression. Is that correct?

  • @Immo Landwerth

    Exactly. I'm currently using a WebClient with added GZip support (SharpGIS.WebClient), I can't possibly go back to uncompressed http. And I think it's the same for many WP7 developers, network compression offers huge performance gains on mobile platforms.

  • Don't use the new Microsoft.Net.Http package from NuGet if you use NuGet Package Restore on your TFS build server because your builds will fail. This package depends on the Microsoft.Bcl library which causes a project to fail to load. nuget.codeplex.com/.../3135

  • > automatic decompression is not enabled by default for HttpClient

    Why not? The protocol was defined in HTTP 1.1, published January 1997.  Been supported by browsers since Internet Explorer 2 and Netscape 2?

    Also, are you going to release the assemblies for people who can't use Nuget? (They don't have an internet connection, or behind a difficult proxy)

Page 1 of 3 (31 items) 123