Portable Compression and HttpClient Working Together

Portable Compression and HttpClient Working Together

Rate This
  • Comments 31

Today we’re happy to announce that we released two NuGet packages:

Before we go into the details, let’s first take a look at why compression is particularly interesting for HttpClient.

Compression and HttpClient

We live in a world where we are permanently surrounded by devices, particularly smart phones. Apps that run on these devices are often not super useful in isolation; they require services to provide data and enhance their features.

As a result many developers use the HttpClient class to access web resources such as REST services. Most service providers want to minimize the data that is being transmitted between a client and server. This is particularly helpful when services are accessed by apps on mobile devices that use metered Internet connections. The cost to use the app is directly affected by the amount of data being transmitted, and less data means lower cost to the end-user.

In HTTP, headers provide a way to negotiate capabilities between the client and server. The Accept-Encoding header allows clients to tell the server that they support reading the response in a compressed format (e.g., gzip or Deflate), and is a logical step when using HTTP from a mobile client. The server may choose to ignore the header or reply in another format the client indicates support for.

Not surprisingly, after we announced a portable HttpClient, you asked us to support the AutomaticDecompression property on HttpClientHandler, particularly for Windows Phone apps. Unfortunately, at the time, we didn’t have a portable compression library at our disposal.

In this blog post, I’d like to show you our new portable compression library and explain how we leverage it to support AutomaticDecompression in HttpClient.

Enter Microsoft.Bcl.Compression

Today we released a beta of Microsoft.Bcl.Compression, a NuGet package that provides compression APIs in a portable class library. The compression support covers the following aspects:

We also released an update to our HttpClient NuGet package that takes a dependency on Microsoft.Bcl.Compression to support automatic decompression.

The Microsoft.Bcl.Compression NuGet package is supported on the following platforms:

  • .NET Framework 4.5
  • Windows Phone 8
  • .NET for Windows Store apps

This package also includes portable class libraries that target any of those platforms.

The platform list is different from the platforms supported by HttpClient. In particular, HttpClient supports the .NET Framework 4 and Windows Phone 7.5, but the Microsoft.Bcl.Compression package isn’t supported on those platforms.

Using Microsoft.Bcl.Compression directly

Of course, a portable compression library has plenty of uses other than just reducing the footprint of networking requests. That's why Microsoft.Bcl.Compression is completely independent from HttpClient and can be used directly. Let’s have a look at how you would use it.

Consider a Windows Phone app that shows a random poem from a collection of poems. The poems are represented as text files. In order to keep the app small, we compressed all poems using gzip. During application startup, a random poem is selected and read from the application data. In order to decompress the contents of the poem, we use the GZipStream class.

        private static string ReadRandomPoem()
        {
            int randomPoemNumber = 42; // Super random
            string poemName = string.Format("Poem{0}.txt.gz", randomPoemNumber);
            using (Stream stream = OpenApplicationFile(poemName))
            using (Stream decompressed = new GZipStream(stream, CompressionMode.Decompress))
            using (StreamReader reader = new StreamReader(decompressed))
            {
                string text = reader.ReadToEnd();
                return text;
            }
        }

Now let’s say that in the next version of the app we allow the user to select a form of poem, such as sonnet or haiku. To implement this, we decide to store all poems that share the same form in the same ZIP archive. To read a ZIP archive we use the ZipArchive class.

        private static string ReadRandomPoem()
        {
            int randomPoemNumber = 42; // Still super random
            string poemName = string.Format("Poem{0}.txt", randomPoemNumber);
            using (Stream archiveStream = OpenApplicationFile("haikus.zip"))
            using (ZipArchive archive = new ZipArchive(archiveStream, ZipArchiveMode.Read))
            {
                ZipArchiveEntry zipArchiveEntry = archive.GetEntry(poemName);
                using (Stream stream = zipArchiveEntry.Open())
                using (StreamReader reader = new StreamReader(stream))
                {
                    string text = reader.ReadToEnd();
                    return text;
                }
            }
        }

Using automatic decompression with HttpClient

Now let’s have look at how you would use the decompression APIs with HttpClient.

First, automatic decompression is not enabled by default for HttpClient. To use it, you need to set the AutomaticDecompression property on HttpClientHandler to GZip or Deflate. This API follows the optional feature pattern; meaning it exposes an API that indicates whether a particular feature is supported. Automatic decompression support is indicated with the SupportsAutomaticDecompression property.

If an implementation of HttpClient doesn’t support automatic decompression, it returns false from the SupportsAutomaticDecompression property and throws a NotSupportedException from the setter and getter for the AutomaticDecompression property. This was how our previous release of the HttpClient NuGet package behaved.

With this pattern in mind, using automatic decompression looks like this:

        var handler = new HttpClientHandler();
        if (handler.SupportsAutomaticDecompression)
        {
            handler.AutomaticDecompression = DecompressionMethods.GZip |
                                             DecompressionMethods.Deflate;
        }
        var httpClient = new HttpClient(handler);
        var str = await httpClient.GetStringAsync("http://en.wikipedia.org/wiki/Gzip");   

The request then provides the additional Accept-Encoding header:

        GET http://en.wikipedia.org/wiki/Gzip HTTP/1.1
        Host: en.wikipedia.org
        Accept-Encoding: gzip, deflate
        Connection: Keep-Alive

Servers that choose to support it will respond and indicate the compression algorithm they used. In this example, the server compressed the body using gzip:

        HTTP/1.1 200 OK
        Server: nginx/1.1.19
        Date: Wed, 13 Mar 2013 14:04:24 GMT
        Content-Type: text/html; charset=UTF-8
        Content-Length: 17765
        Connection: keep-alive
        X-Content-Type-Options: nosniff
        Content-Language: en
        Last-Modified: Tue, 05 Mar 2013 03:38:51 GMT
        Content-Encoding: gzip
        Expires: Thu, 01 Jan 1970 00:00:00 GMT
        Cache-Control: private, s-maxage=0, max-age=0, must-revalidate
        Vary: Accept-Encoding,Cookie
        Age: 179
    

The response stream that HttpClient returns to you will automatically decompress the result using the appropriate algorithm. On my machine I got the following results:

This yields to a reduction by 77%. Needless to say, the actual results depend on the request and how well it compresses. So your results will naturally vary.

The CPU architecture matters

In the .NET Framework 4.5 we changed our DeflateStream implementation to use the popular zlib library. As a result, DeflateStream provides a better implementation of the deflate algorithm and, in most cases, a better compression than in earlier versions of the .NET Framework.

Microsoft.Bcl.Compression also uses the deflate algorithm, which is implemented in native code. This means that Microsoft.Bcl.Compression requires CPU-specific binaries, and each project that consumes it has to be CPU specific. Given that some of the platforms the Microsoft.Bcl.Compression package supports already provide built-in support for compression, only Windows Phone projects are affected by this requirement.

However, we’ve found a way to avoid making each project CPU-specific:

  • Class library projects that consume Microsoft.Bcl.Compression can remain Any CPU. This makes them consumable by other Any CPU projects as well as CPU-specific projects.
  • Windows Phone application projects that use Microsoft.Bcl.Compression (either directly or indirectly via another class library) must add a NuGet reference to Microsoft.Bcl.Compression, and specify either ARM or x86 architecture, for phone or emulator respectively. This ensures we deploy binaries that match the CPU architecture of the application.

We understand that this can be tricky to get right. Therefore, our NuGet package will guide you through the process. Let’s assume that you have a solution with the following two projects:

  • Windows Phone App (configured as Any CPU)
  • Portable class library (configured as Any CPU)

Let’s say the portable class library depends on Microsoft.Bcl.Compression, whereas the phone app doesn’t depend on Microsoft.Bcl.Compression but does depend on the portable class library.

Building the solution will yield the following error message:

Phone Application: Project must install NuGet package Microsoft.Bcl.Compression.

After adding the reference to Microsoft.Bcl.Compression and rebuilding your app, a new MSBuild task (which comes with the NuGet package) will deploy CPU-specific binaries as dependencies of your app. If your app is configured as Any CPU, the build will fail again with this error message:

Phone Application: Microsoft.Bcl.Compression does not support the currently selected platform of 'AnyCPU'. The supported platforms are 'x86' and 'ARM'.

The project should build successfully after you change the platform target to X86 or ARM.

We’d love to hear your feedback around this!

Summary

We've released a beta version of the new Microsoft.Bcl.Compression package that provides portable compression support for your apps. When you're using portable compression, don’t forget to set your CPU architecture for Windows Phone projects. And as you saw earlier: even if you forget it, our package will take care of reminding you.

We've also released a new beta for the existing Microsoft.Net.Http package in version 2.2. It adds support for AutomaticDecompression. Since this added substantial functionality to HttpClient, we decided to ship a beta instead of updating the stable release we shipped a week ago.

Here is our tentative release schedule for turning Microsoft.Net.Http 2.2 into a stable release:

  1. End of June: RC of HttpClient 2.2 with automatic decompression
  2. Around July (depending on feedback): RTM of HttpClient 2.2 with automatic decompression

This timeline depends on your feedback. So please let us know what you think of Microsoft.Bcl.Compression, and how the new version of HttpClient works for you!

Leave a Comment
  • Please add 4 and 4 and type the answer here:
  • Post
  • @Jaans: We are aware of this issue and are working with NuGet to resolve it. Meanwhile, the easiest solution is to check-in the Microsoft.Bcl.Build.targets file (you don't need to check in any binaries though).

    @Matt H: Regarding automatic decompression being opt-in: Good point, I guess it could have been enabled by default. The reason it was decided to make it opt-in was based on the fact that the previous HttpWebRequest already shipped in the same way.

    Regarding shipping standalone binaries: We don't have any plans on shipping standalone binaries as we rely on certain features that are provided by NuGet (for example, installing custom target files in order to participate in the build). However, we do understand that downloading bits from the internet isn’t a viable solution in all cases. Fortunately NuGet has support for that. All you need is someone to download the NuGet packages (*.nupkg) you want be usable in your organization  and copy them to a file share, e.g. \\shared\nuget. You can point NuGet to any package feed, including simple file shares. NuGet also supports storing that information local to your project. Please have a look at the NuGet documentation for more details (docs.nuget.org/.../nuget-2.1).

  • @Jesper:  >> Does it have range requests figured out?

    Sorry -- just realized I never replied to you. Can you explain a bit more what exactly you are looking for?

  • Another request for making this work on Silverlight plz.

  • Good to see more functionality going in the portable .NET classes and less in the legacy classes.   An effort to lower the reliance on win32 legacy APIs would be a plus as well.

  • @Ted: This will happen over time. In the past, the primary reason we offered .NET APIs that were essentially just wrappers around Win32 was due the fact that the programming model of Win32 is so fundamentally different and often isn't as approachable as managed APIs. The new Windows Runtime APIs (WinRT) have basically the exact same programming model as .NET and are virtually indistinguishable in terms of convenience and discoverability.

    We don't see a problem with relying on Win32 APIs to implement managed APIs -- for example, the runtime has to call Win32 APIs to allocate memory or create threads. However, moving forward we want to avoid providing simple wrappers around Windows technologies. In the past that often meant we got in the way when Windows shipped updates to those technologies (for example, WPF didn't expose the new shell features for a while). Given that most of the new APIs are made available as WinRT there is little reason for us to build wrappers.

  • We’ve received several reports that our NuGet packages broke the NuGet package restore feature

  • This definitely needs a global that changes the default value for AutomaticDecompression. Having to rewrite every call that uses HttpClient is rather frustrating, and for third-party libraries may not even be possible.

  • Ugh, it doens't support WP7.  Maybe now is a good time to dump WP7 support in my app since MS is.

  • Sorry to bother you here but I didn't find an official support channel for HttpClient.

    HttpClient instances encapsulate HttpMessageHandler instances. So why AutomaticDecompression doesn't act on the DefaultRequestHeaders.AcceptEncoding instead of every message RequestHeaders.AcceptEncoding?

  • @SebM: AutomaticDecompression is a member of HttpClientHandler, not HttpMessageHandler.  HttpClient knows nothing about the properties exposed by sub-classes of HttpMessageHandler and doesn't expose an extension point for handlers to modify the DefaultHeaderCollection .  I could imagine an API change to HttpMessageHandler that would allow HttpClient to apply headers to the header collection during construction and somehow keep those in sync should they change after construction, but that would need to go in a future version of .Net.  If you'd like such a feature I'd recommend filing a request on visualstudio.uservoice.com.

    Could you let me know if you have a scenario that is broken by not having these header values exposed in DefaultRequestHeaders?  HttpClientHandler should join between DefaultRequestHeaders to allow for applying compression quality settings etc for Accept-Encoding.

  • Hi,

    How are you?

    Does this support compressing requests as gzip?

    I am into writing a client library where I construct JSON requests to be submitted to an endpoint. And the endpoint expects the payload to be gzip compressed. I guess the solution you discuss above is related to decompressing the response payload which is gzip compressed. Correct me if I am wrong.

    I was googling for "compress http post windows 8 phone" and came across this page.

    Wondering if there is a way to do gzip compression for windows 8 phone development. I think there are ways to do it for regular windows 8.

    PS: I am new to .Net and started off writing the client library this week. I need some inputs on compressing payload using gzip for the windows 8 phone client library that I am developing right now.

  • @EricStj - MSFT: Could you redirect me to a public a place where I might argument around my scenarios?

  • @aravind: HttpClient doesn't have support for automatic compression. It only supports automatic decompression. If you want to compress data you can totally do this, but this would be manually by using the GzipStream class to compress the data. If you are targeting Windows Phone you will need the Microsoft.Bcl.Compression NuGet package.

    @SebM: The best place for a discussion would be via mail. You can reach us via www.nuget.org/.../ContactOwners.

  • I do not understand how to use Bcl.Compression for a Windows Phone 8 app.

    On Windows Phone 8 projects the platform target cannot be changed. The Target CPU option is disabled.

  • Referncing Msft.Bcl.Compression is no longger needed with the current PCL HttpClient (2.2.22) correct?

Page 2 of 3 (31 items) 123