.NET Blog

Update to HttpClient and Automatic Decompression

2013-06-19T23:22:00Z

Two weeks ago, we published a beta of HttpClient that provides support for automatic decompression. Since then, we’ve received a fair amount of feedback, especially around the fact that it relies on a native implementation of compression that isn’t available on the same platforms as HttpClient.

Today, we’ve published another beta of HttpClient that will address those concerns.

HttpClient no longer depends on Microsoft.Bcl.Compression

In the last post, we explained why Microsoft.Bcl.Compression depends on native code. To recap, we are using the same implementation of DeflateStream that we shipped in the .NET Framework 4.5. This implementation uses the de facto standard zlib, which is a native library. Unfortunately, Windows Phone 7.1 and Silverlight don't support native code (Windows Phone 8 does), which means that we couldn’t support automatic decompression on those two platforms.

To enable automatic decompression in HttpClient, we had three options:

Add a dependency on Microsoft.Bcl.Compression that doesn’t use native code
Add a dependency on Microsoft.Bcl.Compression that uses native code
Add the (managed) decompression code directly into HttpClient

A main benefit of using the native zlib library is the quality of compression that it provides. It’s not that we couldn’t implement a good compression algorithm in managed code – the problem is that we simply don’t have one and writing one would take considerable resources and a lot more time. So we decided early on not to pursue this option for now, which leaves us with options 2 and 3.

Decompression is still implemented in managed code (both in the .NET Framework 4.5 and in the Microsoft.Bcl.Compression package). Since HttpClient only needs to decompress data, it doesn’t actually depend on anything that is implemented in native code, which makes option 3 an interesting candidate.

However, from an engineering perspective option 2 is cleaner, as there is no code duplication but it comes at the expense of dropping support for platforms that don’t support native code. Previously, we believed it would be better to invest in a good long term solution than to provide a less ideal implementation.

Based on the feedback we received, we decided to revisit that decision and after careful consideration, we decided that the best approach is to remove the dependency on Microsoft.Bcl.Compression and add an internal copy of the managed decompression algorithms to Microsoft.Net.Http. This way, we can support automatic decompression on all platforms.

This doesn’t change Microsoft.Bcl.Compression in any way – it still requires native code.

Using automatic decompression on Silverlight

Before we explain how Silverlight supports automatic decompression let’s recap how automatic decompression works generally in HTTP: The client adds an HTTP request header called Accept-Encoding that specifies which compression algorithm it supports. In the response, the server can either choose to ignore that information and return an uncompressed response or compress the data in one of the algorithms the client requested. The server indicates which algorithm it used via the Content-Encoding response header. For an example with more details, see our earlier blog post on automatic decompression.

Silverlight has two networking stacks: the browser networking stack and the client networking stack. The browser networking stack supports automatic decompression. It always sets the Accept-Encoding header on each request and automatically decompresses the response.

The client networking stack doesn’t have support for setting the HTTP Accept-Encoding header, but throws an exception if you try to do this manually. If you control the server, you can define your own way of activating compression, for example, by using a different HTTP header, by using a query parameter, or by simply compressing all responses. As long as the server sets the Content-Encoding header, HttpClient can automatically decompress the contents.

Now let’s have a look at what that means for the automatic decompression APIs in Silverlight. The SupportsAutomaticDecompression property will return true. The AutomaticDecompression property is set as follows:

Browser stack. The value is ignored. All requests will have set the Accept-Encoding header to indicate they accept both gzip as well as deflate. Responses are automatically decompressed using the appropriate algorithm.
Client stack. The value is ignored when making requests, i.e., the Accept-Encoding header will not be set. However, the Content-Encoding header will be honored when reading responses, i.e., responses are automatically decompressed using the algorithm specified by the Content-Encoding response header.

The route to RTM

Our previously announced, but tentative, schedule looked like the following. We’ve since added about 2 weeks to these dates to accommodate the changes that I’ve described above.

End of June Early July: RC of HttpClient 2.2 with automatic decompression
Mid July Mid July (depending on feedback): RTM of HttpClient 2.2 with automatic decompression

This schedule was based on the assumption that we wouldn’t make major changes to the features of HttpClient. Since we changed the way compression is handled, we decided to ship another beta before an RC release.

This doesn’t change the schedule drastically but we expect to delay both the RC as well as the RTM by about two weeks. We want to give you as well as our partners enough time to absorb the changes and provide feedback.

Summary

The new version of HttpClient no longer depends on Microsoft.Bcl.Compression. Automatic decompression is now supported on all platforms and also allows the application to remain in the AnyCPU architecture.

Microsoft.Bcl.Compression is unaffected by this change; it still isn’t supported on Windows Phone 7.1 or on Silverlight because it requires support for native code.

Please let us know what you think about this update.

NuGet Package Restore Issues

2013-06-13T00:30:00Z

We’ve received several reports that our NuGet packages broke the NuGet package restore feature. In this post, I’ll explain what the issue is, how you can work around it, and finally how we plan on fixing this issue in the long term.

The problem and solution

Microsoft.Bcl.Build and Microsoft.Bcl.Compression require custom target files, which do not work well with NuGet’s package restore feature. The easiest way to fix the package restore issues is by checking in any .targets files that are stored under the packages directory.

What’s package restore?

When you add a NuGet package to your project, NuGet essentially does two things:

It downloads the package and puts it in your solution under a directory called “packages”.
It installs the package to your project. This will add references to additional framework assemblies, references to assemblies provided by the NuGet package, add additional content to your project and last, but not least, import any custom target files. Of course, not all NuGet packages use all those features.

In order to build your project on a build server you have to check in all sources as well as all 3rd party libraries. Many developers cringe when binaries need to be checked in to version control as they typically aren’t stored very efficiently and cause bloat over time. This is especially problematic for distributed version control systems (DVCS) like git or Mercurial where developers have to download the repository with the entire history (typically referred to as “cloning”).

For that reason NuGet has a feature called package restore. You need to enable that feature explicitly by right-clicking your solution and invoking the Enable NuGet Package Restore menu item:

After package restore is enabled you can delete the “packages” directory from your solution. Rebuilding the project will automatically re-create this directory and retrieve all missing packages; whether they are downloaded from the internet or a local file share. This allows excluding the packages directory from version control as the build machine can retrieve the packages and thus doesn’t need a checked-in version of the NuGet packages.

What’s the issue?

Two of our packages provide a custom targets file:

Microsoft.Bcl.Build
Microsoft.Bcl.Compression

Targets files are MSBuild files that provide additional functionality that extend the build process. We use it for several features for which NuGet doesn’t provide declarative features today, for example, binding redirects and choosing the correct binary for the selected architecture. We also use it for additional diagnostics, as explained in this blog post (section “The CPU architecture matters”).

Installing Microsoft.Bcl.Build will change your project by adding an <import> entry for the target file to your project. The target file is located in the packages folder

<Import Project="..\packages\Microsoft.Bcl.Build.1.0.7\tools\Microsoft.Bcl.Build.targets" />

If package restore is enabled, it’s likely the packages folder is excluded so the .targets file isn’t available on the build machine. You may now think “hold on – isn’t package restore supposed to take care of this?” Unfortunately, not in this case. In order to restore packages you need to build the project, since package restore is plugged into the build process. Before building a project MSBuild loads the project, which in turn requires loading all necessary .targets files. Since the .targets files can’t be located at load time the build fails well before package restore has a chance of executing. You will see an error message like this:

YourProject.csproj(44,3): error MSB4019: The imported project "D:\YourProject.csproj\packages\Microsoft.Bcl.Build.1.0.7\tools\Microsoft.Bcl.Build.targets" was not found. Confirm that the path in the <Import> declaration is correct, and that the file exists on disk.

You may be aware that NuGet 2.5 added first class support for installing custom target files. The end result is very similar, except that the import looks like this:

<Import Project="..\packages\Microsoft.Bcl.Build.1.0.7\tools\Microsoft.Bcl.Build.targets"
        Condition="Exists(‘..\packages\Microsoft.Bcl.Build.1.0.7\tools\Microsoft.Bcl.Build.targets’)" />

The import is conditioned on the existence of the file which allows loading the project file even if the .targets file doesn’t exist. Unfortunately this doesn’t solve the problem either.

On the build machine, the build will load the project without our .targets file, run package restore and then build the project. Since MSBuild evaluates all <import> entries before building, the .targets files don’t participate in the build because they weren’t present when the project was loaded. Depending on what a custom .targets file does this can have virtually any impact on the build, for example the build may simply still fail or – worse – succeed with incorrect outputs. For that reason we decided to not use conditional imports because the build at least fails in a deterministic and predictable way.

On the developer machine you have a very similar problem. When a project has missing .targets files it will not successfully open in Visual Studio. You might think that conditional imports would be a good solution there but you’ve fundamentally the same problem as on the build machine: the first build will restore the packages but the build outputs might be bogus without yielding an error. This behavior would remain until you re-open the solution. This issue would be quite hard to diagnose.

Workarounds

In order to make package restore work for packages that have custom target files you have three options:

Stop using package restore and check-in all package files
Explicitly run package restore before building the project
Check-in the .targets files

The first option isn’t a really an option and we certainly wouldn’t recommend it either.

The second option requires changing the way your build is hooked up to your build environment. You need to run nuget.exe to restore packages before you actually run MSBuild on your solution:

        nuget.exe install .\PortableClassLibrary3\packages.config

This ensures that by the time MSBuild loads the project all .targets are already present.

The third option is the easiest solution as it doesn’t require any changes to your environment. You only need to check-in the .targets files – you don’t need to check in any other files from the packages directory. Since target files are simply text files, they aren’t causing the repository to bloat over time. This also solves the issue on the developer’s machine whereby the second option would still require each team member to run package restore before the affected projects can be opened in Visual Studio.

We are working with the NuGet team to make sure we don’t have to use .target files moving forward. The idea is to add some features that allow packages to be declarative as opposed to adding code that runs during the build.

Summary

We understand that this isn’t good long term solution and are working with NuGet to greatly reduce the need for custom .targets files.

Please let us know if you have any questions or additional concerns.

Portable Compression and HttpClient Working Together

2013-06-07T02:08:00Z

Today we’re happy to announce that we released two NuGet packages:

A brand new Microsoft.Bcl.Compression (Beta). A portable compression library that supports stream based compression (gzip, deflate) as well as ZIP archives.
An updated Microsoft.Net.Http 2.2 (Beta). Last week we shipped an RTM version. Today, we released a new beta that provides automatic decompression by taking a dependency on the Microsoft.Bcl.Compression package.

Before we go into the details, let’s first take a look at why compression is particularly interesting for HttpClient.

Compression and HttpClient

We live in a world where we are permanently surrounded by devices, particularly smart phones. Apps that run on these devices are often not super useful in isolation; they require services to provide data and enhance their features.

As a result many developers use the HttpClient class to access web resources such as REST services. Most service providers want to minimize the data that is being transmitted between a client and server. This is particularly helpful when services are accessed by apps on mobile devices that use metered Internet connections. The cost to use the app is directly affected by the amount of data being transmitted, and less data means lower cost to the end-user.

In HTTP, headers provide a way to negotiate capabilities between the client and server. The Accept-Encoding header allows clients to tell the server that they support reading the response in a compressed format (e.g., gzip or Deflate), and is a logical step when using HTTP from a mobile client. The server may choose to ignore the header or reply in another format the client indicates support for.

Not surprisingly, after we announced a portable HttpClient, you asked us to support the AutomaticDecompression property on HttpClientHandler, particularly for Windows Phone apps. Unfortunately, at the time, we didn’t have a portable compression library at our disposal.

In this blog post, I’d like to show you our new portable compression library and explain how we leverage it to support AutomaticDecompression in HttpClient.

Enter Microsoft.Bcl.Compression

Today we released a beta of Microsoft.Bcl.Compression, a NuGet package that provides compression APIs in a portable class library. The compression support covers the following aspects:

Compression streams (Deflate & gzip)
Reading and writing ZIP archives

We also released an update to our HttpClient NuGet package that takes a dependency on Microsoft.Bcl.Compression to support automatic decompression.

The Microsoft.Bcl.Compression NuGet package is supported on the following platforms:

.NET Framework 4.5
Windows Phone 8
.NET for Windows Store apps

This package also includes portable class libraries that target any of those platforms.

The platform list is different from the platforms supported by HttpClient. In particular, HttpClient supports the .NET Framework 4 and Windows Phone 7.5, but the Microsoft.Bcl.Compression package isn’t supported on those platforms.

Using Microsoft.Bcl.Compression directly

Of course, a portable compression library has plenty of uses other than just reducing the footprint of networking requests. That's why Microsoft.Bcl.Compression is completely independent from HttpClient and can be used directly. Let’s have a look at how you would use it.

Consider a Windows Phone app that shows a random poem from a collection of poems. The poems are represented as text files. In order to keep the app small, we compressed all poems using gzip. During application startup, a random poem is selected and read from the application data. In order to decompress the contents of the poem, we use the GZipStream class.

        private static string ReadRandomPoem()
        {
            int randomPoemNumber = 42; // Super random
            string poemName = string.Format("Poem{0}.txt.gz", randomPoemNumber);
            using (Stream stream = OpenApplicationFile(poemName))
            using (Stream decompressed = new GZipStream(stream, CompressionMode.Decompress))
            using (StreamReader reader = new StreamReader(decompressed))
            {
                string text = reader.ReadToEnd();
                return text;
            }
        }

Now let’s say that in the next version of the app we allow the user to select a form of poem, such as sonnet or haiku. To implement this, we decide to store all poems that share the same form in the same ZIP archive. To read a ZIP archive we use the ZipArchive class.

        private static string ReadRandomPoem()
        {
            int randomPoemNumber = 42; // Still super random
            string poemName = string.Format("Poem{0}.txt", randomPoemNumber);
            using (Stream archiveStream = OpenApplicationFile("haikus.zip"))
            using (ZipArchive archive = new ZipArchive(archiveStream, ZipArchiveMode.Read))
            {
                ZipArchiveEntry zipArchiveEntry = archive.GetEntry(poemName);
                using (Stream stream = zipArchiveEntry.Open())
                using (StreamReader reader = new StreamReader(stream))
                {
                    string text = reader.ReadToEnd();
                    return text;
                }
            }
        }

Using automatic decompression with HttpClient

Now let’s have look at how you would use the decompression APIs with HttpClient.

First, automatic decompression is not enabled by default for HttpClient. To use it, you need to set the AutomaticDecompression property on HttpClientHandler to GZip or Deflate. This API follows the optional feature pattern; meaning it exposes an API that indicates whether a particular feature is supported. Automatic decompression support is indicated with the SupportsAutomaticDecompression property.

If an implementation of HttpClient doesn’t support automatic decompression, it returns false from the SupportsAutomaticDecompression property and throws a NotSupportedException from the setter and getter for the AutomaticDecompression property. This was how our previous release of the HttpClient NuGet package behaved.

With this pattern in mind, using automatic decompression looks like this:

        var handler = new HttpClientHandler();
        if (handler.SupportsAutomaticDecompression)
        {
            handler.AutomaticDecompression = DecompressionMethods.GZip |
                                             DecompressionMethods.Deflate;
        }
        var httpClient = new HttpClient(handler);
        var str = await httpClient.GetStringAsync("http://en.wikipedia.org/wiki/Gzip");

The request then provides the additional Accept-Encoding header:

        GET http://en.wikipedia.org/wiki/Gzip HTTP/1.1
        Host: en.wikipedia.org
        Accept-Encoding: gzip, deflate
        Connection: Keep-Alive

Servers that choose to support it will respond and indicate the compression algorithm they used. In this example, the server compressed the body using gzip:

        HTTP/1.1 200 OK
        Server: nginx/1.1.19
        Date: Wed, 13 Mar 2013 14:04:24 GMT
        Content-Type: text/html; charset=UTF-8
        Content-Length: 17765
        Connection: keep-alive
        X-Content-Type-Options: nosniff
        Content-Language: en
        Last-Modified: Tue, 05 Mar 2013 03:38:51 GMT
        Content-Encoding: gzip
        Expires: Thu, 01 Jan 1970 00:00:00 GMT
        Cache-Control: private, s-maxage=0, max-age=0, must-revalidate
        Vary: Accept-Encoding,Cookie
        Age: 179

The response stream that HttpClient returns to you will automatically decompress the result using the appropriate algorithm. On my machine I got the following results:

This yields to a reduction by 77%. Needless to say, the actual results depend on the request and how well it compresses. So your results will naturally vary.

The CPU architecture matters

In the .NET Framework 4.5 we changed our DeflateStream implementation to use the popular zlib library. As a result, DeflateStream provides a better implementation of the deflate algorithm and, in most cases, a better compression than in earlier versions of the .NET Framework.

Microsoft.Bcl.Compression also uses the deflate algorithm, which is implemented in native code. This means that Microsoft.Bcl.Compression requires CPU-specific binaries, and each project that consumes it has to be CPU specific. Given that some of the platforms the Microsoft.Bcl.Compression package supports already provide built-in support for compression, only Windows Phone projects are affected by this requirement.

However, we’ve found a way to avoid making each project CPU-specific:

Class library projects that consume Microsoft.Bcl.Compression can remain Any CPU. This makes them consumable by other Any CPU projects as well as CPU-specific projects.
Windows Phone application projects that use Microsoft.Bcl.Compression (either directly or indirectly via another class library) must add a NuGet reference to Microsoft.Bcl.Compression, and specify either ARM or x86 architecture, for phone or emulator respectively. This ensures we deploy binaries that match the CPU architecture of the application.

We understand that this can be tricky to get right. Therefore, our NuGet package will guide you through the process. Let’s assume that you have a solution with the following two projects:

Windows Phone App (configured as Any CPU)
Portable class library (configured as Any CPU)

Let’s say the portable class library depends on Microsoft.Bcl.Compression, whereas the phone app doesn’t depend on Microsoft.Bcl.Compression but does depend on the portable class library.

Building the solution will yield the following error message:

Phone Application: Project must install NuGet package Microsoft.Bcl.Compression.

After adding the reference to Microsoft.Bcl.Compression and rebuilding your app, a new MSBuild task (which comes with the NuGet package) will deploy CPU-specific binaries as dependencies of your app. If your app is configured as Any CPU, the build will fail again with this error message:

Phone Application: Microsoft.Bcl.Compression does not support the currently selected platform of 'AnyCPU'. The supported platforms are 'x86' and 'ARM'.

The project should build successfully after you change the platform target to X86 or ARM.

We’d love to hear your feedback around this!

Summary

We've released a beta version of the new Microsoft.Bcl.Compression package that provides portable compression support for your apps. When you're using portable compression, don’t forget to set your CPU architecture for Windows Phone projects. And as you saw earlier: even if you forget it, our package will take care of reminding you.

We've also released a new beta for the existing Microsoft.Net.Http package in version 2.2. It adds support for AutomaticDecompression. Since this added substantial functionality to HttpClient, we decided to ship a beta instead of updating the stable release we shipped a week ago.

Here is our tentative release schedule for turning Microsoft.Net.Http 2.2 into a stable release:

End of June: RC of HttpClient 2.2 with automatic decompression
Around July (depending on feedback): RTM of HttpClient 2.2 with automatic decompression

This timeline depends on your feedback. So please let us know what you think of Microsoft.Bcl.Compression, and how the new version of HttpClient works for you!

Get /httpclient/rtm – 200 OK

2013-05-29T18:48:00Z

As promised in our last blog post we’re releasing Microsoft.Net.Http as a stable NuGet package today. Yep, that’s right: You can finally start using the portable HttpClient 2.1 in production!

As we’ve discussed in previous blog posts, HttpClient is a modern networking API which makes it easy to access any resource exposed through HTTP. The HttpClient API has been available in some versions of .NET for a while now. This NuGet package makes a standard set of HttpClient APIs available across a wider array of platforms, including Windows Phone 7.5 and higher, .NET Framework 4.0 and higher, and Windows Store. This enables you to share more .NET code across these platforms.

This package also addresses one of the top User Voice requests for .NET: providing an HttpClient implementation for Portable Class Libraries (and therefore also on Windows Phone 8).

We closed this request when we posted the preview of the new HttpClient back in February. Now, with the release of a stable version, we’re excited that you now have a high-quality implementation to use and build on in your .NET projects.

This package requires Visual Studio 2010 or Visual Studio 2012, and NuGet package manager 2.5 or higher.

Changes from RC

We haven’t changed much since last week's release candidate (RC), but this is by-design. For our NuGet packages, feature work is first released in beta builds only. Once we think we matured a feature enough (based on internal testing as well as customer feedback), we release an RC. If there aren’t any major bugs, we turn (generally the same) bits into an RTM, which NuGet calls a stable release. This process helps ensure that we ship high-quality components.

We did make some very minor changes to our package metadata:

To make the package easier to recognize as an official Microsoft component, we prefixed it with “Microsoft”.
To improve search results, we added tags (e.g., “HTTP” and “REST”).

What’s next for HttpClient?

As I explained in my last blog post, the current release doesn’t include support for automatic decompression. One of our readers commented that each time a web request is issued without gzip support, it hurts baby seals.

We know automatic compression is an important feature for you, and we care deeply about the health of baby seals, so we are working on a portable compression library that provides this feature. Our plan is to ship a beta release of HttpClient 2.2 that includes this library very soon. As a reminder, here is our tentative release schedule:

Mid-June: Beta of HttpClient 2.2 with automatic decompression
End of June: RC of HttpClient 2.2 with automatic decompression
Around July (depending on feedback): RTM of HttpClient 2.2 with automatic decompression

Community support for automatic compression

Until we release a stable version of automatic decompression that you can use in production, there are some alternatives you may want to consider. (Please note this isn’t intended to be a complete list. If you’ve developed a similar feature, let us know by posting a comment below.)

Morten Nielsen, a Microsoft Valued Professional (MVP), has provided an implementation of HttpClientHandler that has support for gzip compression. You can find the source code here:

https://gist.github.com/dotMorten/4981198
Robert McLaws created a derived project and packaged it up as a NuGet package. You can install his NuGet package either via the package manager console:
```
                Install-Package HttpClient.Compression –Pre
```
or by searching for HttpClient.Compression in the NuGet Package Manager dialog. At the time of this writing, HttpClient.Compression wasn’t available as a stable NuGet package, because the stable version of our HttpClient wasn’t released yet. I’m sure it will get updated very soon, though.

Summary

HttpClient 2.1 is now available as a stable release on NuGet. The license allows its use in production and unblocks third parties from building stable NuGet components on top of it.

You can expect a beta release of HttpClient 2.2 with support for automatic decompression very soon. Meanwhile, help protect baby seals by using one of community libraries we linked to above. If there are other community libraries available, please let us know in the comments section!

Portable HttpClient is now available as RC

2013-05-23T00:24:00Z

Three months ago we shipped the first preview of the portable HttpClient. Many of you wondered when we would ship the RTM version. Today, we’re happy to announce the first step towards an RTM: We shipped a release candidate (RC) of HttpClient (Microsoft.Net.Http package on NuGet) that includes all the bug fixes since the preview.

New features

To address some of the platform differences in a portable fashion, we’ve added new capability APIs.

HttpClientHandler.SupportsUseProxy(): The existing HttpClientHandler.SupportsProxy property indicates whether both the UseProxy property and the Proxy property are supported. This created an issue because some platforms (for example, Windows Phone) don’t allow you to configure a proxy explicitly. However, Windows Phone lets you control whether the machine wide proxy should be used. To model this behavior, we added the HttpClientHandler.SupportsUseProxy() extension method. For some platforms that don’t support both, such as Windows Phone, the SupportsProxy property will continue to return false, but the SupportsUseProxy() method will return true.

HttpClientHandler.SupportsAllowAutoRedirect(): The HttpClientHandler.SupportsRedirectConfiguration property had a similar issue: It controls whether both the AllowAutoRedirect and the MaxAutomaticRedirections properties are supported. Windows Phone doesn’t support specifying the maximum number of automatic redirections, but it does support redirection. For that reason, we added the HttpClientHandler.SupportsAllowAutoRedirect() extension method.

Here's how you use these capability APIs:

HttpClientHandler handler = new HttpClientHandler();

// Configure proxy (if supported)
if (handler.SupportsUseProxy())
    handler.UseProxy = true;

// Allow automatic redirection (if supported)
if (handler.SupportsAllowAutoRedirect())
    handler.AllowAutoRedirect = true;

In case you are wondering why we added extension methods instead of regular properties: Some of the platforms that Microsoft.Net.Http supports already provide the HttpClientHandler class, which is used on those platforms. Since we can’t modify the built-in version of the properties directly, we added extension methods that ship in a separate assembly with the NuGet package.

Bug fixes

All bug fixes are listed in the release notes. In this post, I’d like to highlight just one:

Issue: When consuming a portable class library that depends on HttpClient from a Windows Store application, the app can crash with a MissingMethodException.

We’ve fixed this bug by using the technique we explained in our blog post about Microsoft.Bcl.Async: We ensure during the build that the application has a reference to the NuGet package.

Future directions

This RC doesn’t include support for the HttpClientHandler.AutomaticDecompression property. Many of you have asked for this support, and we certainly haven’t ignored it or forgotten about it. In fact, our original plan was to ship another preview with support for automatic decompression, and then ship an RTM version a few weeks later. So why did we change our minds about it?

We had two options:

Block the release of HttpClient until automatic decompression is ready.
Ship an interim release of HttpClient that doesn’t have automatic decompression but is marked as a stable NuGet release and comes with a license that allows usage in production.

We decided to go with the second option, because we believe it delivers functionality sooner to our customers. After all, that's why we ship on NuGet, and we hope you agree.

Let’s recap why our team is investing so much in releasing components via NuGet. The reason is twofold:

First, it allows bridging differences in functionality between platforms we already shipped. A good example is HttpClient, and also includes the support for the async and await keywords. Shipping features out of band also enables us to ship new functionality for multiple platforms via a single portable class library, without having to wait for any of the platforms to add that functionality.

Second, our goal is to strengthen the feedback loop with you, our customers. In the past, we’ve shipped “big” betas, like a beta of the entire .NET Framework. This approach certainly has some advantages, but we’ve seen issues with it as well. The biggest downside is that “big” betas are pretty expensive to ship and are typically very close to RTM, which, in turn, means that we can’t make impactful changes anymore. In fact, we have to turn down a large number of the bug reports we get in “big” betas, either because they affect a relatively small number of customers or because fixing the bugs would place the RTM release at risk. We’re certainly not the first company running into this problem; there is an entire agile software development movement in our industry that focuses on this. I don’t want to go into a philosophical discussion about agile methodologies, but it’s pretty hard to disagree that shipping early and often helps with feedback loop issues.

A good example where we use your feedback quite heavily is in the previews of immutable collections; in contrast to “big” betas, we were able to change the design of the APIs when we believe that’s the correct approach.

Unfortunately, shipping a constant stream of previews isn’t necessarily a helpful solution in all scenarios. On certain platforms, such as the .NET Framework 4.5 and Windows Store, HttpClient is already available as a fully supported, RTM-quality component. For that reason, we can offer a preview of HttpClient only for Windows Phone today.

Here are our tentative plans for future NuGet releases of HttpClient:

Today: RC of HttpClient 2.1.
Wednesday 5/29: RTM of HttpClient 2.1. Yep, it’s only a week away now, but it entirely depends on your feedback. If we find substantial issues, we might change the release date.
Mid-June: Beta of HttpClient 2.2 with automatic decompression
End of June: RC of HttpClient 2.2 with automatic decompression
Around July (depending on feedback): RTM of HttpClient 2.2 with automatic decompression

Summary

We released an RC version of the Microsoft.Net.Http NuGet package. We expect this to be the last preview of HttpClient before we ship an RTM. Please provide feedback if you find any issues.

Of course, we’d also love to hear from you if everything just works. After all, we're happy if you're happy!

.NET Crash Dump and Live Process Inspection

2013-05-02T04:44:00Z

Analyzing crash dumps can be complicated. Although Visual Studio supports viewing managed crash dumps, you often have to resort to more specialized tools like the SOS debugging extensions or WinDbg. In today’s post, Lee Culver, software developer on the .NET Runtime team, will introduce you to a new managed library that allows you to automate inspection tasks and access even more debugging information. --Immo

Today are we excited to announce the beta release of the Microsoft.Diagnostics.Runtime component (called ClrMD for short) through the NuGet Package Manager.

ClrMD is a set of advanced APIs for programmatically inspecting a crash dump of a .NET program much in the same way as the SOS Debugging Extensions (SOS). It allows you to write automated crash analysis for your applications and automate many common debugger tasks.

We understand that this API won’t be for everyone -- hopefully debugging .NET crash dumps is a rare thing for you. However, our .NET Runtime team has had so much success automating complex diagnostics tasks with this API that we wanted to release it publicly.

One last, quick note, before we get started: The ClrMD managed library is a wrapper around CLR internal-only debugging APIs. Although those internal-only APIs are very useful for diagnostics, we do not support them as a public, documented release because they are incredibly difficult to use and tightly coupled with other implementation details of the CLR. ClrMD addresses this problem by providing an easy-to-use managed wrapper around these low-level debugging APIs.

Getting Started

Let's dive right into an example of what can be done with ClrMD. The API was designed to be as discoverable as possible, so IntelliSense will be your primary guide. As an initial example, we will show you how to collect a set of heap statistics (objects, sizes, and counts) similar to what SOS reports when you run the command !dumpheap –stat.

The “root” object of ClrMD to start with is the DataTarget class. A DataTarget represents either a crash dump or a live .NET process. In this example, we will attach to a live process that has the name “HelloWorld.exe” with a timeout of 5 seconds to attempt to attach:

        int pid = Process.GetProcessesByName("HelloWorld")[0].Id;
        using (DataTarget dataTarget = DataTarget.AttachToProcess(pid, 5000))
        {
            string dacLocation = dataTarget.ClrVersions[0].TryGetDacLocation();
            ClrRuntime runtime = dataTarget.CreateRuntime(dacLocation);

            // ...
        }

You may wonder what the TryGetDacLocation method does. The CLR is a managed runtime, which means that it provides additional abstractions, such as garbage collection and JIT compilation, over what the operating system provides. The bookkeeping for those abstractions is done via internal data structures that live within the process. Those data structures are specific to the CPU architecture and the CLR version. In order to decouple debuggers from the internal data structures, the CLR provides a data access component (DAC), implemented in mscordacwks.dll. The DAC has a standardized interface and is used by the debugger to obtain information about the state of those abstractions, for example, the managed heap. It is essential to use the DAC that matches the CLR version and the architecture of the process or crash dump you want to inspect. For a given CLR version, the TryGetDacLocation method tries to find a matching DAC on the same machine. If you need to inspect a process for which you do not have a matching CLR installed, you have another option: you can copy the DAC from a machine that has that version of the CLR installed. In that case, you provide the path to the alternate mscordacwks.dll to the CreateRuntime method manually. You can read more about the DAC on MSDN.

Note that the DAC is a native DLL and must be loaded into the program that uses ClrMD. If the dump or the live process is 32-bit, you must use the 32-bit version of the DAC, which, in turn, means that your inspection program needs to be 32-bit as well. The same is true for 64-bit processes. Make sure that your program’s platform matches what you are debugging.

Analyzing the Heap

Once you have attached to the process, you can use the runtime object to inspect the contents of the GC heap:

        ClrHeap heap = runtime.GetHeap();
        foreach (ulong obj in heap.EnumerateObjects())
        {
            ClrType type = heap.GetObjectType(obj);
            ulong size = type.GetSize(obj);
            Console.WriteLine("{0,12:X} {1,8:n0} {2}", obj, size, type.Name);
        }

This produces output similar to the following:

         23B1D30       36 System.Security.PermissionSet
         23B1D54       20 Microsoft.Win32.SafeHandles.SafePEFileHandle
         23B1D68       32 System.Security.Policy.PEFileEvidenceFactory
         23B1D88       40 System.Security.Policy.Evidence

However, the original goal was to output a set of heap statistics. Using the data above, you can use a LINQ query to group the heap by type and sort by total object size:

        var stats = from o in heap.EnumerateObjects()
                    let t = heap.GetObjectType(o)
                    group o by t into g
                    let size = g.Sum(o => (uint)g.Key.GetSize(o))
                    orderby size
                    select new
                    {
                        Name = g.Key.Name,
                        Size = size,
                        Count = g.Count()
                    };

        foreach (var item in stats)
            Console.WriteLine("{0,12:n0} {1,12:n0} {2}", item.Size, item.Count, item.Name);

This will output data like the following -- a collection of statistics about what objects are taking up the most space on the GC heap for your process:

           564           11 System.Int32[]
           616            2 System.Globalization.CultureData
           680           18 System.String[]
           728           26 System.RuntimeType
           790            7 System.Char[]
         5,788          165 System.String
        17,252            6 System.Object[]

ClrMD Features and Functionality

Of course, there’s a lot more to this API than simply printing out heap statistics. You can also walk every managed thread in a process or crash dump and print out a managed callstack. For example, this code prints the managed stack trace for each thread, similar to what the SOS !clrstack command would report (and similar to the output in the Visual Studio stack trace window):

        foreach (ClrThread thread in runtime.Threads)
        {
            Console.WriteLine("ThreadID: {0:X}", thread.OSThreadId);
            Console.WriteLine("Callstack:");

            foreach (ClrStackFrame frame in thread.StackTrace)
                Console.WriteLine("{0,12:X} {1,12:X} {2}", frame.InstructionPointer, frame.StackPointer, frame.DisplayString);

            Console.WriteLine();
        }

This produces output similar to the following:

        ThreadID: 2D90
        Callstack:
                   0       90F168 HelperMethodFrame
            660E3365       90F1DC System.Threading.Thread.Sleep(Int32)
              C70089       90F1E0 HelloWorld.Program.Main(System.String[])
                   0       90F36C GCFrame

Each ClrThread object also contains a CurrentException property, which may be null, but if not, contains the last thrown exception on this thread. This exception object contains the full stack trace, message, and type of the exception thrown.

ClrMD also provides the following features:

Gets general information about the GC heap:
- Whether the GC is workstation or server
- The number of logical GC heaps in the process
- Data about the bounds of GC segments
Walks the CLR’s handle table (similar to !gchandles in SOS).
Walks the application domains in the process and identifies which modules are loaded into them.
Enumerates threads, callstacks of those threads, the last thrown exception on threads, etc.
Enumerates the object roots of the process (as the GC sees them for our mark-and-sweep algorithm).
Walks the fields of objects.
Gets data about the various heaps that the .NET runtime uses to see where memory is going in the process (see ClrRuntime.EnumerateMemoryRegions in the ClrMD package).

All of this functionality can generally be found on the ClrRuntime or the ClrHeap objects, as seen above. IntelliSense can help you explore the various properties and functions when you install the ClrMD package. In addition, you can also use the attached sample code.

Please use the comments under this post to let us know if you have any feedback!

Social Engineering

2013-04-24T18:36:00Z

When people in our industry use the term “social engineering” they are usually referring to security attacks. Politicians and lobbyists would mean something entirely different too. But that’s not what I will talk about here. Instead, I’ll talk about the kind of social life we’ve around in our team.

Why am I using the term engineering then? Because I strongly believe that building a great team is a conscious effort. Wikipedia defines the term engineering as follows:

Engineering is the application of scientific, economic, social, and practical knowledge, in order to design, build, and maintain structures, machines, devices, systems, materials and processes.

From my point of view, building great software is a lot easier if you have a great team. You obviously can’t build a great team by writing a spec and triaging bugs. But you nonetheless need to invest into it as it doesn’t come for free or even happen automatically.

In this post, I’d like to show some of the big and small things that my team does and what makes working with this team so great – even when being under stress.

Official: Morale Events

About twice a year each team at Microsoft has a morale event. It’s up to the team to decide what to do but it typically involves some kind of team sport and food. Usually, these events are paid for by Microsoft. But before you think Formula 1 and Champaign with caviar: like in almost every company budget for those type of things is limited. However, I don’t think a successful morale event needs a lot of money. All it needs is a point in time and a place to gather.

About a month ago, our team went out to play kickball and have a BBQ in a public park. In case you wonder whether we had fun: you can judge by browsing the pictures below.

(Click picture or this link to see more pictures)

Semi Official: Get Together

It would miss the point if I’d state that it’s only the employer’s responsibility to invest into team building. In fact, I’m not even sure building a team like that is possible. From personal experience I’d say that in all the teams I’ve worked in the ones with the best culture had one thing in common: most of the team building was driven by individual team members.

In our team, we usually hang out during lunch time instead of using this time of the day to be productive in front of the computer. However, I don’t mean to say you absolutely have to have a team lunch. But in my experience talking to your coworkers in an unofficial environment, such as the company’s cafeteria or a sushi place, is super helpful. For one, you have a venue to talk about personal stuff, such as renovating your living room or the tough nights you had because your daughter is getting teeth. And secondly, chances are you talk about work stuff as well. I found that people are more open to make compromises when they talk about the issue in a relaxing environment. Also, I’ve found on more than one occasion we developed a brilliant idea during lunch.

Joel takes this one step further and has lunch delivered to his employees so that they can all sit together as one big family.

In addition to that, we meet every Friday after work in a social area in our building. Each week, someone else is responsible for getting the beer, chips and cake. Yes, there will be cake.

Unofficial: Office Pranks

At Microsoft, we have a long tradition of office pranks. They range from smaller jokes like removing the computer equipment to more elaborate pranks like replacing the door with a fully painted dry-wall. Needless to say they are unofficial and not driven by Microsoft, but completely organized by individual team members.

When I went on a conference trip to Germany a few weeks ago, my team pulled off a super awesome office prank. Since I’m a big Star Trek fan they got inspired by this episode and flooded my office with tribbles. Great pictures and epic making-of mails can be found below.

(Click picture or this link to see more pictures)

Special thanks to our release PM Marc Goodner who organized and PM’ed this epic prank!

Summary

In case you didn’t notice, I wrote this post primarily to say Thanks to my team. We talk a lot about the software we ship but cool stuff deserves to be shared – even it’s not shipping and especially when it involves people.

In case you thought “I wish I could be part of such a team” I’ve good news for you. We are hiring!

As always, we love to hear what you have to say. What do you do to keep up the team spirit? Share your story in the comments!

.NET Memory Allocation Profiling with Visual Studio 2012

2013-04-04T15:54:00Z

This post was written by Stephen Toub, a frequent contributor to the Parallel Programming in .NET blog. He shows us how Visual Studio 2012 and an attention to detail can help you discover unnecessary allocations in your app that can prevent it from achieving higher performance.

Visual Studio 2012 has a wealth of valuable functionality, so much so that I periodically hear developers that already use Visual Studio asking for a feature the IDE already has and that they’ve just never discovered. Other times, I hear developers asking about a specific feature, thinking it’s meant for one purpose, not realizing it’s really intended for another.

Both of these cases apply to Visual Studio’s .NET memory allocation profiler. Many developers that could benefit from it don’t know it exists, and other developers have an incorrect expectation for its purpose. This is unfortunate, as the feature can provide a lot of value for particular scenarios; many developers will benefit from understanding first that it exists, and second the intended scenarios for its use.

Why memory profiling?

When it comes to .NET and memory analysis, there are two primary reasons one would want to use a diagnostics tool:

To discover memory leaks. Leaks on a garbage-collecting runtime like the CLR manifest differently than do leaks in a non-garbage-collected environment, such as in code written in C/C++. A leak in the latter typically occurs due to the developer not manually freeing some memory that was previously allocated. In a garbage collected environment, however, manually freeing memory isn’t required, as that’s the duty of the garbage collector (GC). However, the GC can only release memory that is provably no longer being used, meaning as long as there are no rooted references to the memory. Leaks in .NET code manifest then when some memory that should have been collected is incorrectly still rooted, e.g. a reference to the object occurs in an event handler registered with a static event. A good memory analysis tool might help you to find such leaks, such as by allowing you to take snapshots of the process at two different points and then comparing those snapshots to see which objects stuck around for the second point, and more importantly, why.
To discover unnecessary allocations. In .NET, allocation is often quite cheap. This cost is deceptive, however, as there are more costs later when the GC needs to clean up. The more memory that gets allocated, the more frequently the GC will need to run, and typically the more objects that survive collections, the more work the GC needs to do when it runs to determine which objects are no longer reachable. Thus, the more allocations a program does, the higher the GC costs will be. These GC costs are often negligible to the program’s performance profile, but for certain kinds of apps, especially those on servers that require high-throughput operation, these costs can add up quickly and make a noticeable impact to the performance of the app. As such, a good memory analysis tool might help you to understand all of the allocation being done by the program, in order to help spot allocations you can potentially avoid.

The .NET memory profiler included in Visual Studio 2012 (Professional and higher versions) was designed primarily to address the latter case of helping to discover unnecessary allocations, and it’s quite useful towards that goal, as the rest of this post will explore. The tool is not tuned for the former case of finding and fixing memory leaks, though this is an area the Visual Studio diagnostics team is looking to address in depth for the future (you can see such an experience for JavaScript that was added to Visual Studio as part of VS2012.1). While the tool today does have an advanced option to track when objects are collected, it doesn’t help you to understand why objects weren’t collected or why they were held onto longer than was expected.

There are also other useful tools in this space. The downloadable PerfView tool doesn’t provide as user-friendly an interface as does the .NET memory profiler in Visual Studio 2012, but it is a very powerful tool that supports both tasks of finding memory leaks and discovering unnecessary allocations. It also supports profiling Windows Store apps, which the .NET memory allocation profiler in Visual Studio 2012 does not currently support as of the writing of this post.

Example to be optimized

To better understand the memory profiler’s role and how it can help, let’s walk through an example. We’ll start with the core method that we’ll be looking to optimize (in a real-world case, you’d likely be analyzing your whole application and narrowing in on the particular offending areas, but for the purpose of this example, we’ll keep this constrained):

public static async Task<T> WithCancellation1<T>(this Task<T> task, CancellationToken cancellationToken)
{
    var tcs = new TaskCompletionSource<bool>();
    using (cancellationToken.Register(() => tcs.TrySetResult(true)))
    if (task != await Task.WhenAny(task, tcs.Task))
        throw new OperationCanceledException(cancellationToken);
    return await task;
}

The purpose of this small method is to enable code to await a task in a cancelable manner, meaning that regardless of whether the task has completed, the developer wants to be able to stop waiting for it. Instead of writing code like:

T result = await someTask;

the developer would write:

T result = await someTask.WithCancellation1(token);

and if cancellation is requested on the relevant CancellationToken before the task completes, an OperationCanceledException will be thrown. This is achieved in WithCancellation1 by wrapping the original task in an async method. The method creates a second task that will complete when cancellation is requested (by Registering a call to TrySetResult with the CancellationToken), and then uses Task.WhenAny to wait for either the original task or the cancellation task to complete. As soon as either does, the async method completes, either by throwing a cancellation exception if the cancellation task completed first, or by propagating the outcome of the original task by awaiting it. (For more details, see the blog post “How do I cancel non-cancelable async operations?”)

To understand the allocations involved in this method, we’ll use a small harness method:

using System;
using System.Threading;
using System.Threading.Tasks;
 
class Harness
{
    static void Main() 
     { 
         Console.ReadLine(); // wait until profiler attaches
         TestAsync().Wait(); 
     }
    static async Task TestAsync()
    {
        var token = CancellationToken.None;
        for(int i=0; i<100000; i++)
            await Task.FromResult(42).WithCancellation1(token);
    }
}
 

static class Extensions
{
    public static async Task<T> WithCancellation1<T>(
    this Task<T> task, CancellationToken cancellationToken)
    {
        var tcs = new TaskCompletionSource<bool>();
        using (cancellationToken.Register(() => tcs.TrySetResult(true)))
            if (task != await Task.WhenAny(task, tcs.Task))
                throw new OperationCanceledException(cancellationToken);
        return await task;
    }
}

The TestAsync method will iterate 100,000 times. Each time, it creates a new task, invokes the WithCancellation1 on it, and awaits the result of that WithCancellation1 call. This await will complete synchronously, as the task created by Task.FromResult is returned in an already completed state, and the WithCancellation1 method itself doesn’t introduce any additional asynchrony such that the task it returns will complete synchronously as well.

Running the .NET memory allocation profiler

To start the memory allocation profiler, in Visual Studio go to the Analyze menu and select “Launch Performance Wizard…”. This will open a dialog like the following:

Choose “.NET memory allocation (sampling)”, click Next twice, followed by Finish (if this is the first time you’ve used the profiler since you logged into Windows, you’ll need to accept the elevation prompt so the profiler can start). At that point, the application will be launched and the profiler will start monitoring it for allocations (the harness code above also requires that you press ‘Enter’, in order to ensure the profiler has attached by the time the program starts the real test). When the app completes, or when you manually choose to stop profiling, the profiler will load symbols and will start analyzing the trace. That’s a good time to go and get yourself a cup of coffee, or lunch, as depending on how many allocations occurred, the tool can take a while to do this analysis.

When the analysis completes, we’re presented with a summary of the allocations that occurred, including highlighting the functions that allocated the most memory, the types that resulted in the most memory allocated, and the types with the most instances allocated:

From there, we can drill in further, by looking at the allocations summary (choose “Allocation” from the “Current View” dropdown):

Here, we get to see a row for each type that was allocated, with the columns showing information about how many allocations were tracked, how much space was associated with those allocations, and what percentage of allocations mapped back to that type. We can also expand an entry to see the stack of method calls that resulted in these allocations:

By selecting the “Functions” view, we can get a different pivot on this data, highlighting which functions allocated the most objects and bytes:

Interpreting and acting on the profiling results

With this capability, we can analyze our example’s results. First, we can see that there’s a substantial number of allocations here, which might be surprising. After all, in our example we were using WithCancellation1 with a task that was already completed, which means there should have been very little work to do (with the task already done, there is nothing to cancel), and yet from the above trace we can see that each iteration of our example is resulting in:

Three allocations of Task`1 (we ran the harness 100K times and can see there were ~300K allocations)

Two allocations of Task[]

One allocation each of TaskCompletionSource`1, Action, a compiler-generated type called <>c_DisplayClass2`1, and some type called CompleteOnInvokePromise

That’s nine allocations for a case where we might expect only one (the task allocation we explicitly asked for in the harness by calling Task.FromResult), with our WithCancellation1 method incurring eight allocations.

For helper operations on tasks, it’s actually fairly common to deal with already completed tasks, as often times operations implemented to be asynchronous will actually complete synchronously (e.g. one read operation on a network stream may buffer into memory enough additional data to fulfill a subsequent read operation). As such, optimizing for the already completed case can be really beneficial for performance. Let’s try. Here’s a second attempt at WithCancellation, one that optimizes for several “already completed” cases:

    public static Task<T> WithCancellation2<T>(this Task<T> task, 
                             CancellationToken cancellationToken)
    {
        if (task.IsCompleted || !cancellationToken.CanBeCanceled)
            return task;
        else if (cancellationToken.IsCancellationRequested)
            return new Task<T>(() => default(T), cancellationToken);
        else
            return task.WithCancellation1(cancellationToken);
    }

This implementation checks:

First, whether the task is already completed or whether the supplied CancellationToken can’t be canceled; in both of those cases, there’s no additional work needed, as cancellation can’t be applied, and as such we can just return the original task immediately rather than spending any time or memory creating a new one.

Then whether cancellation has already been requested; if it has, we can allocate a single already-canceled task to be returned, rather than spending the eight allocations we previously paid to invoke our original implementation.

Finally, if none of these fast paths apply, we fall through to calling the original implementation.

Re-profiling our micro-benchmark while using WithCancellation2 instead of WithCancellation1 provides a much improved outlook (you’ll likely notice that the analysis completes much more quickly than it did before, already a sign that we’ve significantly decreased memory allocation). Now we have just have the primary allocation we expected, the one from Task.FromResult called from our TestAsync method in the harness:

So, we’ve now successfully optimized the case where the task is already completed, where cancellation can’t be requested, or where cancellation has already been requested. What about the case where we do actually need to invoke the more complicated logic? Are there any improvements that can be made there?

Let’s change our benchmark to use a task that’s not already completed by the time we invoke WithCancellation2, and also to use a token that can have cancellation requested. This will ensure we make it to the “slow” path:

    static async Task TestAsync()
    {
        var token = new CancellationTokenSource().Token;
        for (int i = 0; i < 100000; i++)
        {
            var tcs = new TaskCompletionSource<int>();
            var t = tcs.Task.WithCancellation2(token);
            tcs.SetResult(42);
            await t;
        }
    }

Profiling again provides more insight:

On this slow path, there are now 14 total allocations per iteration, including the 2 from our TestAsync harness (the TaskCompletionSource<int> we explicitly create, and the Task<int> it creates). At this point, we can use all of the information provided by the profiling results to understand where the remaining 12 allocations are coming from and to then address them as is relevant and possible. For example, let’s look at two allocations specifically: the <>c__DisplayClass2`1 instance and one of the two Action instances. These two allocations will likely be logical to anyone familiar with how the C# compiler handles closures. Why do we have a closure? Because of this line:

using(cancellationToken.Register(() => tcs.TrySetResult(true)))

The call to Register is closing over the ‘tcs’ variable. But this isn’t strictly necessary: the Register method has another overload which instead of taking an Action takes an Action<object> and the object state to be passed to it. If we instead rewrite this line to use that state-based overload, along with a manually cached delegate, we can avoid the closure and those two allocations:

private static readonly Action<object> s_cancellationRegistration =
    s => ((TaskCompletionSource<bool>)s).TrySetResult(true);
…
using(cancellationToken.Register(s_cancellationRegistration, tcs))

Rerunning the profiler confirms those two allocations are no longer occurring:

Start profiling today!

This cycle of profiling, finding and eliminating hotspots, and then going around again is a common approach towards improving the performance of your code, whether using a CPU profiler or a memory profiler. So, if you find yourself in a scenario where you determine that minimizing allocations is important for the performance of your code, give the .NET memory allocation profiler in Visual Studio 2012 a try. Feel free to download the sample project used in this blog post.

For more on profiling, see the blog of the Visual Studio Diagnostics team, and ask them questions in the Visual Studio Diagnostics forum.

Stephen Toub

Follow us on Twitter (@dotnet) and Facebook (dotnet). You can follow other .NET teams, too: @aspnet/asp.net, @efmagicunicorns/efmagicunicorns, @visualstudio/visualstudio.

.NET Framework Documentation Improvements

2013-02-19T20:07:49Z

The CLR documentation team has been busy responding to feedback and making updates and changes to the .NET Framework documentation in the MSDN Library. We would like to tell you about the most recent set of document updates, which were published earlier in February.

Performance content

We have received extensive customer feedback regarding the importance of performance in .NET Framework apps, and we wanted to make it easier for you to find relevant content.

As a result, we’ve reworked the existing performance and reliability topic to include more performance guidance as well as links to performance analysis tools and technology-specific performance content. We’ll continue to improve this documentation area in the coming months.

Do take a look at the revised Performance in .NET Framework Apps topic. Please give us your feedback to help us choose which area to focus on next.

Overload list topics

We are experimenting with a new approach to document methods that have overloads. These pages have been an area of consistent feedback. We have heard that the current approach requires too many clicks to get to the overload page that you want. We are piloting this new approach with the following method overload pages:

Let’s take a look at why we decided to try something different with these pages. In the MSDN Library, our current documentation design for overloaded methods and constructors uses an overload list to provide summary information and links to individual overload pages. Typically, these pages include just the overload list, requiring you to choose one of the overloads to find any technical information for that method beyond just the overload signatures. For example, the following image shows a portion of the String.Compare page.

The overload list provides an abbreviated syntax for each overload (without its return value or parameter names) and a brief, typically one-sentence description. The page was designed to provide just enough information for you to select an overload, and then follow a link to detailed documentation about that overload. However, we have heard many of you ask for more detail on the overload page, such as individual parameters, return value information, and examples.

Until now, our strategy has been to direct you to the detailed information on the individual overload pages. We had done two things to achieve that goal:

Added the following note before the overload list: "This member is overloaded. For complete information about this member, including syntax, usage, and examples, click a name in the overload list."
Reduced the visibility of these pages by ensuring that they do not appear in searches and member lists.

However, even after those changes, we were still hearing requests for more information. To address this issue, we would like to solicit your feedback about the new approach we're trying out. As already stated, we've revised four overload list topics to provide complete documentation for all the individual method overloads. In each case, the overload list topic should serve as a one-stop source for information about that method and all its overloads.

Note, though, that our build system does not yet support this revised format. Because of this, the boilerplate text "This method is overloaded…" continues to appear on these overload list pages. At a later time, we will remove that text.

The expanded documentation is in the Remarks section and includes complete method syntax, parameter and return value descriptions, a list of exceptions, a table to help choose an overload, extended discussion of using the method, and at least one example for each overload. Here are a few screenshots from the updated String.Format overload page to give you a better idea of the information that you can expect on overload pages with this new approach. Feel free to check out the whole page for yourself.

Tell us what you think of this approach. Does it make it easier to get to the information you need? Do you have any suggestions for further improvements? We look forward to hearing from you.

Follow us on Twitter (@dotnet) and Facebook (dotnet). You can follow other .NET teams, too: @aspnet/asp.net, @efmagicunicorns/efmagicunicorns.

C# is the “Language of the Year”

2013-01-04T01:25:09Z

It’s a great way to start a new year: the PYPL index has named C# the language of the year. This index focuses on the leading indicator of learning a language. It’s nice to see that there’s been increased interest in C# over the past year. In the spirit of learning something new, here’s what we suggest you check out if you’re looking to learn a new C# trick.

Learn to program with Async. Async is finally unleashing the power of multicore hardware. Where I/O is involved, we expect almost all new platform APIs to be exposed as async versions. This new language capability ensures that you can quickly build highly scalable and highly responsive apps for the cloud and devices.
Learn to build a Windows Store App. All of our favorite Windows Store apps were written with C# – yours can be too. Many of the new Windows APIs are based on WinRT, which means that all of these new APIs are amazing with C#. To top it off, you can use XAML or DirectX.
Learn to build (or use) cross-platform libraries. .NET is now integral to every platform Microsoft builds. And with so many platforms to target, we’ve made it easy to write portable libraries so you can distribute your binary once for all of them. Many of these libraries are available on NuGet such as Json.NET. Not to mention that your C# code also works on billions of non-Microsoft devices thanks to tools from Xamarin.
Learn about the performance advantages of .NET. One of the best aspects of programming with C# is that your apps get faster by just upgrading to the latest version of Windows or .NET Framework. With Windows Server 2012, Windows 8 and .NET Framework 4.5 your app can see significant improvements in garbage collection and startup time. Bing saw some amazing performance wins from .NET too.
Learn about cool libraries that build on top of C#. What makes C# so cool is that our innovations in the language have inspired new programming patterns and libraries. LINQ is awesome and it’s led to the Reactive Extensions (Rx). Just spend some time understanding how to use Rx and you’ll take your programming game to a new level.

What new C# skill would you recommend everyone learn? Share your thoughts in the comments.