.NET Blog

Portable HttpClient is now available as RC

Immo Landwerth [MSFT] — Thu, 23 May 2013 00:24:00 GMT

Three months ago we shipped the first preview of the portable HttpClient. Many of you wondered when we would ship the RTM version. Today, we’re happy to announce the first step towards an RTM: We shipped a release candidate (RC) of HttpClient (Microsoft.Net.Http package on NuGet) that includes all the bug fixes since the preview.

New features

To address some of the platform differences in a portable fashion, we’ve added new capability APIs.

HttpClientHandler.SupportsUseProxy(): The existing HttpClientHandler.SupportsProxy property indicates whether both the UseProxy property and the Proxy property are supported. This created an issue because some platforms (for example, Windows Phone) don’t allow you to configure a proxy explicitly. However, Windows Phone lets you control whether the machine wide proxy should be used. To model this behavior, we added the HttpClientHandler.SupportsUseProxy() extension method. For some platforms that don’t support both, such as Windows Phone, the SupportsProxy property will continue to return false, but the SupportsUseProxy() method will return true.

HttpClientHandler.SupportsAllowAutoRedirect(): The HttpClientHandler.SupportsRedirectConfiguration property had a similar issue: It controls whether both the AllowAutoRedirect and the MaxAutomaticRedirections properties are supported. Windows Phone doesn’t support specifying the maximum number of automatic redirections, but it does support redirection. For that reason, we added the HttpClientHandler.SupportsAllowAutoRedirect() extension method.

Here's how you use these capability APIs:

HttpClientHandler handler = new HttpClientHandler();

// Configure proxy (if supported)
if (handler.SupportsUseProxy())
    handler.UseProxy = true;

// Allow automatic redirection (if supported)
if (handler.SupportsAllowAutoRedirect())
    handler.AllowAutoRedirect = true;

In case you are wondering why we added extension methods instead of regular properties: Some of the platforms that Microsoft.Net.Http supports already provide the HttpClientHandler class, which is used on those platforms. Since we can’t modify the built-in version of the properties directly, we added extension methods that ship in a separate assembly with the NuGet package.

Bug fixes

All bug fixes are listed in the release notes. In this post, I’d like to highlight just one:

Issue: When consuming a portable class library that depends on HttpClient from a Windows Store application, the app can crash with a MissingMethodException.

We’ve fixed this bug by using the technique we explained in our blog post about Microsoft.Bcl.Async: We ensure during the build that the application has a reference to the NuGet package.

Future directions

This RC doesn’t include support for the HttpClientHandler.AutomaticDecompression property. Many of you have asked for this support, and we certainly haven’t ignored it or forgotten about it. In fact, our original plan was to ship another preview with support for automatic decompression, and then ship an RTM version a few weeks later. So why did we change our minds about it?

We had two options:

Block the release of HttpClient until automatic decompression is ready.
Ship an interim release of HttpClient that doesn’t have automatic decompression but is marked as a stable NuGet release and comes with a license that allows usage in production.

We decided to go with the second option, because we believe it delivers functionality sooner to our customers. After all, that's why we ship on NuGet, and we hope you agree.

Let’s recap why our team is investing so much in releasing components via NuGet. The reason is twofold:

First, it allows bridging differences in functionality between platforms we already shipped. A good example is HttpClient, and also includes the support for the async and await keywords. Shipping features out of band also enables us to ship new functionality for multiple platforms via a single portable class library, without having to wait for any of the platforms to add that functionality.

Second, our goal is to strengthen the feedback loop with you, our customers. In the past, we’ve shipped “big” betas, like a beta of the entire .NET Framework. This approach certainly has some advantages, but we’ve seen issues with it as well. The biggest downside is that “big” betas are pretty expensive to ship and are typically very close to RTM, which, in turn, means that we can’t make impactful changes anymore. In fact, we have to turn down a large number of the bug reports we get in “big” betas, either because they affect a relatively small number of customers or because fixing the bugs would place the RTM release at risk. We’re certainly not the first company running into this problem; there is an entire agile software development movement in our industry that focuses on this. I don’t want to go into a philosophical discussion about agile methodologies, but it’s pretty hard to disagree that shipping early and often helps with feedback loop issues.

A good example where we use your feedback quite heavily is in the previews of immutable collections; in contrast to “big” betas, we were able to change the design of the APIs when we believe that’s the correct approach.

Unfortunately, shipping a constant stream of previews isn’t necessarily a helpful solution in all scenarios. On certain platforms, such as the .NET Framework 4.5 and Windows Store, HttpClient is already available as a fully supported, RTM-quality component. For that reason, we can offer a preview of HttpClient only for Windows Phone today.

Here are our tentative plans for future NuGet releases of HttpClient:

Today: RC of HttpClient 2.1.
Wednesday 5/29: RTM of HttpClient 2.1. Yep, it’s only a week away now, but it entirely depends on your feedback. If we find substantial issues, we might change the release date.
Mid-June: Beta of HttpClient 2.2 with automatic decompression
End of June: RC of HttpClient 2.2 with automatic decompression
Around July (depending on feedback): RTM of HttpClient 2.2 with automatic decompression

Summary

We released an RC version of the Microsoft.Net.Http NuGet package. We expect this to be the last preview of HttpClient before we ship an RTM. Please provide feedback if you find any issues.

Of course, we’d also love to hear from you if everything just works. After all, we're happy if you're happy!

.NET Crash Dump and Live Process Inspection

Immo Landwerth [MSFT] — Thu, 02 May 2013 04:44:00 GMT

Analyzing crash dumps can be complicated. Although Visual Studio supports viewing managed crash dumps, you often have to resort to more specialized tools like the SOS debugging extensions or WinDbg. In today’s post, Lee Culver, software developer on the .NET Runtime team, will introduce you to a new managed library that allows you to automate inspection tasks and access even more debugging information. --Immo

Today are we excited to announce the beta release of the Microsoft.Diagnostics.Runtime component (called ClrMD for short) through the NuGet Package Manager.

ClrMD is a set of advanced APIs for programmatically inspecting a crash dump of a .NET program much in the same way as the SOS Debugging Extensions (SOS). It allows you to write automated crash analysis for your applications and automate many common debugger tasks.

We understand that this API won’t be for everyone -- hopefully debugging .NET crash dumps is a rare thing for you. However, our .NET Runtime team has had so much success automating complex diagnostics tasks with this API that we wanted to release it publicly.

One last, quick note, before we get started: The ClrMD managed library is a wrapper around CLR internal-only debugging APIs. Although those internal-only APIs are very useful for diagnostics, we do not support them as a public, documented release because they are incredibly difficult to use and tightly coupled with other implementation details of the CLR. ClrMD addresses this problem by providing an easy-to-use managed wrapper around these low-level debugging APIs.

Getting Started

Let's dive right into an example of what can be done with ClrMD. The API was designed to be as discoverable as possible, so IntelliSense will be your primary guide. As an initial example, we will show you how to collect a set of heap statistics (objects, sizes, and counts) similar to what SOS reports when you run the command !dumpheap –stat.

The “root” object of ClrMD to start with is the DataTarget class. A DataTarget represents either a crash dump or a live .NET process. In this example, we will attach to a live process that has the name “HelloWorld.exe” with a timeout of 5 seconds to attempt to attach:

        int pid = Process.GetProcessesByName("HelloWorld")[0].Id;
        using (DataTarget dataTarget = DataTarget.AttachToProcess(pid, 5000))
        {
            string dacLocation = dataTarget.ClrVersions[0].TryGetDacLocation();
            ClrRuntime runtime = dataTarget.CreateRuntime(dacLocation);

            // ...
        }

You may wonder what the TryGetDacLocation method does. The CLR is a managed runtime, which means that it provides additional abstractions, such as garbage collection and JIT compilation, over what the operating system provides. The bookkeeping for those abstractions is done via internal data structures that live within the process. Those data structures are specific to the CPU architecture and the CLR version. In order to decouple debuggers from the internal data structures, the CLR provides a data access component (DAC), implemented in mscordacwks.dll. The DAC has a standardized interface and is used by the debugger to obtain information about the state of those abstractions, for example, the managed heap. It is essential to use the DAC that matches the CLR version and the architecture of the process or crash dump you want to inspect. For a given CLR version, the TryGetDacLocation method tries to find a matching DAC on the same machine. If you need to inspect a process for which you do not have a matching CLR installed, you have another option: you can copy the DAC from a machine that has that version of the CLR installed. In that case, you provide the path to the alternate mscordacwks.dll to the CreateRuntime method manually. You can read more about the DAC on MSDN.

Note that the DAC is a native DLL and must be loaded into the program that uses ClrMD. If the dump or the live process is 32-bit, you must use the 32-bit version of the DAC, which, in turn, means that your inspection program needs to be 32-bit as well. The same is true for 64-bit processes. Make sure that your program’s platform matches what you are debugging.

Analyzing the Heap

Once you have attached to the process, you can use the runtime object to inspect the contents of the GC heap:

        ClrHeap heap = runtime.GetHeap();
        foreach (ulong obj in heap.EnumerateObjects())
        {
            ClrType type = heap.GetObjectType(obj);
            ulong size = type.GetSize(obj);
            Console.WriteLine("{0,12:X} {1,8:n0} {2}", obj, size, type.Name);
        }

This produces output similar to the following:

         23B1D30       36 System.Security.PermissionSet
         23B1D54       20 Microsoft.Win32.SafeHandles.SafePEFileHandle
         23B1D68       32 System.Security.Policy.PEFileEvidenceFactory
         23B1D88       40 System.Security.Policy.Evidence

However, the original goal was to output a set of heap statistics. Using the data above, you can use a LINQ query to group the heap by type and sort by total object size:

        var stats = from o in heap.EnumerateObjects()
                    let t = heap.GetObjectType(o)
                    group o by t into g
                    let size = g.Sum(o => (uint)g.Key.GetSize(o))
                    orderby size
                    select new
                    {
                        Name = g.Key.Name,
                        Size = size,
                        Count = g.Count()
                    };

        foreach (var item in stats)
            Console.WriteLine("{0,12:n0} {1,12:n0} {2}", item.Size, item.Count, item.Name);

This will output data like the following -- a collection of statistics about what objects are taking up the most space on the GC heap for your process:

           564           11 System.Int32[]
           616            2 System.Globalization.CultureData
           680           18 System.String[]
           728           26 System.RuntimeType
           790            7 System.Char[]
         5,788          165 System.String
        17,252            6 System.Object[]

ClrMD Features and Functionality

Of course, there’s a lot more to this API than simply printing out heap statistics. You can also walk every managed thread in a process or crash dump and print out a managed callstack. For example, this code prints the managed stack trace for each thread, similar to what the SOS !clrstack command would report (and similar to the output in the Visual Studio stack trace window):

        foreach (ClrThread thread in runtime.Threads)
        {
            Console.WriteLine("ThreadID: {0:X}", thread.OSThreadId);
            Console.WriteLine("Callstack:");

            foreach (ClrStackFrame frame in thread.StackTrace)
                Console.WriteLine("{0,12:X} {1,12:X} {2}", frame.InstructionPointer, frame.StackPointer, frame.DisplayString);

            Console.WriteLine();
        }

This produces output similar to the following:

        ThreadID: 2D90
        Callstack:
                   0       90F168 HelperMethodFrame
            660E3365       90F1DC System.Threading.Thread.Sleep(Int32)
              C70089       90F1E0 HelloWorld.Program.Main(System.String[])
                   0       90F36C GCFrame

Each ClrThread object also contains a CurrentException property, which may be null, but if not, contains the last thrown exception on this thread. This exception object contains the full stack trace, message, and type of the exception thrown.

ClrMD also provides the following features:

Gets general information about the GC heap:
- Whether the GC is workstation or server
- The number of logical GC heaps in the process
- Data about the bounds of GC segments
Walks the CLR’s handle table (similar to !gchandles in SOS).
Walks the application domains in the process and identifies which modules are loaded into them.
Enumerates threads, callstacks of those threads, the last thrown exception on threads, etc.
Enumerates the object roots of the process (as the GC sees them for our mark-and-sweep algorithm).
Walks the fields of objects.
Gets data about the various heaps that the .NET runtime uses to see where memory is going in the process (see ClrRuntime.EnumerateMemoryRegions in the ClrMD package).

All of this functionality can generally be found on the ClrRuntime or the ClrHeap objects, as seen above. IntelliSense can help you explore the various properties and functions when you install the ClrMD package. In addition, you can also use the attached sample code.

Please use the comments under this post to let us know if you have any feedback!

Social Engineering

Immo Landwerth [MSFT] — Wed, 24 Apr 2013 18:36:00 GMT

When people in our industry use the term “social engineering” they are usually referring to security attacks. Politicians and lobbyists would mean something entirely different too. But that’s not what I will talk about here. Instead, I’ll talk about the kind of social life we’ve around in our team.

Why am I using the term engineering then? Because I strongly believe that building a great team is a conscious effort. Wikipedia defines the term engineering as follows:

Engineering is the application of scientific, economic, social, and practical knowledge, in order to design, build, and maintain structures, machines, devices, systems, materials and processes.

From my point of view, building great software is a lot easier if you have a great team. You obviously can’t build a great team by writing a spec and triaging bugs. But you nonetheless need to invest into it as it doesn’t come for free or even happen automatically.

In this post, I’d like to show some of the big and small things that my team does and what makes working with this team so great – even when being under stress.

Official: Morale Events

About twice a year each team at Microsoft has a morale event. It’s up to the team to decide what to do but it typically involves some kind of team sport and food. Usually, these events are paid for by Microsoft. But before you think Formula 1 and Champaign with caviar: like in almost every company budget for those type of things is limited. However, I don’t think a successful morale event needs a lot of money. All it needs is a point in time and a place to gather.

About a month ago, our team went out to play kickball and have a BBQ in a public park. In case you wonder whether we had fun: you can judge by browsing the pictures below.

(Click picture or this link to see more pictures)

Semi Official: Get Together

It would miss the point if I’d state that it’s only the employer’s responsibility to invest into team building. In fact, I’m not even sure building a team like that is possible. From personal experience I’d say that in all the teams I’ve worked in the ones with the best culture had one thing in common: most of the team building was driven by individual team members.

In our team, we usually hang out during lunch time instead of using this time of the day to be productive in front of the computer. However, I don’t mean to say you absolutely have to have a team lunch. But in my experience talking to your coworkers in an unofficial environment, such as the company’s cafeteria or a sushi place, is super helpful. For one, you have a venue to talk about personal stuff, such as renovating your living room or the tough nights you had because your daughter is getting teeth. And secondly, chances are you talk about work stuff as well. I found that people are more open to make compromises when they talk about the issue in a relaxing environment. Also, I’ve found on more than one occasion we developed a brilliant idea during lunch.

Joel takes this one step further and has lunch delivered to his employees so that they can all sit together as one big family.

In addition to that, we meet every Friday after work in a social area in our building. Each week, someone else is responsible for getting the beer, chips and cake. Yes, there will be cake.

Unofficial: Office Pranks

At Microsoft, we have a long tradition of office pranks. They range from smaller jokes like removing the computer equipment to more elaborate pranks like replacing the door with a fully painted dry-wall. Needless to say they are unofficial and not driven by Microsoft, but completely organized by individual team members.

When I went on a conference trip to Germany a few weeks ago, my team pulled off a super awesome office prank. Since I’m a big Star Trek fan they got inspired by this episode and flooded my office with tribbles. Great pictures and epic making-of mails can be found below.

(Click picture or this link to see more pictures)

Special thanks to our release PM Marc Goodner who organized and PM’ed this epic prank!

Summary

In case you didn’t notice, I wrote this post primarily to say Thanks to my team. We talk a lot about the software we ship but cool stuff deserves to be shared – even it’s not shipping and especially when it involves people.

In case you thought “I wish I could be part of such a team” I’ve good news for you. We are hiring!

As always, we love to hear what you have to say. What do you do to keep up the team spirit? Share your story in the comments!

.NET Memory Allocation Profiling with Visual Studio 2012

The .NET Team — Thu, 04 Apr 2013 15:54:00 GMT

This post was written by Stephen Toub, a frequent contributor to the Parallel Programming in .NET blog. He shows us how Visual Studio 2012 and an attention to detail can help you discover unnecessary allocations in your app that can prevent it from achieving higher performance.

Visual Studio 2012 has a wealth of valuable functionality, so much so that I periodically hear developers that already use Visual Studio asking for a feature the IDE already has and that they’ve just never discovered. Other times, I hear developers asking about a specific feature, thinking it’s meant for one purpose, not realizing it’s really intended for another.

Both of these cases apply to Visual Studio’s .NET memory allocation profiler. Many developers that could benefit from it don’t know it exists, and other developers have an incorrect expectation for its purpose. This is unfortunate, as the feature can provide a lot of value for particular scenarios; many developers will benefit from understanding first that it exists, and second the intended scenarios for its use.

Why memory profiling?

When it comes to .NET and memory analysis, there are two primary reasons one would want to use a diagnostics tool:

To discover memory leaks. Leaks on a garbage-collecting runtime like the CLR manifest differently than do leaks in a non-garbage-collected environment, such as in code written in C/C++. A leak in the latter typically occurs due to the developer not manually freeing some memory that was previously allocated. In a garbage collected environment, however, manually freeing memory isn’t required, as that’s the duty of the garbage collector (GC). However, the GC can only release memory that is provably no longer being used, meaning as long as there are no rooted references to the memory. Leaks in .NET code manifest then when some memory that should have been collected is incorrectly still rooted, e.g. a reference to the object occurs in an event handler registered with a static event. A good memory analysis tool might help you to find such leaks, such as by allowing you to take snapshots of the process at two different points and then comparing those snapshots to see which objects stuck around for the second point, and more importantly, why.
To discover unnecessary allocations. In .NET, allocation is often quite cheap. This cost is deceptive, however, as there are more costs later when the GC needs to clean up. The more memory that gets allocated, the more frequently the GC will need to run, and typically the more objects that survive collections, the more work the GC needs to do when it runs to determine which objects are no longer reachable. Thus, the more allocations a program does, the higher the GC costs will be. These GC costs are often negligible to the program’s performance profile, but for certain kinds of apps, especially those on servers that require high-throughput operation, these costs can add up quickly and make a noticeable impact to the performance of the app. As such, a good memory analysis tool might help you to understand all of the allocation being done by the program, in order to help spot allocations you can potentially avoid.

The .NET memory profiler included in Visual Studio 2012 (Professional and higher versions) was designed primarily to address the latter case of helping to discover unnecessary allocations, and it’s quite useful towards that goal, as the rest of this post will explore. The tool is not tuned for the former case of finding and fixing memory leaks, though this is an area the Visual Studio diagnostics team is looking to address in depth for the future (you can see such an experience for JavaScript that was added to Visual Studio as part of VS2012.1). While the tool today does have an advanced option to track when objects are collected, it doesn’t help you to understand why objects weren’t collected or why they were held onto longer than was expected.

There are also other useful tools in this space. The downloadable PerfView tool doesn’t provide as user-friendly an interface as does the .NET memory profiler in Visual Studio 2012, but it is a very powerful tool that supports both tasks of finding memory leaks and discovering unnecessary allocations. It also supports profiling Windows Store apps, which the .NET memory allocation profiler in Visual Studio 2012 does not currently support as of the writing of this post.

Example to be optimized

To better understand the memory profiler’s role and how it can help, let’s walk through an example. We’ll start with the core method that we’ll be looking to optimize (in a real-world case, you’d likely be analyzing your whole application and narrowing in on the particular offending areas, but for the purpose of this example, we’ll keep this constrained):

public static async Task<T> WithCancellation1<T>(this Task<T> task, CancellationToken cancellationToken)
{
    var tcs = new TaskCompletionSource<bool>();
    using (cancellationToken.Register(() => tcs.TrySetResult(true)))
    if (task != await Task.WhenAny(task, tcs.Task))
        throw new OperationCanceledException(cancellationToken);
    return await task;
}

The purpose of this small method is to enable code to await a task in a cancelable manner, meaning that regardless of whether the task has completed, the developer wants to be able to stop waiting for it. Instead of writing code like:

T result = await someTask;

the developer would write:

T result = await someTask.WithCancellation1(token);

and if cancellation is requested on the relevant CancellationToken before the task completes, an OperationCanceledException will be thrown. This is achieved in WithCancellation1 by wrapping the original task in an async method. The method creates a second task that will complete when cancellation is requested (by Registering a call to TrySetResult with the CancellationToken), and then uses Task.WhenAny to wait for either the original task or the cancellation task to complete. As soon as either does, the async method completes, either by throwing a cancellation exception if the cancellation task completed first, or by propagating the outcome of the original task by awaiting it. (For more details, see the blog post “How do I cancel non-cancelable async operations?”)

To understand the allocations involved in this method, we’ll use a small harness method:

using System;
using System.Threading;
using System.Threading.Tasks;
 
class Harness
{
    static void Main() 
     { 
         Console.ReadLine(); // wait until profiler attaches
         TestAsync().Wait(); 
     }
    static async Task TestAsync()
    {
        var token = CancellationToken.None;
        for(int i=0; i<100000; i++)
            await Task.FromResult(42).WithCancellation1(token);
    }
}
 

static class Extensions
{
    public static async Task<T> WithCancellation1<T>(
    this Task<T> task, CancellationToken cancellationToken)
    {
        var tcs = new TaskCompletionSource<bool>();
        using (cancellationToken.Register(() => tcs.TrySetResult(true)))
            if (task != await Task.WhenAny(task, tcs.Task))
                throw new OperationCanceledException(cancellationToken);
        return await task;
    }
}

The TestAsync method will iterate 100,000 times. Each time, it creates a new task, invokes the WithCancellation1 on it, and awaits the result of that WithCancellation1 call. This await will complete synchronously, as the task created by Task.FromResult is returned in an already completed state, and the WithCancellation1 method itself doesn’t introduce any additional asynchrony such that the task it returns will complete synchronously as well.

Running the .NET memory allocation profiler

To start the memory allocation profiler, in Visual Studio go to the Analyze menu and select “Launch Performance Wizard…”. This will open a dialog like the following:

Choose “.NET memory allocation (sampling)”, click Next twice, followed by Finish (if this is the first time you’ve used the profiler since you logged into Windows, you’ll need to accept the elevation prompt so the profiler can start). At that point, the application will be launched and the profiler will start monitoring it for allocations (the harness code above also requires that you press ‘Enter’, in order to ensure the profiler has attached by the time the program starts the real test). When the app completes, or when you manually choose to stop profiling, the profiler will load symbols and will start analyzing the trace. That’s a good time to go and get yourself a cup of coffee, or lunch, as depending on how many allocations occurred, the tool can take a while to do this analysis.

When the analysis completes, we’re presented with a summary of the allocations that occurred, including highlighting the functions that allocated the most memory, the types that resulted in the most memory allocated, and the types with the most instances allocated:

From there, we can drill in further, by looking at the allocations summary (choose “Allocation” from the “Current View” dropdown):

Here, we get to see a row for each type that was allocated, with the columns showing information about how many allocations were tracked, how much space was associated with those allocations, and what percentage of allocations mapped back to that type. We can also expand an entry to see the stack of method calls that resulted in these allocations:

By selecting the “Functions” view, we can get a different pivot on this data, highlighting which functions allocated the most objects and bytes:

Interpreting and acting on the profiling results

With this capability, we can analyze our example’s results. First, we can see that there’s a substantial number of allocations here, which might be surprising. After all, in our example we were using WithCancellation1 with a task that was already completed, which means there should have been very little work to do (with the task already done, there is nothing to cancel), and yet from the above trace we can see that each iteration of our example is resulting in:

Three allocations of Task`1 (we ran the harness 100K times and can see there were ~300K allocations)

Two allocations of Task[]

One allocation each of TaskCompletionSource`1, Action, a compiler-generated type called <>c_DisplayClass2`1, and some type called CompleteOnInvokePromise

That’s nine allocations for a case where we might expect only one (the task allocation we explicitly asked for in the harness by calling Task.FromResult), with our WithCancellation1 method incurring eight allocations.

For helper operations on tasks, it’s actually fairly common to deal with already completed tasks, as often times operations implemented to be asynchronous will actually complete synchronously (e.g. one read operation on a network stream may buffer into memory enough additional data to fulfill a subsequent read operation). As such, optimizing for the already completed case can be really beneficial for performance. Let’s try. Here’s a second attempt at WithCancellation, one that optimizes for several “already completed” cases:

    public static Task<T> WithCancellation2<T>(this Task<T> task, 
                             CancellationToken cancellationToken)
    {
        if (task.IsCompleted || !cancellationToken.CanBeCanceled)
            return task;
        else if (cancellationToken.IsCancellationRequested)
            return new Task<T>(() => default(T), cancellationToken);
        else
            return task.WithCancellation1(cancellationToken);
    }

This implementation checks:

First, whether the task is already completed or whether the supplied CancellationToken can’t be canceled; in both of those cases, there’s no additional work needed, as cancellation can’t be applied, and as such we can just return the original task immediately rather than spending any time or memory creating a new one.

Then whether cancellation has already been requested; if it has, we can allocate a single already-canceled task to be returned, rather than spending the eight allocations we previously paid to invoke our original implementation.

Finally, if none of these fast paths apply, we fall through to calling the original implementation.

Re-profiling our micro-benchmark while using WithCancellation2 instead of WithCancellation1 provides a much improved outlook (you’ll likely notice that the analysis completes much more quickly than it did before, already a sign that we’ve significantly decreased memory allocation). Now we have just have the primary allocation we expected, the one from Task.FromResult called from our TestAsync method in the harness:

So, we’ve now successfully optimized the case where the task is already completed, where cancellation can’t be requested, or where cancellation has already been requested. What about the case where we do actually need to invoke the more complicated logic? Are there any improvements that can be made there?

Let’s change our benchmark to use a task that’s not already completed by the time we invoke WithCancellation2, and also to use a token that can have cancellation requested. This will ensure we make it to the “slow” path:

    static async Task TestAsync()
    {
        var token = new CancellationTokenSource().Token;
        for (int i = 0; i < 100000; i++)
        {
            var tcs = new TaskCompletionSource<int>();
            var t = tcs.Task.WithCancellation2(token);
            tcs.SetResult(42);
            await t;
        }
    }

Profiling again provides more insight:

On this slow path, there are now 14 total allocations per iteration, including the 2 from our TestAsync harness (the TaskCompletionSource<int> we explicitly create, and the Task<int> it creates). At this point, we can use all of the information provided by the profiling results to understand where the remaining 12 allocations are coming from and to then address them as is relevant and possible. For example, let’s look at two allocations specifically: the <>c__DisplayClass2`1 instance and one of the two Action instances. These two allocations will likely be logical to anyone familiar with how the C# compiler handles closures. Why do we have a closure? Because of this line:

using(cancellationToken.Register(() => tcs.TrySetResult(true)))

The call to Register is closing over the ‘tcs’ variable. But this isn’t strictly necessary: the Register method has another overload which instead of taking an Action takes an Action<object> and the object state to be passed to it. If we instead rewrite this line to use that state-based overload, along with a manually cached delegate, we can avoid the closure and those two allocations:

private static readonly Action<object> s_cancellationRegistration =
    s => ((TaskCompletionSource<bool>)s).TrySetResult(true);
…
using(cancellationToken.Register(s_cancellationRegistration, tcs))

Rerunning the profiler confirms those two allocations are no longer occurring:

Start profiling today!

This cycle of profiling, finding and eliminating hotspots, and then going around again is a common approach towards improving the performance of your code, whether using a CPU profiler or a memory profiler. So, if you find yourself in a scenario where you determine that minimizing allocations is important for the performance of your code, give the .NET memory allocation profiler in Visual Studio 2012 a try. Feel free to download the sample project used in this blog post.

For more on profiling, see the blog of the Visual Studio Diagnostics team, and ask them questions in the Visual Studio Diagnostics forum.

Stephen Toub

Follow us on Twitter (@dotnet) and Facebook (dotnet). You can follow other .NET teams, too: @aspnet/asp.net, @efmagicunicorns/efmagicunicorns, @visualstudio/visualstudio.

.NET Framework Documentation Improvements

Brandon Bray — Tue, 19 Feb 2013 20:07:49 GMT

The CLR documentation team has been busy responding to feedback and making updates and changes to the .NET Framework documentation in the MSDN Library. We would like to tell you about the most recent set of document updates, which were published earlier in February.

Performance content

We have received extensive customer feedback regarding the importance of performance in .NET Framework apps, and we wanted to make it easier for you to find relevant content.

As a result, we’ve reworked the existing performance and reliability topic to include more performance guidance as well as links to performance analysis tools and technology-specific performance content. We’ll continue to improve this documentation area in the coming months.

Do take a look at the revised Performance in .NET Framework Apps topic. Please give us your feedback to help us choose which area to focus on next.

Overload list topics

We are experimenting with a new approach to document methods that have overloads. These pages have been an area of consistent feedback. We have heard that the current approach requires too many clicks to get to the overload page that you want. We are piloting this new approach with the following method overload pages:

Let’s take a look at why we decided to try something different with these pages. In the MSDN Library, our current documentation design for overloaded methods and constructors uses an overload list to provide summary information and links to individual overload pages. Typically, these pages include just the overload list, requiring you to choose one of the overloads to find any technical information for that method beyond just the overload signatures. For example, the following image shows a portion of the String.Compare page.

The overload list provides an abbreviated syntax for each overload (without its return value or parameter names) and a brief, typically one-sentence description. The page was designed to provide just enough information for you to select an overload, and then follow a link to detailed documentation about that overload. However, we have heard many of you ask for more detail on the overload page, such as individual parameters, return value information, and examples.

Until now, our strategy has been to direct you to the detailed information on the individual overload pages. We had done two things to achieve that goal:

Added the following note before the overload list: "This member is overloaded. For complete information about this member, including syntax, usage, and examples, click a name in the overload list."
Reduced the visibility of these pages by ensuring that they do not appear in searches and member lists.

However, even after those changes, we were still hearing requests for more information. To address this issue, we would like to solicit your feedback about the new approach we're trying out. As already stated, we've revised four overload list topics to provide complete documentation for all the individual method overloads. In each case, the overload list topic should serve as a one-stop source for information about that method and all its overloads.

Note, though, that our build system does not yet support this revised format. Because of this, the boilerplate text "This method is overloaded…" continues to appear on these overload list pages. At a later time, we will remove that text.

The expanded documentation is in the Remarks section and includes complete method syntax, parameter and return value descriptions, a list of exceptions, a table to help choose an overload, extended discussion of using the method, and at least one example for each overload. Here are a few screenshots from the updated String.Format overload page to give you a better idea of the information that you can expect on overload pages with this new approach. Feel free to check out the whole page for yourself.

Tell us what you think of this approach. Does it make it easier to get to the information you need? Do you have any suggestions for further improvements? We look forward to hearing from you.

Follow us on Twitter (@dotnet) and Facebook (dotnet). You can follow other .NET teams, too: @aspnet/asp.net, @efmagicunicorns/efmagicunicorns.

C# is the “Language of the Year”

Brandon Bray — Fri, 04 Jan 2013 01:25:09 GMT

It’s a great way to start a new year: the PYPL index has named C# the language of the year. This index focuses on the leading indicator of learning a language. It’s nice to see that there’s been increased interest in C# over the past year. In the spirit of learning something new, here’s what we suggest you check out if you’re looking to learn a new C# trick.

Learn to program with Async. Async is finally unleashing the power of multicore hardware. Where I/O is involved, we expect almost all new platform APIs to be exposed as async versions. This new language capability ensures that you can quickly build highly scalable and highly responsive apps for the cloud and devices.
Learn to build a Windows Store App. All of our favorite Windows Store apps were written with C# – yours can be too. Many of the new Windows APIs are based on WinRT, which means that all of these new APIs are amazing with C#. To top it off, you can use XAML or DirectX.
Learn to build (or use) cross-platform libraries. .NET is now integral to every platform Microsoft builds. And with so many platforms to target, we’ve made it easy to write portable libraries so you can distribute your binary once for all of them. Many of these libraries are available on NuGet such as Json.NET. Not to mention that your C# code also works on billions of non-Microsoft devices thanks to tools from Xamarin.
Learn about the performance advantages of .NET. One of the best aspects of programming with C# is that your apps get faster by just upgrading to the latest version of Windows or .NET Framework. With Windows Server 2012, Windows 8 and .NET Framework 4.5 your app can see significant improvements in garbage collection and startup time. Bing saw some amazing performance wins from .NET too.
Learn about cool libraries that build on top of C#. What makes C# so cool is that our innovations in the language have inspired new programming patterns and libraries. LINQ is awesome and it’s led to the Reactive Extensions (Rx). Just spend some time understanding how to use Rx and you’ll take your programming game to a new level.

What new C# skill would you recommend everyone learn? Share your thoughts in the comments.

Downloading the .NET Framework and Visual Studio Express Editions and SDKs

Brandon Bray — Mon, 17 Dec 2012 22:35:32 GMT

We see a lot of daily traffic through the .NET blog and the .NET dev center on MSDN. It is clear from the web traffic data that a key task for many of you is downloading more .NET software, to expand your options and experience developing .NET apps. The number one download for .NET is the .NET Framework –the latest version being the .NET Framework 4.5. However, we also see a lot of activity downloading Visual Studio and the many SDKs that you can install to expand its capabilities.

The purpose of this post is to direct you to the most streamlined way of downloading the latest versions of .NET, Visual Studio Express Editions and SDKs. By “SDKs”, I mean SDK releases like Windows Azure and Windows Phone. Again, it is clear from the traffic data that a lot of you are after those downloads, to start building apps for the various platforms that Microsoft offers to you. However, it appears that many of would probably appreciate a more streamlined way to achieve your download tasks.

Most popular .NET Framework downloads

To start off, I’m including download links to the most popular .NET Framework downloads.

Just in time download

As part of Visual Studio 2010, we added a new experience to help you find the products that you want to download, right when you want them. We improved on this within Visual Studio 2012, however, the general experience is the same. You’ve probably seen this option before, and may have taken advantage of it.

The experience within Visual Studio takes you to a web page, specific to that version of Visual Studio. These pages list the additional downloads that are available to you, to expand your development options. If you notice that a product download is missing from that list, please tell us! Here are links to those two pages:

Here is a partial screenshot of the page for Visual Studio 2012, to give you a quick idea of what you can expect from these pages.

You can bookmark those links for later use. As I said earlier, we integrated getting to these pages into Visual Studio. You can think of these pages as an extension of Visual Studio. Let’s take a look at the various places within Visual Studio 2012 and Visual Studio 2010 that enable you to quickly get to these pages. I’ll show all of the experiences available, always showing the Visual Studio 2012 experience first. Feel free to try them all.

New Project Dialog

Selecting “More Frameworks” will take you to the .NET Framework Multi-Targeting Packs and SDKs download page.

Target Framework in Project Properties

Selecting “Install other Frameworks” will take you to the .NET Framework Multi-Targeting Packs and SDKs download page.

Add or Change Target Framework for a Portable Class Library

Selecting “Install additional Frameworks” will take you to the .NET Framework Multi-Targeting Packs and SDKs download page.

I only provided one set of screenshots since the UI is almost identical between Visual Studio 2012 and 2010.

Popular Downloads are available at the .NET Framework Dev Center on MSDN

The .NET Framework dev center is another place where we make download links available. In the screenshot below, you can see a set of .NET tools, SDKS and Visual Studio Express Editions that you can download and install.

On another part of the .NET Framework dev center page, you can find a list of .NET Framework downloads and NuGet Packages

You can also find a lot of other useful links on the Visual Studio dev center page.

Wrapping Up

The opportunities for .NET developers seems to grow every year. It is easy to target each of the Microsoft platforms with your existing skills and code. You can take the first step by downloading a new version of .NET, an SDK for Visual Studio or a Visual Studio Edition.

Once again, if you are looking for a .NET version, SDK or tool, take a look at one of these pages:

We intend for these pages to be complete. Please do tell us we’re missing some important Microsoft SDKs or tools for .NET that you feel should be on this page.

Follow us on Twitter (@dotnet) and Facebook (dotnet). You can follow other .NET teams, too: @aspnet/asp.net, @efmagicunicorns/efmagicunicorns.

Talking about .NET in the Community Press

Brandon Bray — Wed, 12 Dec 2012 23:35:12 GMT

We follow a lot of blogs and digital papers relating to both .NET and also technology in general. There is a lot of great content that gets published every day about .NET. We’re starting a new series to publish these great links, doing our part to help give that content a larger audience.

For this post, I wanted to call out some of the journalists in the community press that have been regularly including links to our posts. We’ve noticed the traffic from your domains! Thanks for expanding awareness to our blog and many other great .NET blogs.

Paper.Li

Blogs

If you are not familiar with these papers and blogs, please do visit them.

Follow us on Twitter (@dotnet) and Facebook (dotnet). You can follow other .NET teams, too: @aspnet/asp.net, @efmagicunicorns/efmagicunicorns.

Asynchronous Programming for Windows Store Apps: .NET is up to the Task

Brandon Bray — Fri, 30 Nov 2012 15:07:00 GMT

Programming with async and await is the driving force behind delivering “fast and fluid” user experiences. Compiler diagnostics for async APIs guide developers towards best practices – it’s a wonderful example of platform and tooling working together. Since the Windows Runtime uses a slightly different implementation for async programming than .NET, Greg Paperin – a developer on the .NET base class libraries – explains how C# and Visual Basic still provide a unified development experience between the two sets of APIs. –Brandon

Asynchronous Programming in Windows Runtime APIs

Asynchronous programming is one of the central themes of the .NET Framework 4.5 and Visual Studio 2012 releases. The async and await keywords in C# 5 and Visual Basic 11 make writing and consuming asynchronous code easier than ever before. Visual Studio 2012 makes it easier to write and debug asynchronous code. And the .NET Framework 4.5 completes the Async story with new Task-based “XyzAsync” APIs, expansions of the Task Parallel Library and performance enhancements. We have blogged about some of these features recently (e.g. "Async in 4.5: Worth the Await" and "Enabling Progress and Cancellation in Async APIs").

The Windows Runtime (WinRT) has also adopted the asynchronous programming model. The WinRT APIs, introduced in Windows 8, are central to writing Windows Store apps. Most WinRT APIs that can have an execution time that is noticeable to a human user are asynchronous, and synchronous alternatives are not available. These choices were made to ensure that developers consistently picked APIs that deliver great end-user experiences.

WinRT async APIs are different than the .NET Task-based async APIs. However, you can use WinRT async APIs as seamlessly as pure .NET APIs, with the same keywords and the same usage principles. In this article we will go into detail on how you can do that.

We will begin with an overview of the most important types in the WinRT async infrastructure, and how they relate to similar .NET concepts. After that we will walk over creating a complete Windows Store App with several different asynchronous operations. We will learn how to start different kinds of asynchronous operations, how to retrieve results, and how to work with cancellations and progress monitoring.

This article assumes that you have a basic familiarity with using the async and await keywords with .NET APIs in C#. If you do not, try reading some of our previous articles on this topic first (introductory article; details on cancellation and on progress monitoring).

The WinRT Asynchronous model for .NET programmers

Before we begin, let us take a look at one example of calling an asynchronous WinRT API. The following statement fetches a list of files in the local user's video library folder.

IReadOnlyList<StorageFile> videoFiles = await KnownFolders.VideosLibrary.GetFilesAsync();

Remarkably, GetFilesAsync() is not a .NET Task-based method, but a WinRT asynchronous API:

public interface IStorageFolder : IStorageItem {
// ... 
IAsyncOperation<IReadOnlyList<StorageFile>> GetFilesAsync();
}

And yet, we use it just like a first class Task-based asynchronous method: with the await keyword. Intrigued? Let's get started.

Key WinRT Async types

In the .NET Framework 4.5, asynchronous await-able APIs return an instance of type Task or Task<TResult>. You can think of a Task object as a receipt certifying that you requested an operation to be undertaken. You hold on to the receipt and use it to determine if and when the operation has completed, whether any errors occurred and what the result was.

You can use a Task in an await statement to ensure that subsequent statements are executed after the work represented by this "receipt" has completed and the result became available, without blocking. Operations that have a result (e.g., read some data from a stream) return an object of type Task<TResult>, and actions that cause an effect, but do not have a result return an object of type Task (e.g. send some bytes over a wire).

The proper computer science term for this "receipt"-concept is future, because the receipt (Task) represents an operation that may be completed in the future. (Note: Futures are alternatively known as promises because they represent a promise to complete an operation; some people also make subtle differences between the meaning of futures and promises, but that detail is not relevant for our current purposes.)

WinRT also has a concept of futures, however it does not use the same types as .NET to represent them. A future type in WinRT must implement one of the four interfaces IAsyncAction, IAsyncActionWithProgress, IAsyncOperation, IAsyncOperationWithProgress, and all of these four interfaces inherit from one common interface IAsyncInfo. The following diagram describes the four interfaces and lists their public APIs (see also the API reference on Windows Dev Center).

WinRT types for representing futures

Relationship between key Async types in WinRT and .NET

These four WinRT types have a close correspondence to tasks in .NET:

IAsyncAction and IAsyncActionWithProgress<TProgress> describe an asynchronous action that does not return anything directly – they correspond to Task.
IAsyncOperation<TResult> and IAsyncOperationWithProgress<TResult, TProgress> represent an asynchronous operation that returns a value of type TResult – they correspond to Task<TResult>.

The progress-related versions are identical to each respective interface with the additional ability to publish the progress of an on-going asynchronous operation. We will overlook this detail for now and come back to it later in the section "Cancellation and Progress in WinRT asynchronous operations". The following figure illustrates the conceptual relationship between tasks and WinRT Async Infos.

Conceptual relationship between asynchronous actions and operations in WinRT and .NET

.NET also has the IProgress interface used to monitor the progress of an asynchronous operation and the CancellationToken/CancellationTokenSource tandem used to signal that a consumer requested an operation to cancel. WinRT does not have corresponding types. Instead, cancellation and progress monitoring is achieved using member APIs of the four IAsyncXxxx types.

You may wonder about the reason behind these differences between .NET and WinRT Async types. Why not just use Task? Remember that WinRT APIs are available in languages that do not use the .NET Common Language Runtime to execute. For instance, you can write Windows Store apps in non-.NET languages such as JavaScript and native C++, and all WinRT types need to be available there too.

Task is a CLR type and is only available in languages that rely on the .NET runtime. This is why WinRT introduces its own asynchrony abstractions and .NET makes sure that these abstractions can be seamlessly imported into the Task-based universe while still accounting for their distinctive features. What does this look like? Let's see it first-hand!

Awaiting WinRT Async APIs like Tasks

Recall the earlier example where we used await with the WinRT GetFilesAsync() method that returns an instance of IAsyncOperation<IReadOnlyList<StorageFile>> and not of Task<IReadOnlyList<StorageFile>>.

await KnownFolders.VideosLibrary.GetFilesAsync();

If we dig deeper into the workings of the "await" keyword we learn that it causes the GetAwaiter() method to be called on the awaited entity. Although the IAsyncOperation<TResult> interface returned from GetFilesAsync() does not define such a method, an equivalent extension method is defined by the .NET Framework in the System.Runtime.WindowsRuntime assembly. This extension method creates a Task that represents the specified WinRT async operation in the Task-based universe and simply uses the task's already existing GetAwaiter() infrastructure. For that, it uses the AsTask() extension method, also defined in the System.Runtime.WindowsRuntime assembly. Here is a simplified implementation of the GetAwaiter() extension method:

public static TaskAwaiter GetAwaiter<TResult>(this IAsyncOperation<TResult> source) {
return AsTask(source).GetAwaiter();
}

The bulk of the logic that allows the .NET and WinRT asynchronous models to interoperate lies within the AsTask(..) extension method. It is a public method that you can use directly, however, in many cases the compiler will make sure it is called automatically on your behalf. Nevertheless, in some scenarios you will need to call AsTask(..) explicitly. We will discuss an example of that in section "Cancellation and Progress in WinRT asynchronous operations".

Using WinRT Async APIs in a Windows Store App based on .NET

Let's walk through writing a small Windows Store app that makes use of several different asynchronous operations. We will use WinRT APIs to create a video file format converter – an application that lets the user select a .WMV file, converts it to an .MP4 file and stores it into the local video library.

We will focus on the async APIs and will make due with a very simple UI. Our app looks like this:

Sample video converter Windows Store App user interface after start-up

Asynchronous interactions with the user

The first thing our app needs to be able to do is to pick an input video file for conversion. Similar to traditional Windows desktop applications, there is an API for displaying a prompt for picking a file from the disk. However, in contrast to classic desktop applications we do not need to deal with modal dialog windows.

Asynchrony is all about thinking in terms of the operations that need to occur and the results of these operations, while abstracting away when the operation is actually performed (the system decides that for you). WinRT allows us to say "ask the user to pick a video file and use the result when it becomes available", and the async infrastructure will take care of displaying the input screen at the right time.

Note that the WinRT file picker takes up the full screen, but it is not modal: it does not block other UI components from working, even though you may not see them on the screen since they are covered. How many of you had to deal with annoying deadlocks caused by modal input dialogs? Asynchronous user input APIs solve this problem!

The pattern for picking an input file looks like this:

// Create a file picker component:
FileOpenPicker picker = new Windows.Storage.Pickers.FileOpenPicker();
// Configure the file picker:
picker.SuggestedStartLocation =
Windows.Storage.Pickers.PickerLocationId.VideosLibrary;
picker.FileTypeFilter.Add(".wmv");
// Request the system to ensure that the user has picked a file:
StorageFile file = await picker.PickSingleFileAsync();
// The system will get to this line once the user has picked a file,
// however long this might take.
// However, the UI thread does not block and can continue executing other
// work in the meantime.
// Now the picked file can be used:
if (file == null) {
// ...
} else {
// ...
}

The PickSingleFileAsync method delivers the magic that resolves the modal UI problem. Let's take a look at its signature:

public sealed class FileOpenPicker {
//... 
public IAsyncOperation<StorageFile> PickSingleFileAsync();
}

Note that it does not return a Task. Instead, it returns an IAsyncOperation object. However, you can use it with await just like as if it was a Task. Neat, isn't it? By the time you finished reading this article you will gain a general understanding of how this works.

Asynchronous WinRT operations can be awaited
in the same manner as managed Task-based operations.

To complete the picture, here is the complete method body for the "Pick input video"-button handler:

private async void PickInputButton_Click(Object sender, RoutedEventArgs e) {

// prepare the UI:
Log("Picking input file.");
PageState prevPageState = DisableButtons();

// Create and configure the file picker component:
FileOpenPicker picker = new Windows.Storage.Pickers.FileOpenPicker();
picker.SuggestedStartLocation =
Windows.Storage.Pickers.PickerLocationId.VideosLibrary;
picker.FileTypeFilter.Add(".wmv");
// Request the file picker to query the user for an input file asynchronously:
StorageFile file = await picker.PickSingleFileAsync();
// If no file was picked (e.g. picker was aborted), do nothing:
if (file == null) {
Log("Input file selection cancelled.");
SetPageState(prevPageState);
return;
}

// If a file was picked, store the file and enable Start Conversion button:
Log("Input file selected: " + file.Path);
_inputFile = file;
SetPageState(PageState.ReadyToConvert);
InputFileName.Text = file.Path;
}

I/O-bound asynchronous operations

In the previous section we saw how to use await to make sure that your Windows Store app is not blocked while the user is performing some input. Another event that may cause an application to become unresponsive is data input and output. While some I/O operations are fast (think reading a few bytes from a modern solid state drive), others can stall your applications for a significant time (think spinning up a disk drive, accessing a network location or a web service, or just reading a very large file).

In our case we need to transcode a video file. At the very least, it involves reading the entire file and writing the transcoded version back to disk, and you know that video files can very quickly grow up to many megabytes, if not gigabytes. To ensure that your applications remain responsive, all I/O bound operations in WinRT are asynchronous, including video transcoding. Let us see how this is done.

The transcoding operation consists of two stages. We need to first determine if a specific input file can be converted to a specific output format. In our case, we want to convert a WMV file to MP4. To achieve this, WinRT provides a universal transcoder component that can answer these questions. The transcoder component will examine the required format information and the input file on the disk and it will report whether a conversion is possible by returning a PrepareTranscodeResult object. This process can take a couple of milliseconds up to a few seconds.

Second, if the conversion is indeed possible, we request from the PrepareTranscodeResult object to actually perform the conversion. At that stage, the entire input file will be read and the output file will be written. Depending on the length of the video, this process can take several minutes.

The coding pattern for this looks like this:

// Create the transcoder component and specify the output video file format:
MediaTranscoder transcoder = new MediaTranscoder();
MediaEncodingProfile profile=MediaEncodingProfile.CreateMp4(VideoEncodingQuality.Vga);

// Analyze input file and output format to determine if conversion if possible:
// (we do this asynchronously, so the app does not freeze)
PrepareTranscodeResult transcode =
await transcoder.PrepareFileTranscodeAsync(_inputFile, outputFile, profile);

// When we continue here, the transcode result is already available.
if (!transcode.CanTranscode) {
// Transcoding of the provided input file into the specified output
// format is not supported. Report and give up.
} else {
// Transcoding is possible. Call transcode.TranscodeAsync() to perform conversion. 
// We will see soon how this is done.
}

Cancellation and Progress in WinRT asynchronous operations

In the previous section, we used the await keyword with the I/O bound asynchronous operation PrepareFileTranscodeAsync(). Because it is a WinRT method, rather than returning a Task, it returns a WinRT asynchronous type; in this case, IAsyncOperation<PrepareTranscodeResult>. Notably, PrepareFileTranscodeAsync does not need to do much work to provide its result – the system only needs to examine a fraction of the input file to understand its format. Even if the user changes their mind right after clicking the conversion button, they can wait until the operation is completed.

However, the actual transcoding (achieved by the PrepareTranscodeResult.TranscodeAsync() API) can take many minutes depending on the length of the video file. A user may have every reason to want to cancel the on-going conversion process. Moreover, a user-friendly application will certainly want to communicate the progress of the operation to the user: After all, wouldn't you want to know whether you have the time for an instant coffee, or whether you have enough time to prepare a nice latte? Or perhaps even enough time to finally clean those dishes? No, you don't want to do that! See – the cancel feature is essential.

As discussed in our previous article on Async, Task-based asynchronous APIs that support cancellation and/or progress monitoring have overloads that take CancellationToken and/or IProgress<TProgress> parameters respectively. But if we inspect the WinRT PrepareTranscodeResult class looking for overloads of TranscodeAsync, we cannot discover any trace of CancellationToken or IProgress.

Recall that in section "Key WinRT Async types" above we mentioned that in contrast to the Task-based universe, the cancellation control and the progress observer in WinRT are located directly on the future types (IAsyncXzy). So how can we use them? We could use the object returned from TranscodeAsync() directly, instead of awaiting it. There is a Cancel() method we can call and a Progress delegate property we can set. In fact, this would work in many cases. However, this is a different thought model compared to Task-based APIs and this approach would mean that we needed to use different paradigms when consuming WinRT versus Task-based asynchronous APIs. Not ideal. Instead, we want to ensure an identical first-class experience for Task-based and WinRT futures.

And indeed, such first-class experience is available. TranscodeAsync() returns an instance of IAsyncActionWithProgress<Double>. Let's consider what actually happens when the following line is evaluated:

await transcode.TranscodeAsync();

Recall from section "Awaiting WinRT Async APIs like Tasks" that the await keyword causes the compiler to emit a call to a GetAwaiter() method. The .NET Framework defines GetAwaiter() extension methods for all four WinRT async interface types in the System.Runtime.WindowsRuntime assembly. For instance, here is a simplified implementation of GetAwaiter() for

IAsyncActionWithProgress<TProgress>:

public static TaskAwaiter GetAwaiter<TProgress>(
this IAsyncActionWithProgress<TProgress> source) {
return AsTask(source).GetAwaiter();
}

GetAwaiter() creates a Task that represents the specified WinRT async action in the Task-based universe and then uses the task's already existing GetAwaiter() infrastructure. The heavy lifting is done by the AsTask(..) extension method. It is a public method that you can use directly and it does have overloads that support the .NET-style cancellation and progress infrastructure.

The WindowsRuntimeSystemExtensions.AsTask() method family is a collection of factory methods that can take any of the four WinRT future types and create a Task that represents that WinRT future in the Task-based universe. In addition, AsTask factories can take CancellationToken and IProgress<TProgress> instances as parameters and appropriately connect them to the returned Task and thus to the underlying WinRT future. As a result, if a cancellation is requested using a token connected in this way, the cancellation request will be automatically forwarded to the WinRT operation by invoking its Cancel() method. Similarly, any progress updates published by the WinRT operation will be automatically forwarded to the specified progress observer.

The WindowsRuntimeSystemExtensions.AsTask() conversion methods create a Task that represents a WinRT asynchronous action or operation in the Task-based .NET world.

These conversion methods support optional parameters to specify a CancellationToken for requesting that a WinRT asynchronous procedure be cancelled, and an IProgress<T> observer to monitor the progress of the asynchronous procedure.

The following code sample demonstrates how you can use mix-and-match the WinRT async model with the .NET model using AsTask().

CancellationTokenSource _cancelControl = new CancellationTokenSource();

class TranscodeProgressHandler : IProgress<Double> {
public void Report(Double progressInfo) {
// Display a progress update to the user.
}
}

async void Transcode() {
PrepareTranscodeResult transcode = . . .
try {
await transcode.TranscodeAsync().AsTask(_cancelControl.Token,
new TranscodeProgressHandler());
} catch (OperationCanceledException) {

// There is no error.
// The transcoding simply responded to the cancellation request. Handle it.
} catch (Exception ex) {
// An error has occurred. Handle the error.
}
}

Recall that in .NET, cancellation – although not an error – is an exceptional condition, and is handled using exceptions. So if you hook up a potential cancellation request, you need to be prepared to handle an OperationCanceledException, as shown in the code sample above.

Let's now take a look at the full implementation of the "Start conversion" and the "Cancel conversion" button handlers and the relevant helper methods in our sample app:

// The cancellation controller used to cancel an ongoing transcoding.
// May be null when transcoding is idle:

private CancellationTokenSource _transcodeCancelControl;

/// <summary>
/// "Cancel conversion"-button handler.
/// </summary>
/// <param name="sender">Ignored.</param>
/// <param name="e">Ignored</param>

private void CancelConversionButton_Click(Object sender, RoutedEventArgs e) {

// If we have an initialised cancel controller, signal a cancellation request:
CancellationTokenSource cancelControl =Volatile.Read(ref _transcodeCancelControl);

if (cancelControl != null)
cancelControl.Cancel();
}

/// <summary>
/// Initialises the output file for the currently selected output file.
/// </summary>
/// <returns>an initialised output file
/// or <code>null</code> if it could not be initialised.</returns>

private async Task<StorageFile> CreateOutputFile() {
StorageFile outputFile = null;
try {
// Construct the output file name / path:
String inputPath = _inputFile.Path;
String baseName = Path.GetFileNameWithoutExtension(inputPath);
String extension = Path.GetExtension(inputPath);
String outputFileName = baseName + ".ConversionOutput.mp4";
// Asynchronously initialise the output file in the pictures library:
outputFile = await KnownFolders.VideosLibrary.CreateFileAsync(outputFileName,
CreationCollisionOption.GenerateUniqueName);
// Done:
return outputFile;
} catch (Exception ex) {
// If we cannot obtain an output file objects, we cannot convert.
 // Need another input file:
CleanupOutputFile(outputFile); 
Log(ex.ToString());
SetPageState(PageState.InputFileNeeded);
return null;
}
}

/// <summary>
/// Deletes unneeded output file.
/// </summary>
/// <param name="outputFile">File to delete.</param>
private static async void CleanupOutputFile(StorageFile outputFile) {
// Nothing to delete:
if (outputFile == null)
return;
try {
// Delete specified file asynchronously.
// Deletion errors are benign to this application, we will ignore them.
await outputFile.DeleteAsync();
} catch { };
}

/// <summary>
/// Event handler for the "Start Conversion"-button.
/// 1. Init output file.
/// 2. Validate transcoding can occur.
/// 3. Perform transcoding.
/// Also hook up and handle cancellation, progress updates and errors.
/// </summary>
/// <param name="sender">Ignored</param>
/// <param name="e">Ignored</param>

private async void StartConversionButton_Click(Object sender, RoutedEventArgs e) {
// Prepare the UI:
PageState prevPageState = DisableButtons();
// Determine the output file.
// If we cannot prepare the output, we cannot start conversion:
StorageFile outputFile = await CreateOutputFile();
if (outputFile == null)
return;
// Update the UI:
OutputFileName.Text = outputFile.Path;
try {
// Defensive checks for current application state:
// (I/O files must be ready)
if (_inputFile == null)
throw new InvalidOperationException("Unexpected state: _inputFile==null");
if (outputFile == null)
throw new InvalidOperationException("Unexpected state: outputFile==null");
Log("Preparing transcode to file: " + outputFile.Path);
// Asynchronously check if input file can be transcoded to target format:
MediaEncodingProfile profile =
MediaEncodingProfile.CreateMp4(VideoEncodingQuality.Vga);
PrepareTranscodeResult transcode =
await _transcoder.PrepareFileTranscodeAsync(_inputFile,
outputFile, profile);
// If we cannot transcode specified input to the target format, bail out:
if (!transcode.CanTranscode) {
Log("Transcode preparation failed. Reason: " +
transcode.FailureReason.ToString());
CleanupOutputFile(outputFile);
SetPageState(PageState.InputFileNeeded);
return;
}
// OK, we can transcode:
// Reset progress bar and set the UI state:
ProgressBar.Value = 0;
SetPageState(PageState.Converting);
// Transcode asynchronously:
await transcode.TranscodeAsync().AsTask(
_transcodeCancelControl.Token,
new Progress<Double>((p) => ProgressBar.Value = p));
// Done transcoding. Update UI:
Log("Transcoding complete.");
SetPageState(PageState.InputFileNeeded);
} catch (OperationCanceledException) {

// Transcoding has been cancelled. Delete incomplete output file and reset UI:
Log("Transcoding has been cancelled");
CleanupOutputFile(outputFile);
SetPageState(prevPageState);
} catch (Exception ex) {

// Error during transcoding. Delete incomplete output file and reset the UI:
Log("Error during transcoding: " + ex.ToString());
CleanupOutputFile(outputFile);
SetPageState(prevPageState);
}
}

There are a few things to note here:

We do not explicitly create a new type to monitor the progress. Instead we use the Progress<TProgressInfo> type and a lambda expression to specify the action to occur on progress update in-line.
The "Start conversion"-button handler is responsible for several things: First, it asynchronously initialises the output file on the hard drive. If this fails, we bail out before the conversion is even attempted. Then, it verifies whether the specified input file can be successfully converted to the specified format. If the conversion is not possible, it also bails out. Finally, it performs the actual conversion. Note that exceptional conditions such as cancellation and errors are handled in-line using the common try-catch mechanism. During the entire process the hander updates the UI as appropriate.
Recall that await causes the "rest of the method", i.e. the statements that follow the await statement, to be executed as a continuation of the future being awaited. I.e., when these subsequent statements are executed we know that the awaited operation has completed. If the execution proceeded with the line directly after the await – the execution has completed normally; if it jumped to the OperationCanceledException's catch clause – the execution has been successfully cancelled; if it jumped to the default catch clause – the operation faulted. Either way – it has finished. As a result, we can do some clean-up. For instance, we will dispose the _transcodeCancelControl in the SetPageState(..) method. It is also the SetPageState(..) method that initialises the cancel control lazily – right in time when the cancel-button is activated. You can see how this is done by downloading the complete source code of the video transcoding application we discussed in this article.

Summary and Pointers

In this article we learned about the Windows Runtime (WinRT) model for asynchronous programming. The model differs from the .NET Task-based universe because it must work well with other WinRT technologies such as JavaScript and native C++. However, .NET makes is simple to await WinRT asynchronous operations in exactly the same manner as you await Task-based operations. In addition, the AsTask(..) method family allows using traditional .NET patterns for cancellation and progress monitoring of asynchronous WinRT operations (i.e. CancellationToken/CancellationTokenSource pattern and the Progress<T> pattern).

It is possible to use methods and properties available on WinRT future types directly, but this is discouraged. Doing so would require learning a different set of programming patterns and prevent the use of the await keyword. It is complicated and very easy to get wrong. It also can require explicit use of the CoreDispatcher and/or the SynchronizationContext – complex concepts that do not need to otherwise be dealt with by most application programmers who use async and await keywords for writing Windows Store apps.

We walked through writing a complete Windows Store application that uses a number of asynchronous actions and operations exposed by WinRT. You can download the complete source for this project here: .NET Asynchronous Programming for Windows Store Apps.

There is another, larger, transcoding media sample app available from the Windows Dev Center. While here we primarily focussed on consuming asynchronous APIs, that sample goes into more detail on using WinRT transcoding APIs with different options.

There is a number of articles that can help you learning more about asynchronous programming in .NET in general and how to apply it in Windows Store Apps specifically:

"Asynchronous Programming with Async and Await (C# and Visual Basic)" docs on MSDN.
"Asynchronous programming (Windows Store apps)" docs in Windows Dev Center.
"Async in 4.5: Worth the Await" by Alok Shriram.
"Async in 4.5: Enabling Progress and Cancellation in Async APIs" by Alok Shriram.
"Keeping apps fast and fluid with asynchrony in the Windows Runtime" by Jason Olson.
"Diving deep with WinRT and await" by Steve Toub.
"Exposing .NET tasks as WinRT asynchronous operations" by Steve Toub.
Parallel Programming with .NET team blog.

Enjoy!

Building Windows Store Apps with .NET

Brandon Bray — Tue, 20 Nov 2012 21:12:35 GMT

For .NET developers, this is the best time to build client apps. Never before have the Windows APIs been so easy to use from C# or Visual Basic without wrapping them with custom libraries. So far, we’ve seen some amazing apps in the Windows Store built using C#. Richard Lander, program manager for the CLR and frequent contributor to this blog, provides the highlights and references for building your Windows Store apps with .NET. -- Brandon

The Windows Store is a new opportunity

Windows 8 redefines apps on Windows, improving both the experiences of creating apps and, most importantly, in using them. The Windows Store provides a new opportunity for you to distribute and sell the apps that you create. Millions of end-users around the world will be looking for apps in a single place, in the Windows Store. Your app might be just the one that they are looking for. That’s a new and compelling proposition.

As a .NET developer, you’ll find that there are multiple new development options available for you to build compelling Windows Store Apps using .NET. .NET code can be used for user interface, business logic and background processing. You can continue to use .NET on the server too, as the implementation of REST, WCF or ODATA APIs that you call from within your Windows Store Apps. For all apps, .NET is a great choice for server logic.

Your app goes here.

Getting started writing .NET apps

The best way to get started is to download the tools, including Visual Studio 2012 and Blend for Visual Studio 2012. You will find several templates for .NET apps. In less than an hour, you can download and install everything you need, try out the templates and actually build the beginnings of your first real Windows Store app.

Visual Studio Express for Windows 8 is free to use and you can build your entire app with it alone. You just need Windows 8 and an internet connection to download the tools. From there, you are on your way. If you don’t have Windows 8, you can install a free evaluation copy for developers. If you already have Windows 8 and Visual Studio 2012, then you don’t need anything more.

I have been using the Windows SDK samples a lot. There are samples for most of the UI patterns and technologies that you will use to build Windows Store Apps. For example, there are samples dedicated to snap, semantic zoom, Lock screen presence, XAML data binding and file access. These samples are a great way to quickly learn how to add something to your app or can provide exactly the insight you need to get something to work.

MSDN documentation has all of the reference topics you need to learn more about APIs and new concepts. The documentation is split into Windows.* WinRT APIs and the System.* .NET for Windows Store APIs. You can also browse the .Net Framework APIs within the .NET Framework 4.5 class library. That documentation clearly displays which APIs are available for Windows Store apps, with a small green shopping bag. You may also want to read the post that we published on the .NET APIs for Windows Store Apps earlier in the year, on this blog.

There is also another blog dedicated to discussing Windows Store app development. You will want to consult it, as it goes into a lot more detail about Windows Store apps, including apps written with .NET.

Windows 8 adds the Windows Runtime

Before you get too far into building Windows Store Apps, you will benefit from a basic understanding of the Windows Runtime (WinRT). The WinRT is a new subsystem within Windows that forms much of the foundation of Windows Store apps. Most of the new Windows APIs that you will use are exposed via the WinRT. The .NET APIs are not exposed via WinRT, but continue to work as they always have, exposed by the CLR.

.NET developers are not strangers to interop technologies. You can use both COM Interop and P/invoke to call into native APIs from .NET code. The Windows Runtime is a big step ahead for .NET developers, resolving challenges with those existing interop technologies. It establishes a level playing field across languages, enabling C#, Visual Basic, C++ and JavaScript to depend on the same set of APIs in the same way.

All languages use the same metadata format for describing WinRT APIs. In fact, this metadata form is CLR metadata, with a small set of tweaks. As a result, it was very easy for Visual Studio to create an IntelliSense experience for WinRT APIs that is similar to the .NET Framework and for the .NET compilers to reference WinRT metdata. You can open WinRT metadata files, called WinMDs, in ILDasm and Reflector! In addition to being able to call WinRT APIs, you can use C# and VB to create Windows Runtime Components, which expose WinRT APIs.

Shawn Farkas, from the Common Language Runtime team, published a very useful and technical MSDN Magazine article on Windows Runtime interop with managed code, called Underneath the Hood with .NET and the Windows Runtime. It will answer many of the lower-level questions that you might have about how managed code can WinRT interoperate.

Resource: NET Framework Support for Windows Store Apps and Windows Runtime

Writing Windows Store XAML Apps

Many of you will see a natural affinity between XAML and .NET. You may have experience building Silverlight and WPF apps, which also use XAML. If you are familiar with XAML, then Windows Store XAML apps will be a natural fit. You will find that you are immediately comfortable with and able to build Windows Store XAML apps. The code-behind model is the same and the XAML syntax is largely what you’ve used before. You can continue to use your preferred design pattern, such as MVVM or MVC. You can read about the differences in the XAML porting guide

Visual Studio 2012 makes it easy to build XAML apps, using a new designer for XAML apps. You can also use Blend for Visual Studio 2012 to create visually compelling apps. The process of creating XAML apps is very similar in practice to what you are used to. The main difference is that all of the XAML APIs will appear in Windows.UI.XAML instead of System.Windows.

Resource: XAML overview (Windows Store apps using C#/VB/C++ and XAML)

Resource: Quickstart: Create a UI with XAML (Windows Store apps using C#/VB/C++ and XAML)

XAML App Windows 8 SDK Sample

I’ve included a screenshot of a XAML app from the Windows 8 SDK, called XAML essential controls sample. This sample demonstrates the basic set of XAML controls you have available. The screenshot demonstrates the progress bars available within XAML apps. You can look at the many other XAML samples available in the Windows 8 SDK, in C# and VB.

Updated XAML App Windows 8 SDK Sample

The sample above is largely composed of XAML, without code affecting the presentation of the controls. I’ve made a small update to the sample to show you how you can write C# code that modifies the controls that you see above. All of the XAML controls that are built-in to the platform have been built with C++ and are exposed by WinRT for both .NET and C++ callers. That means, the C# code that you will see below needs to call through WinRT in order to update the controls. You will quickly notice that while the code calls into WinRT, that it looks like .NET code, and is very similar to the code that you would write for the other XAML platforms (Silverlight, WPF) that expose a managed code surface area.

I added a button to the XAML (“Update Progress”, below), which is intended to update the state of the determinate progress bar, otherwise set to a constant 30%. Within the Visual Studio 2012 XAML designer, I double-clicked on the button to create an event handler for the click event for the button. Last, I wrote this small bit of code to update the progress bar with each button click. All of the code you see below is .NET code, however, the two calls to determinateProgress.Value are interop calls via WinRT. The WinRT calls blend in very cleanly with the rest of the code that you write. The objects in the method signature are also WinRT objects, however, I am not using them in this sample.

private void Button_Click_1(object sender, RoutedEventArgs e)
{
    var currentProgress = determinateProgress.Value;
    
    currentProgress += 10;
    if (currentProgress > 90)
        currentProgress = 10;

    determinateProgress.Value = currentProgress;
}

You can see the updated SDK sample below, with the addition of my button. Notice that the progress bar is in a different place. I clicked it a few times before I took this second screenshot.

Writing Windows Store DirectX Games -- SharpDX

DirectX games written in C# are another popular option that we are seeing, including games built by Microsoft. The Windows 8 SDK does not include a DirectX API for .NET developers, but only for C++. Fortunately for you, the open source SharpDX DirectX SDK is available to .NET developers to write Windows Store apps. It should also be noted that you can create a XAML app with .NET that hosts DirectX component s written in C++.

In addition, Unity has announced that they will support their Unity3D game engine for .NET developers, for Windows Store apps and Windows Phone 8. This release is still pending and doesn’t have a specific release date.

SharpDX is a low-level set of managed wrappers over the native DirectX API. These wrappers are tool-generated directly from DirectX header files, which enables them to be complete and also results in them including just the minimal interop code necessary to call the native APIs. It supports all of the DirectX API, including 2D, 3D, Sound and Input. You can get the SharpDX SDK from the SharpDX site. An update to SharpDX 2.4 was recently released, and also supports Windows Phone.

The following screenshot demonstrates a sample SharpDX app, which is a spinning cube.

The following screenshot shows some of the code that displays the spinning cube above, in Visual Studio 2012.

Closing

The Windows Store creates an important new opportunity for you, with Windows 8. More than 1,000 PCs and tablets, including Microsoft Surface, are available for consumers and businesses to buy. I expect that the Windows Store app within Windows 8 will be a frequent destination for users, wanting to find apps that meet their needs and interests. That app might be yours. The proposition to developers is a unique one, given that there are over 1 billion Windows users worldwide. Many of these users will approach the Windows Store with fresh enthusiasm, having never experienced an app store before. Your app might become part of their daily routine and part of their lives.

You can create .NET apps with XAML UI. For games, you may choose one of the DirectX SDKs that 3^rd party groups have already made available. It really is a great time to be a .NET developer. The Windows Store provides great ways to build your app with the .NET technologies that you’ve already learned. Visual Studio 2012 and the .NET Framework 4.5 are available to help you build .NET Windows Store apps. The Windows Store is open to developers in 120 countries around the world. Users and businesses will start buying Windows 8 in large numbers. The only thing left is your app.

Once again, please do check out the Windows 8 app developer blog to learn more about Windows Store development.