.NET Blog

Introducing the .NET Framework 4.5 RC

Brandon Bray — Thu, 31 May 2012 18:48:00 GMT

Today, we are announcing the .NET Framework 4.5 RC. We are also announcing Visual Studio 2012 RC, as you can read on Jason Zander’s and Soma’s blog. Please visit the Visual Studio 2012 RC downloads page to install both products.

We have made many improvements since we released the .NET Framework 4.5 Beta in February. Let’s talk about the changes at a high level:

Windows Metro style app performance: We significantly improved the performance of apps that you build using the .NET APIs for Metro style apps across many dimensions since Beta. In particular, we reduced startup time substantially, with improvements observed in the 10% range. We updated the garbage collector to aggressively reclaim memory upon app suspension, which benefits both the end-user experience and the app (avoids termination). We also improved file I/O performance when using the .NET Framework stream adapters by adding automatic buffering of the underlying Windows Runtime streams. Separately, we improved the algorithm by which Metro style apps are optimized for performance, via automatic native image generation.

Metro style app programming experience for C# and Visual Basic: We made a large number of improvements and also made adjustments to align the .NET Framework with changes in the Windows Runtime API. Examples include better support for IRandomAccessStream from the .NET Framework. We also introduced .NET Framework performance testing in the Windows App Certification Kit.
ASP.NET: Many updates, including: better support for async, cancellation, and threading; support for Entity Framework enumerations and spatial data types in Dynamic Data; support for unit testing; support for Entity Framework 5.0 RC; and improvements for extending the Web Forms compilation system. We also updated the ASP.NET Web API to use Json.Net for JSON formatting, added support for creating custom help pages, improved integration with IoC containers, and provided a monitoring and tracing infrastructure.
WCF: Many improvements, including: better performance, reliability and scale with WebSockets; support for client-side validation of the server SSL certificate using WCF’s custom X509 certificate validator on a per-request basis , and; support for adding simple WCF service references inside a managed Windows metadata (.WinMD) file, allowing JavaScript apps to call WCF services via the Windows Runtime.

Broad range of product improvements: Many areas of the product benefited from small but important improvements addressing app reliability, compatibility, and performance. We also responded to over 200 Connect feedback bugs and fixed many of the reported issues.

Take a look at What’s New in the .NET Framework 4.5 RC to learn more about the new features. We also made significant updates to our performance guidance, in the Performance best practices for C# and Visual Basic topic. Also see Scott Hanselman’s blog for more information on ASP.NET. Once again, please do download the .NET Framework 4.5 RC release.

As always, we would like to hear from you. Please don’t hesitate to post a comment on the blog or at one of the forums that we monitor: Connect (report bugs), UserVoice (request features), and MSDN Forums (ask for help).

.NET for Metro style apps

Brandon Bray — Tue, 17 Apr 2012 21:56:37 GMT

.NET is now a core part of several Microsoft platforms, and each has focused on specific subset of APIs. A lot of thought has been put into crafting each API surface area. Many of you have asked how the .NET APIs available for Metro style apps were chosen. In the following post, Immo Landwerth – a program manager on the CLR’s Core Framework team – provides an answer to this question. -- Brandon

Since the releases of the Windows 8 Consumer Preview and the Windows Developer Preview, developers have been busy exploring Windows 8, and many have asked questions about the subset of.NET Framework APIs that can be used to develop Windows Metro style apps. The natural tendency has been to compare the .NET APIs for Metro style apps to those available for other platforms such as Windows Phone. In particular, developers are wondering how much of their existing C# or Visual Basic source code they can expect to reasonably port to build a new Metro style app. We asked ourselves the same question when we carefully designed .NET APIs for Metro style apps.

In this post, I’d like to give you an overview of the APIs that are available to you for building Metro style apps. I will primarily discuss the design principles and requirements we used to decide which .NET Framework APIs to make available for Metro style app development. I will also touch on the relationships between .NET APIs for Metro style apps and other .NET API profiles.

Design goals

From the start of this project, we realized that we had a great opportunity to simplify app development, but also some interesting challenges to resolve. Metro style apps enable developers to provide compelling end-user experiences across all the supported Windows 8 devices. In order to achieve this goal, many important changes were made within Windows, some of which affected our design approach to the .NET Framework APIs that we would expose.

The biggest change, in terms of .NET Framework APIs, was the introduction of the Windows Runtime APIs. The Windows Runtime is a new API surface in Windows that exposes the functionality necessary to write Metro style apps. These APIs were designed to be used from a variety of programming languages: C# and Visual Basic and also native C++ and JavaScript. The task for our design team was to ensure that .NET Framework developers would be able to use both .NET Framework and Windows Runtime APIs together in a way that felt natural and intuitive. Also, the Windows Runtime team worked closely with us and approached the same task from the opposite direction.

We established the following goals to define the API surface of .NET for Metro style apps:

Avoid duplication of functionality between the Windows Runtime and the .NET Framework.
Provide a clear focus on .NET Framework APIs that are needed to write Metro style apps.
Make sure that existing .NET Framework developers will feel at home with this profile.
Make it easy to port existing C# and Visual Basic code to the profile.

We also looked at a collection of Windows Phone and Silverlight apps to determine how developers used .NET Framework APIs in practice. These apps helped us assess the two last goals, in particular.

Designing the API profile

To design a new profile for Metro style apps, it made sense to start from an existing subset of APIs, and to pare it down while extending it toward the goal, which was to provide simple and well-designed .NET Framework APIs that specifically target Metro style app development. Note that the following diagram is intended for illustrative purposes only, and is not drawn to scale.

Figure 1: .NET Framework profiles in context

In this diagram, the smaller circles represent .NET profiles. The intersection of the profiles represents APIs that are common to all the profiles. Conceptually, you can think of .NET for Metro style apps as a subset of the .NET Framework that shares a relationship with other .NET Framework profiles. Portable Class Library does not show up in this diagram as a separate profile, but is a separate concept that deserves a post of its own.

With that diagram in mind, we decided to start with the .NET API subset for the Windows Phone, with selective additions from the full .NET Framework and Silverlight. While Windows Phone apps were likely to be similar to Metro style apps, the introduction of the Windows Runtime in Windows 8 meant that the Windows Phone profile was only a starting point.

API selection process

In addition to the goals discussed earlier, we established a rigorous selection process. We made sure that every API we considered adding passed the criteria defined below:

Is the API applicable to Metro style apps? Of course, the answer isn't always obvious. For example, ASP.NET APIs are clearly not relevant to Metro style app development, but console APIs could be considered helpful (for example, during testing). When in doubt, we asked ourselves whether we’d want to ship a Metro style app that called the given API, and we removed APIs that simply won’t work in Metro style apps. For example, file access using paths isn't supported in Metro style apps, because file access is done through a broker process and requires using Windows Runtime APIs, so we removed those APIs.
Is the API obsolete or outdated? This includes APIs that are difficult to use correctly, are confusing, or don’t follow basic design guidelines.
Is the API duplicated by another .NET Framework API in the profile or by a Windows Runtime API? Duplication of APIs means that you have to choose, and this choice is often arbitrary. Duplication also means that you cannot easily share code with other developers, because they might have chosen to use different types in their signatures.

Arriving at the final set of APIs

After our design phase and a progressive series of refinements, we arrived at a set of APIs that met the stated design goals. We validated this set of APIs with the apps that we had available, and requested feedback from Microsoft developers who were involved in early Metro style app development. We made more changes to enable the set of scenarios that the validation exercise uncovered. With the exception of a few more minor changes, this is the surface area we made available for Metro style app development with Visual Studio 11 Developer Preview, and more recently, with Visual Studio 11 Beta.

The diagram below shows the functionality exposed in the .NET for Metro style apps profile. The diagram should match what you see available in Visual Studio 11 Beta. The Windows Dev Center is a great place to explore these new APIs.

Figure 2: Feature areas supported by .NET for Metro style apps

For a more quantitative view, the following table compares the new profile to existing profiles you might be familiar with. It's not surprising that .NET for Metro style apps is much smaller than the full .NET Framework, and similar in size to the profile for Windows Phone. .NET for Metro style apps is even smaller than the Windows Phone profile, because we removed functionality such as the UI stack and sensors, which is exposed by the Windows Runtime.

	.NET for Metro style apps	Windows Phone 7.1	.NET Framework 4.5
Namespace	72	95	447
Type	1,246	1,788	14,936
Member	15,674	20,291	217,166

Table: API surface counts, by .NET Profile

Learn more

If you want to learn more about the thinking process that went into the design of the API surface area, watch Krzysztof Cwalina’s A .NET developer's view of Windows 8 app development talk from the BUILD conference.

If you’re interested in porting your existing .NET Framework code, watch Daniel Plaisted’s BUILD talk Bringing existing managed code into Metro style apps, and check out the .NET for Metro style apps overview in the Windows Dev Center, especially the section about converting your existing .NET Framework code.

Have you used the .NET for Metro style apps and the Windows Runtime APIs to build Metro style apps? Which APIs do you think are missing? Are there any additional APIs that should have been removed? Please tell us what you think.

Async in 4.5: Worth the Await

Brandon Bray — Tue, 03 Apr 2012 17:17:23 GMT

Developers often ask for guidance on how to write responsive user interfaces. Reading most books about the .NET Framework over the past ten years, you’ll see coverage of the asynchronous programming model which requires a lot of careful attention while coding. That’s why the async features in the latest versions of C# and Visual Basic are tremendous step forward. And language innovation like async needs a great library to bring forward the potential. In this post, Alok Shriram – a Program Manager from the .NET Base Class Library team – shows the work done in the .NET Framework to be async ready.

"Performance" is a term that is used a lot when talking about apps, but it's actually a pretty vague term. There are at least two aspects of performance that most people think about: app launch time and throughput. Both of these can be measured and described with actual numbers. The true test of an app, however, is end-user perception. A stop-watch may tell us one thing, but a user may see something else. End-user perception is the basis of asynchrony and related features that we have built into the .NET Framework 4.5 Beta. Essentially, we can provide a more responsive UI experience if some of the more expensive app operations can be delayed. This will make the end-user happier, and it will help us make better use of computing resources than would be possible otherwise. While you’ve always been able to implement this pattern in the .NET Framework, the implementation has been difficult in practice. We’ve fixed that in the .NET Framework 4.5 Beta.

This post introduces the asynchronous features in the .NET Framework 4.5 Beta. Many of you have been introduced to “async” via the language changes in C#, but we also want to tell you about accompanying changes that we’ve made in the .NET Framework.

Asynchrony in the .NET Framework

If you have ever written any UI code, chances are that you have used the asynchronous capabilities of the .NET Framework. Asynchrony keeps the UI responsive while expensive operations are being performed. For instance, if you implement a mouse click event that has to go and fetch a large page from the Internet, and you implement it using the WebClient.DownloadString method, depending on the amount of time this call takes to return, the thread on which the UI is running will be blocked. This means that any subsequent actions performed on the UI will be delayed, and you will wind up getting a frozen UI. However, if you use an asynchronous version of the method instead (specifically, WebClient.DownloadStringAsync), the request will return control immediately, and subsequent UI interactions can be handled as they arrive, while the content is being asynchronously fetched from the resource.

In server apps, resource requests like database calls and networks calls are typically written in a synchronous manner, so a large number of threads are blocked on I/O operations. This approach can lead to a larger memory footprint for the app and poor utilization of the server hardware.

Writing code that utilizes asynchronous paradigms is useful for both client and server apps. However, despite all the benefits, asynchronous code is hard to write, debug, and maintain. The new language features introduced by the .NET Framework 4.5 make writing asynchronous code comparable to writing synchronous code.

The new async/await keywords

Before the .NET Framework 4.5 Beta, asynchronous APIs in the .NET Framework followed three patterns:

The Asynchronous Programming Model (APM), which has the format BeginMethodName and EndMethodName.
The Event based Asynchronous Pattern (EAP), which relies on assigning delegates to event handlers that will be invoked when an event is triggered.
The Task-based Asynchronous Pattern (TAP), which relies on the Task Parallel Library (TPL) and the System.Threading.Task namespace that was introduced with the .NET Framework 4.

I won't get into the pros and cons of the different schemes, since there are plenty of articles covering that. (A great starting point is Stephen Toub’s TAP doc.) However, to illustrate the differences in complexity, let’s consider a synchronous implementation of CopyTo( ) and the code that we would have to write in the TAP and APM models to implement a CopyToAsync() function.

Synchronous example:

public static void CopyTo(Stream source, Stream destination)
{
  byte[] buffer = new byte[0x1000]; 
  int numRead;
  while((numRead = source.Read(buffer, 0, buffer.Length)) > 0)
  {
    destination.Write(buffer, 0, numRead);
  }
}

APM example:

public static IAsyncResult BeginCopyTo(Stream source, Stream destination)
{
  var tcs = new TaskCompletionSource();

  byte[] buffer = new byte[0x1000];
  Action<IAsyncResult> readWriteLoop = null;
  readWriteLoop = iar =>
  {
    try
    {
      for (bool isRead = iar == null; ; isRead = !isRead)
      {
        switch (isRead)
        {
          case true:
            iar = source.BeginRead(buffer, 0, buffer.Length, readResult =>
            {
              if (readResult.CompletedSynchronously) return;
              readWriteLoop(readResult);
            }, null);
            if (!iar.CompletedSynchronously) return;
            break;

          case false:
            int numRead = source.EndRead(iar);
            if (numRead == 0)
            {
              tcs.TrySetResult(true);
              return;
            }
            iar = destination.BeginWrite(buffer, 0, numRead, writeResult =>
            {
              try
              {
                if (writeResult.CompletedSynchronously) return;
                destination.EndWrite(writeResult);
                readWriteLoop(null);
              }
              catch(Exception e) { tcs.TrySetException(e); }
            }, null);
            if (!iar.CompletedSynchronously) return;
            destination.EndWrite(iar);
            break;
          }
        }
      }
    }
    catch(Exception e) { tcs.TrySetException(e); }
  };
  readWriteLoop(null);

  return tcs.Task;
}

public static void EndCopyTo(IAsyncResult asyncResult)
{
  ((Task)asyncResult).Wait();
}

TAP example with the new C# language async support:

public static async void CopyToAsync(Stream source, Stream destination)
{
  byte[] buffer = new byte[0x1000];
  int numRead;
  while((numRead = await source.ReadAsync(buffer, 0, buffer.Length)) > 0)
  {
    await destination.WriteAsync(buffer, 0, numRead);
  }
}

It’s pretty clear that there is a huge difference in the complexity of the code that you have to write with the APM, as compared to either of the other two examples. You may have written asynchronous code before, but you probably didn't find it straightforward. Ideally, the C# and Visual Basic languages (and compilers) should be doing all of the heavy lifting, and you should only need to declare your intent via a set of new keywords.

There are two new language keywords introduced in C# and Visual Basic: async and await (highlighted in yellow in the code above). There are also a significant number of new .NET Framework methods that support asynchronous apps. For example, the .NET Framework 4 Stream class has a CopyTo method. In the .NET Framework 4.5 Beta, we added a CopyToAsync method, which (not coincidentally) shares an implementation not unlike the TAP example above.

Starting with synchronous code

Most of you will likely end up using async with synchronous code, as opposed to moving code that is already asynchronous to this new pattern. So we’ll start with synchronous code, too.

This post covers the experience of developing a simple app that gets a list of movies from the Netflix OData catalog and displays the box art and movie title/summary. We will look at the initial, synchronous implementation and explain what we did to convert it to a responsive async app (it wasn't much).

First, let's start off by declaring a simple class called Movie to hold some details about each movie. Let's also add a single utility method and use it to parse an HTTP response into a Movie object:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Net.Http;
using System.Threading.Tasks;
using System.Xml.Linq;

public class Movie
{
  public string Title
  { get; set; }
  public string BoxArtUrl
  { get; set; }
  public string Summary
  { get; set; }
}
    
public class MainPage
{
  private IEnumerable<Movie> GetMovieDataFromXDoc(XDocument doc)
  {
    XNamespace dRss = "http://www.w3.org/2005/Atom";
    XNamespace odata = "http://schemas.microsoft.com/ado/2007/08/dataservices";

    List<XElement> dox = doc.Descendants(dRss + "entry").ToList();
    List<Movie> movieList = new List<Movie>();
    foreach (XElement ele in dox)
    {
      XElement movieSummary = ele.Descendants(dRss + "summary").First();
      XElement movieName = ele.Descendants(odata + "Name").First();
      XElement boxArtUrl = ele.Descendants(odata + "MediumUrl").First();
      Movie tempMovie = new Movie() { 
                              BoxArtUrl = boxArtUrl.Value, 
                              Title = movieName.Value, 
                              Summary = movieSummary.Value };
      movieList.Add(tempMovie);
    }
    return movieList;
  }
}

Adding async support

The new async support is built on the Task Parallel Library (TPL) that was introduced in the .NET Framework 4. It uses the Task framework, and adds language support through the async and await keywords to make writing asynchronous code almost equivalent to writing synchronous code.

You use the async keyword to annotate that this method could be invoked asynchronously. You then apply the await keyword to a task inside an async method to suspend the execution of the method until that task is complete. Let’s explore these concepts by starting with some synchronous code in the MainPage class:

public List<Movie> GetMovies()
{
  string query = 
    @"http://odata.netflix.com/Catalog/Titles?" +
    @"$filter=ReleaseYear le 1989 and ReleaseYear ge 1980 " +
    @"and AverageRating gt 4";
  HttpClient client = new HttpClient();
  client.MaxResponseContentBufferSize = Int32.MaxValue;
  try
  {
    HttpResponseMessage resp = client.Get(new Uri(query));
    string content = resp.Content.ReadAsString();
    XDocument xml = XDocument.Parse(content);
    List<Movie> movieList = GetMovieDataFromXDoc(xml).ToList();
    return movieList;
  }
  catch (Do your catch logic here)
  {
    ... and here 
  }
}

The code above constructs an HTTP query, sets the buffer size to a large value, and does a synchronous get to retrieve the contents of the target. It queries the Netflix OData catalog for all of the movies that were released in 1989 and that have a Netflix rating greater than four stars. We wrap it in a try/catch block to make sure we handle any exceptions, and we make a blocking call to the network with the client.Get(new Uri(query)) call. If this call were made from a UI app and the returned page was large or network latency was high or server processing time was long, then your UI would likely freeze while waiting for the query to return. We’d much prefer to make the code run the same way, but not freeze up the UI, right? To show you how to achieve that, I’ve updated the code for async, and grayed out the code that hasn't changed (most of it).

public async Task<List<Movie>> GetMoviesAsync()
{
  string query = 
    @"http://odata.netflix.com/Catalog/Titles?" +
    @"$filter=ReleaseYear le 1989 and ReleaseYear ge 1980 " +
    @"and AverageRating gt 4";
  HttpClient client = new HttpClient();
  client.MaxResponseContentBufferSize=Int32.MaxValue;
  try
  {
    HttpResponseMessage resp = await client.GetAsync(new Uri(query));
    string content = resp.Content.ReadAsString();
    List<Movie> movieList = GetMovieDataFromXDoc(xml).ToList();
    return movieList;
  }
  catch (Do your catch logic here)
  {
    ... and here
  }
}

As you can see, the changes that we had to make are minor. First we use the async keyword to mark our method so that we’re allowed to use the await keyword in it. We also change the return type to Task, so that callers of our method will have a handle through which to wait for our asynchronous operation’s completion. Then, we use the GetAsync method instead of the Get method on the HttpClient class. This returns a Task<HttpResponseMessage> (instead of the HttpResponseMessage that Get returns), which we await.

Finally, the pattern specifies that any method that uses an await internally should be annotated with an Async suffix, so we rename our method to GetMoviesAsync. That’s it! Now, when this method is called in the UI, it will not lock up the UI, because it will return immediately. The UI will be populated when the results are obtained. Since the UI is still interactive, you should make sure that any user interactions that might require reading or writing data (a list of movies in this example) are prepared for the case where the data is not yet populated because it is waiting on an async call.

How async works

Before we move on, it's probably a good idea to understand what is happening in terms of code flow when we encounter an await. Let's take a look at our asynchronous code again.

public async Task<List<Movie>> GetMoviesAsync()
{
  string query =  "...";
    ...
  try
  {
    HttpResponseMessage resp = await client.GetAsync(new Uri(query));
    string content = resp.Content.ReadAsString();
    List<Movie> movieList = GetMovieDataFromXDoc(xml).ToList();
    return movieList;
  }
  catch (Do your catch logic here)
  {
    ... and here
  }
}

In terms of control flow, what happens here is that when the compiler parses an await keyword, it wraps all the code following the await into a continuation function, which is invoked when the call to GetAsync() returns a result. This essentially means that when GetMoviesAsync() is called, control is going to flow until the await client.GetAsync(new Uri(query)) method. None of the code following this method is going to be executed until this function has returned. So, although it might appear that the GetAsync is a non-blocking call, it’s actually doing a “yield return” to the calling function. For a deeper look at the mechanics of how async works, see the resources provided at the end of this post.

Propagation of the async keyword through the call stack

Once you start using async, you’ll quickly find that it isn’t intended to be used in just a single function, but for an entire flow in your application. You’ll see that your use of await and async keywords will propagate all of the way up to the top of a logical call stack. Let’s take a look.

We defined the following async method in the earlier example:

public async Task<List<Movie>> GetMoviesAsync() {...}

This tells us that we will have to do an await on any call to this method, such as in the example below:

await viewer.GetMoviesAsync();

This essentially means that the method that contains the GetMoviesAsync() call will have to be marked as an async, which leads to the async keyword being propagated through the call stack. For a UI app, this async propagation would likely end at an event handler. When you think about it, that’s exactly what needs to happen in order to get the benefit of asynchrony. The language forces this behavior, leading you to do the right thing.

async void SearchButton_Click(object sender, RoutedEventArgs e)
{
  ...
  await viewer.GetMoviesAsync();
  ...
}

This propagation is normal and just annotates the methods that are potentially awaitable. The propagation of the keywords and the return type change (on each method signature) are really the bulk of the work to migrate your code from the sync to async models. That’s pretty reasonable!

Another point to keep in mind is that the async keyword propagation will stop once you have a void return on a method. A void return async function is essentially a “fire and forget” async call. This is why, for example, you can mark an event handler delegate with the async keyword in a UI app and not worry about doing anything in the app logic that invokes the event handler.

Async APIs in .NET Framework 4.5 Beta

You are probably wondering where to first start using async in your app. An important consideration is the set of APIs that you call within your apps. First and foremost among those will be the .NET Framework APIs. In the .NET Framework 4.5 Beta, we included a significant set of new async variants of existing synchronous APIs. You will want to move to using these new APIs when you move to async for your own code.

The following is the set of async APIs that we added just for the Stream class, to give you a sense of what we’ve done in the .NET Framework to support async. You’ll find this same pattern on many other types. (You’ll be able to recognize the async APIs by the “Async” in their names.)

System.IO.Stream

public Task<Int32> ReadAsync(Byte[] buffer, Int32 offset, Int32 count);
public virtual Task<Int32> ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken);
public Task WriteAsync(Byte[] buffer, Int32 offset, Int32 count);
public virtual Task WriteAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken);
public Task FlushAsync();
public virtual Task FlushAsync(CancellationToken cancellationToken);
public Task CopyToAsync(Stream destination);
public Task CopyToAsync(Stream destination, Int32 bufferSize);
public virtual Task CopyToAsync(Stream destination, Int32 bufferSize, CancellationToken cancellationToken);

Summary

Async is an important new feature in the .NET Framework 4.5 Beta. The new async and await keywords enable you to provide significant user experience improvements in your apps without much effort on your part. In addition, we've added a large set of new async APIs within the .NET Framework in the areas where they make the most sense. We see the combination of the language features and the new .NET Framework APIs as completing the async scenario, leaving the rest to you.

The Visual Studio Asynchronous Programming site on MSDN is a great resource for samples, white papers and talks on the new language features and support. You can also learn more about the Task-based Asynchronous Pattern by reading Stephen Toub’s document on the topic. You might also want to check out Asynchronous File I/O and Asynchronous Programming with Async and Await in the MSDN Library, and Asynchronous programming in the Windows Dev Center.

Have you used async? Was it as easy to migrate your synchronous code to async as this post suggested? Are there any other .NET Framework APIs that you think need async variants? Please tell us what you think.

Cheers
-Alok

Improving Launch Performance for Your Desktop Applications

Brandon Bray — Tue, 20 Mar 2012 22:55:00 GMT

Application performance is something we hear about all the time. It almost always falls into one of the top three issues when we aggregate all of the feedback channels from .NET developers. While performance has many characteristics, application startup time is something that everyone can easily relate to. With the .NET Framework 4.5 Beta and the “Visual Studio 11” Beta we’re now sharing with you some of the same techniques we’ve successfully used to make the .NET Framework faster. The following post was written by Ashwin Kamath, a program manager on the CLR performance team. --Brandon

With every release of the .NET Framework, the .NET Framework product team looks for new ways to improve the launch performance of desktop applications. We often find opportunities for the .NET Framework to do less work on startup to achieve small performance improvements, but we also look for broader changes. In the .NET Framework 2.0, we created a new technology that significantly decreases the time it takes to read and execute precompiled native images. We have been using this technology on .NET Framework assemblies since, and it has resulted in significant improvements in application launch performance (startup times) for all managed applications. We call this technology managed profile guided optimization (MPGO). In Visual Studio 11 Ultimate Beta, MPGO is available for you to use with your application code, so you can achieve faster application startup times.

In this blog post, I’ll discuss this new performance technology and how you can use it with managed desktop applications.

.NET Framework Code Compilation Background

.NET Framework assemblies contain intermediate language (IL) byte-code that needs to be compiled to machine code in order to run on a given processor. The .NET Framework provides two options for code compilation: just-in-time (JIT) compilation and precompilation. With JIT compilation, IL code is compiled at run time prior to execution. This approach is very convenient since it happens automatically and only compiles the subset of your application that is actually needed or executed. You can think of precompilation as ahead-of-time compilation. It requires the use of the Native Image Generator (Ngen.exe) on the same machine as the application in order to precompile a set of assemblies. All of the .NET Framework assemblies (for example, System.dll) are typically precompiled.

Precompiled native images typically result in better application launch performance because JIT compilation is no longer necessary. There are also other important benefits, such as a lower working set (memory usage). We recommend that you create precompiled native images for scenarios where application launch performance is critical, typically for larger desktop applications.

The generation of precompiled native images follows the workflow illustrated in the figure below. Invoking the NGen tool to generate native images must occur on the end-user machine (denoted by the blue box). Native images are not included with your application setup package – the application is shipped to end users as IL, and the NGen tool invoked as part of the application setup on the end-user machine to create native images. For more details on NGen, you may want to refer to the MSDN article on NGen.

Figure 1: Workflow for precompiled native image generation

Even Faster Precompiled Native Images

The .NET Framework has been using an innovative technology to further improve the performance of precompiled .NET Framework native images for years. Most managed applications already enjoy the benefits of this technology, by virtue of the .NET Framework using it. For larger applications more performance wins may be gained by adopting this technology, and for these applications, we recommend its use.

We call this technology managed profile guided optimization (MPGO). It can improve the startup and working set (memory usage) of managed applications by optimizing the layout of precompiled native images. In particular, MPGO co-locates frequently used image data within a native image, such that each request for a disk page of image data has a higher density of useful image data for the running program.

As a result, the running program will make fewer disk page requests -- often referred to as “page faults” -- which will result in improved performance. MPGO provides its greatest benefit on machines with rotational hard disks; however, it is still beneficial to use on customer machines with solid state disks (SSDs).

MPGO is centered on a process called “training.” You use the MPGO tool to launch your application and collect training data. During training, you exercise a variety of representative user scenarios within your application. It is important to not exercise every possible scenario that your application offers, but the most typical ones for your users. The benefit of MPGO will be based on the quality of this training process. So, you should experiment with this process until you determine the set of training scenarios that result in the best performance for your customers.

Training data is stored in a profile (hence the name of the technology), and the profile is stored as a resource within each IL assembly that is trained. This profile is then used by the NGen tool during compilation of each IL assembly, to optimize the layout of precompiled native images that it generates. MPGO can be thought of as providing extra information to NGen in order to generate higher-quality native images.

The generation of training data and precompiled native images follows the workflow illustrated in the figure below. As in Figure 1, red boxes represent artifacts created on the developer machine and blue boxes represent artifacts generated on end-user machines.

Figure 2: Workflow for optimized precompiled native image generation

Developer Guidance

As already stated, both NGen and MPGO are recommended for use with larger desktop applications, because the benefit of precompiled native images is typically seen only when it replaces significant JIT compilation at run time. In addition, these technologies are not recommended for ASP.NET and WCF services, given the dynamic deployment models available for those application types, and because startup is typically less important in those scenarios.

Assuming that you do have a large desktop application that is a candidate for MPGO, we recommend that you start by using NGen and measure the performance improvement. If you measure native image performance and still need additional performance improvements, you can also try using MPGO. The MPGO tool does not support Windows Metro style apps in the .NET Framework 4.5 Beta.

Your customers do not need to have the MPGO tool on their machines. They only need to have the NGen tool, which is included in the .NET Framework 4.5 Beta.

Using the MPGO Tool

The MPGO tool is included with Visual Studio 11 Ultimate Beta. The tool can be found at the following path, after you have installed Visual Studio 11 Ultimate Beta:

C:\program files(x86)\microsoft visual studio 11.0\team tools\performance tools\mpgo.exe

You can experiment with the tool by following these steps:

Obtain a machine with Visual Studio 11 Ultimate Beta and your application installed.
Run the MPGO tool (as an administrator) with the necessary parameters:
MPGO -scenario MyLargeApp.exe -AssembyList *.* -OutDir C:\Optimized\
The optimized IL assemblies are created in the C:\Optimized folder.
Run the NGen tool (as an administrator) with the necessary parameters for each application DLL:
NGEN.exe myLargeApp.exe
Run your application – it will now use the optimized native images.

In closing, the .NET Framework 4.5 enables a new innovative technology for use with desktop applications, called managed profile guided optimization. This technology is used by the .NET Framework itself to benefit all managed applications, including ASP.NET, WPF, WCF, and Metro style apps. In the case of larger desktop applications, it is worth considering both NGen and MPGO as part of your application launch performance plan.

Have you tried NGen and MPGO? Did you see a startup performance improvement with your application? Were you able to integrate these tools into your existing process? Tell us what you think in the comments and with our other feedback options.

Optimizing the .NET Framework Deployment Experience for Users and Developers

Brandon Bray — Wed, 07 Mar 2012 22:32:00 GMT

Rapid development has been one of the enduring themes behind the design of the .NET Framework. We know that authoring application installers is particularly difficult and could be improved. The following post is by Richard Lander from the program management team on the Common Language Runtime. He explains our motivations for introducing a new model for deploying the .NET Framework as seen in the Windows 8 Developer Preview and updated in the Windows 8 Consumer Preview. – Brandon

Everyone knows that it's all about the apps. I can think back to the early 1990s, when I was using both WordPerfect 5.1 and Microsoft Word 2.0c, both awesome desktop apps for the time, on Windows. Fast forward to the 2000s, and you’ll find many developers using the .NET Framework to build both desktop apps and websites and services. For many developers, the .NET Framework is the only Windows development platform that they know and love.

In this blog post, I’ll discuss how Windows 8 significantly improves the end-user experience of using .NET Framework apps, when the app depends on a .NET Framework version that isn’t a built-in component of a particular version of Windows. We will look at the integrated experience that Windows 8 provides to run apps built for the .NET Framework 3.5 and earlier versions.

User Experience on Windows 7 (and earlier versions)

While developers have created huge numbers of great apps with the .NET Framework, they don't always deploy the required .NET Framework version as part of the installation experience, and sometimes leave that step to end-users. In that case, end-users have to go out and download the .NET Framework themselves. In the early years of the .NET Framework, end-users had just one or two versions to choose from, and could manage this task. By 2012, there are quite a few .NET Framework versions that have been shipped, and knowing which version is the right or best one to install to get an app to run presents a little too much of a guessing-game for most end-users.

On Windows 7 (and earlier Windows versions), we provide an easier experience for Windows users who attempt to run .NET Framework apps but do not have the correct version of the .NET Framework installed. This experience is oriented around a simple error dialog, which takes users directly to the download page for the required .NET Framework version.

Error dialog for missing .NET Framework versions in Windows 7

All in all, this is a reasonable experience, but when you think about it in broader terms, the dialog and web-page are simply guiding the users through a manual, if well-orchestrated, .NET Framework installation process. In the planning of Windows 8, we decided that this experience wasn’t good enough for our customers.

Looking at the Numbers

The dialog above simply directs end-users to a set of web-pages, so we have access to data that gives us a partial view into which versions of the .NET Framework are actively in use, and which versions end-users most often need to (sadly) deploy themselves. The following chart provides a high-level view of the data coming in through our existing dialog.

.NET Framework versions missing on Windows XP, Windows Vista, and Windows 7

This chart is open to wild interpretation unless you have the right context. You may have noticed two major trends with the .NET Framework and Windows over the last decade:

The .NET Framework is built into Windows, starting with later Windows XP SKUs (for example, the Media Center Edition) and then in earnest with Windows Server 2003 and Windows Vista.
Each Windows version includes only one version of the .NET Framework.

Re-interpreting the chart with this information in mind, one would guess that:

A significant percentage of the .NET Framework 2.0 traffic is coming from Windows XP, which generally did not ship with a .NET Framework version, whereas both Windows Vista and Windows 7 can run .NET Framework 2.0 and 3.5 apps.
The .NET Framework 4 is a significant part of the chart above, since no released Windows version has included that version.

The chart below demonstrates the theory that nearly all the missing .NET Framework 2.0 installations are specific to Windows XP.

End-users missing .NET Framework 2.0 (or 3.5), by Windows OS version

User Experience on Windows 8

The Windows 8 Consumer Preview includes .NET Framework 4.5 Beta, and will include .NET Framework 4.5 RTM when the new OS ships. Note that the .NET Framework 4.5 can be thought of as including the .NET Framework 4, because the .NET Framework 4 does not need to be additionally installed. That leaves end-users in the position of needing to deploy the .NET Framework 3.5 onto their Windows 8 machines, to run .NET Framework 2.0, 3.0, and 3.5 desktop apps, if we continued with the Windows 7 user experience. Given the data from Windows XP and the fact that the .NET Framework 3.5 is included in Windows Vista and Windows 7, one can assume that there are a great number of .NET Framework 3.5 apps that Windows 8 customers will want to run on their PCs.

Unlike Windows 7, Windows 8 automatically downloads and installs the .NET Framework 3.5 from Windows Update. There are no links to follow, no risk of missteps by customers who are not sure which .NET Framework versions to download and install from MSDN. The whole experience requires just a few simple clicks, and then you are done.

The new experience centers around a new dialog box, which users will typically see when they attempt to install a .NET Framework 3.5 (or earlier version) app, or run the app, if it doesn’t have an installer. We released this new experience with the Windows Developer Preview at the Microsoft BUILD conference, and have updated it for the Windows 8 Consumer Preview. See the latest user experience in the image below.

The .NET Framework 3.5 deployment experience in the Windows 8 Consumer Preview build

We’ve seen significant uptake of this new experience. With the Windows Developer Preview, we saw users opt to download the .NET Framework 3.5 on ~25% of machines via the new experience.

Additionally, the .NET Framework 3.5 can be installed via the existing control panel experience.

For more information about these user experiences, see Installing the .NET Framework 3.5 on Windows 8 Consumer Preview in the MSDN Library.

In Closing

Developers have built many apps with the .NET Framework platform. On Windows 8, developers can count on the .NET Framework 4.5 being built into the operating system, and .NET Framework 3.5 being easily accessible via Windows Update. With both versions at hand, end-users will have a great experience running .NET Framework apps on Windows 8.

Do you like this experience? Do you see it as an improvement? Is there anything about it that you would change?

Update 4.0.3 for the Microsoft .NET Framework 4

Layla Driscoll — Tue, 06 Mar 2012 02:04:00 GMT

We continue to bring certain features through updates to .NET 4 without having to wait for the .NET 4.5 release. This post covers what the latest platform update has to offer. Vivek Mishra from the .NET Servicing team authored the following. – Layla

Update 4.0.3 for the Microsoft .NET Framework 4 is now available for download. This is the third platform update that offers new features on top of the initial release for the .NET Framework 4. You can learn about earlier platform updates via the following links: Update 4.0.2 and Update 4.0.1.

Update 4.0.3 introduces five new features spanning the common language runtime, Windows Workflow Foundation (WF), and SqlClient. In addition, Update 4.0.3 includes all the features released in Update 4.0.1 and 4.0.2.

This post briefly describes the features included in Update 4.0.3 and provides links to the download packages and Microsoft Knowledge Base articles, which describe the changes in detail. You can find further information about how to use this release, setup instructions, and links to documentation in the .NET Framework Developer Center.

What are the features in 4.0.3?

The following table lists all the features included in this update. Note that features from earlier platform updates are also included in 4.0.3 and remain unchanged from their initial release.

Feature	4.0.1	4.0.2	4.0.3
Support for running workflows in partial trust			✓
Windows Workflow Foundation messaging activities enabled across multiple hosts			✓
New GCLatencyMode enumeration value for garbage collection behavior			✓
Support for Correlated Connection ID in SqlClient			✓
New support in .NET Framework 4 Portable Class Libraries			✓
AlwaysOn support in SqlClient		✓	✓
SQL Server Express Local Database Runtime support in SqlClient		✓	✓
StateMachine (including designer support)	✓	✓	✓
SqlWorkflowInstanceStore on SQL Azure	✓	✓	✓
Compensation for custom hosts	✓	✓	✓

Where can I get 4.0.3?

If you want to set up your development environment for building .NET Framework 4.0.3 applications in Visual Studio 2010 Service Pack 1 (SP1), download and install the Design-time Update (KB2600214); for more information, see the Knowledge Base article on the update. You must install Visual Studio 2010 SP1 before you can use this package.
If you want to set up your development environment for building .NET Framework 4.0.3 applications in Visual Studio 11 Beta or later, download the Multi-Targeting Pack (KB2600213) standalone installer; for more information, see the Knowledge Base article on this update.
If you don't need the design-time features and just want to deploy your applications built with the .NET Framework 4.0.3, redistribute the Run-time Update (KB2600211) with your applications; for more information, see the Knowledge Base article. You must install the .NET Framework 4 or .NET Framework 4 Client Profile before installing the Run-time Update.

If you discover any issues while working with this update, please send them our way via Microsoft Connect.

Introducing .NET Framework 4.5 Beta

Layla Driscoll — Wed, 29 Feb 2012 14:29:00 GMT

In September, we released the .NET Framework 4.5 Developer Preview. The Developer Preview was essentially a “feature complete” release, enabling you to “kick the tires” and otherwise get a good sense of what we were in the process of building for you. Across the many downloads of the Developer Preview, we expect that many of you tried a broad range of developer activities with the new pre-release version of the product, such as: using the new .NET Framework 4.5 features, trying out Visual Studio 11 Developer Preview and building proof-of-concept Windows Metro style apps. We send a big shout-out to the developers who submitted an app to the Windows 8 First Apps Contest. Thank you for all of your interest and support.

As you may have seen on Jason’s blog, we’ve now taken another step forward with .NET Framework 4.5, releasing .NET Framework 4.5 Beta and Visual Studio 11 Beta.

.NET Framework 4.5 Beta

At this point, most product features in the core of the .NET Framework are in a near-final state and are ready for you to take the proof-of-concept apps that you built with the Developer Preview and transform them into actual apps that you will make available to your customers. Obviously, you can update your existing .NET Framework 4 and 3.5 apps, too.

You can download .NET Framework 4.5 Beta now.

Take a look at What’s New in the .NET Framework 4.5 Beta to learn more about the new features.

Next on the .NET Blog

Over the next while, we will tell you the in-depth stories about a number of features that are included in the beta. We know that you will want much more inside information about the features that we’ve built, and what you can expect from them.

You can expect to see posts on the following topics shortly, and other topics will follow:

.NET Framework 3.5 Feature on Demand, on Windows 8
A response to the comments about in-place update releases
[What you care about goes here – post suggestions in the comments]

Update 4.0.2 for the Microsoft .NET Framework 4

Brandon Bray — Thu, 27 Oct 2011 22:24:22 GMT

With .NET 4, we’ve focused on bringing certain features to market through updates without having to wait for the .NET 4.5 release. This post covers what the latest platform update has to offer. Vivek Mishra from the .NET Servicing team authored the following. -- Brandon

The .NET Framework team is excited to announce that Update 4.0.2 for the Microsoft .NET Framework 4 is now available. This is a platform update that offers a set of additional functionality on top of the .NET Framework 4. In this release, we have added support for AlwaysOn and LocalDB in SqlClient. These features were introduced in SQL Server Code Name "Denali." This post briefly describes what is included in the release and links to the download packages.

You can find further information about how to use this release, setup instructions, and links to documentation in the .NET Framework Developer Center.

What are the features in 4.0.2?

4.0.2 introduces new features for supporting AlwaysOn and LocalDB with SqlClient. 4.0.2 also includes the features that were released as part of 4.0.1. Here's a complete list of features included in this update:

AlwaysOn support in SqlClient

New 4.0.2 feature

SQL Server Express Local Database Runtime support in SqlClient

New 4.0.2 feature

StateMachine (including designer support)

Existing 4.0.1 feature

SqlWorkflowInstanceStore on SQL Azure

Existing 4.0.1 feature

Compensation for custom hosts

Existing 4.0.1 feature

Note that we did not make any changes to the features that were shipped in Update 4.0.1.

Where can I get 4.0.2?

If you want to set up your development environment for building .NET Framework 4.0.2 applications in Visual Studio 2010 Service Pack 1 (SP1), download and install the Design-time Update (KB2544525); for more information, see the Knowledge Base article. You must install Visual Studio 2010 SP1 first, before you can use this package.
If you want to deploy your .NET Framework 4.0.2 applications, download and install the Run-time Update (KB2544514); for more information, see the Knowledge Base article. You must install the .NET Framework 4 or .NET Framework 4 Client Profile before installing the Run-time update.
If you want to set up your development environment for building .NET Framework 4.0.2 applications in Visual Studio 11 Developer Preview or later, download the Multi-Targeting Pack (KB2544526) standalone installer for Visual Studio 11 Developer Preview or later; for more information, see the Knowledge Base article.

Target framework changes in Visual Studio

Based on recent customer feedback, 4.0.2 exposes the full .NET Framework version number in the format Major.Minor,Revision (.NET Framework 4.0.2) in the Target framework dropdown in Visual Studio. Note that this update does not change the file versioning system for the .NET Framework files -- it only exposes the underlying target framework that you can use.

If you discover any issues while working with this update, please send them our way via Microsoft Connect.

Large Object Heap Improvements in .NET 4.5

Brandon Bray — Tue, 04 Oct 2011 01:57:00 GMT

Garbage collection is one of premiere features of the .NET managed coding platform. As the platform has become more capable, we’re seeing developers allocate more and more large objects. Since large objects are managed differently than small objects, we’ve heard a lot of feedback requesting improvement. Today’s post is by Surupa Biswas and Maoni Stephens from the garbage collection feature team. -- Brandon

The CLR manages two different heaps for allocation, the small object heap (SOH) and the large object heap (LOH). Any allocation greater than or equal to 85,000 bytes goes on the LOH. Copying large objects has a performance penalty, so the LOH is not compacted unlike the SOH. Another defining characteristic is that the LOH is only collected during a generation 2 collection. Together, these have the built-in assumption that large object allocations are infrequent.

Because the LOH is not compacted, memory management is more like a traditional allocator. The CLR keeps a free list of available blocks of memory. When allocating a large object, the runtime first looks at the free list to see if it will satisfy the allocation request. When the GC discovers adjacent objects that died, it combines the space they used into one free block which can be used for allocation. Because a lot of interaction with the free list takes place at the time of allocation, there are tradeoffs between speed and optimal placement of memory blocks.

A condition known as fragmentation can occur when nothing on the free list can be used. This can result in an out-of-memory exception despite the fact that collectively there is enough free memory. For developers who work with a lot of large objects, this error condition may be familiar. We’ve received a lot of feedback requesting for a solution to LOH fragmentation.

A Better LOH Allocator

In .NET 4.5, we made two improvements to the large object heap. First, we significantly improved the way the runtime manages the free list, thereby making more effective use of fragments. Now the memory allocator will revisit the memory fragments that earlier allocation couldn’t use. Second, when in server GC mode, the runtime balances LOH allocations between each heap. Prior to .NET 4.5, we only balanced the SOH. We’ve observed substantial improvements in some of our LOH allocation benchmarks as a result of both changes.

We’re also starting to collect telemetry about how the LOH is used. We’re tracking how often out-of-memory conditions in managed applications are due to LOH fragmentation. We’ll use this data to measure and improve memory management of real-world applications.

We still recommend some traditional techniques are for getting the best performance from the LOH. Many large objects are quite similar in nature, which creates the opportunity for object pooling. Frequently, types allocated on the LOH are byte-buffers that are filled by third-party libraries or devices. Rather than allocating and freeing the buffer, an object pool would let you reuse a previously-allocated buffer. Since fewer allocations and collections take place on the LOH, fragmentation is less likely to occur and the program’s performance is likely to improve.

Compatibility of .NET Framework 4.5

Brandon Bray — Tue, 27 Sep 2011 02:36:00 GMT

Fundamentals were a big part of our focus while building .NET 4.5. We divided fundamentals into seven areas called “tenets”. One of these tenets is compatibility. Today’s post is by Manish Agnihotri, a program manager who is driving compatibility across the .NET Framework. -- Brandon

.NET Framework 4.5 is an in-place update that replaces .NET Framework 4 (rather than a side-by-side installation). Our goal is for .NET 4.5 to be fully backward compatible with applications built for .NET 4 (.NET 3.5 and .NET 4.5 will be side-by-side). We’ll talk about the compatibility story for .NET 3.5 in a later post. One of the first things you’ll notice about .NET 4.5 is the version number (4.0.30319) is the same as .NET 4; this is the practice used by other in-place updates.

Our primary concern is guaranteeing applications you use do not break after you install .NET 4.5. We accomplish this by running hundreds of application in our compatibility lab to find issues as soon as they’re introduced. While designing new features or changing existing code, we keep compatibility in mind. And a small group of us, the Developer Division Compatibility Council (DDCC), monitor changes made by developers. We review potential breaking changes, and help teams understand and assess the compatibility impact of new features and bug fixes. For .NET 4.5, members of DDCC reviewed every proposed breaking change, every new feature, and a majority of the bug fixes for the release.

We’ve put a lot of effort into maintaining a consistently high bar for compatibility across the product, yet we know some issues may get past us. Many applications will exercise the .NET Framework in ways that we did not expect or we lack test coverage for. Still we care about knowing every issue, even those that may seem like corner cases. Once you install .NET 4.5 Developer Preview on a machine that previously had .NET 4, any compatibility issues can be sent to the Connect feedback site.

Types of compatibility issues

There are three kinds of version compatibility testing we do: (1) binary compatibility, (2) source compatibility, and (3) serialization compatibility. You may also find these approaches useful in your testing, and should an issue arise this may help you narrow down the root cause. Having a wide range of scenarios within each kind of tests is also critical to ensuring good compatibility coverage.

Binary compatibility uses binaries built targeting .NET 4 and then are run on .NET 4.5. Essentially, we’re testing that the behavior of newer .NET libraries is equivalent to previous versions. This can range from making sure the return value of a function is the same or that the same exceptions are raised. The hardest issues are multi-threading behaviors – sometimes performance improvements can be a breaking change.

Source compatibility takes an application that builds and runs successfully against .NET 4 then does the same with .NET 4.5. An interesting case is a machine where Visual Studio 2010 runs on a machine with .NET 4.5 installed – since reference assemblies are used for targeting versions of the framework, building an app in VS2010 and then running it is actually binary compatibility testing. To do source compatibility testing, we build against the .NET 4.5 targeting pack (which is part of the Visual Studio 11 Developer Preview).

Serialization compatibility deals with data and has several permutations. First, we let an app running on .NET 4 serialize data (e.g. save to disk) and then an app running on .NET 4.5 de-serializes (e.g. read from disk). Second, we do this in reverse by serializing from .NET 4.5 apps and de-serialize in .NET 4 apps. Third, we take a web service running on .NET 4.5 reading serialized data created by a .NET 4 based client. Fourth and last, we do the inverse with a .NET 4 web service and a .NET 4.5 client. A serialization problem will often show up as corrupted data or some kind of serialization exception.

Feedback welcome

Compatibility is an important part of migrating apps forward, though in rare cases it is necessary for a framework library to make breaking changes. We try to avoid this, using it as a last resort. MSDN documents the known breaking changes in .NET 4.5. We also have a migration guide with advice for testing your app.

If you discover any compatibility issues while working with previews and betas, we want to hear about them via Connect. Hopefully, all our work to make this a highly compatible release makes it easy for you to upgrade to .NET 4.5.

Introducing .NET Framework 4.5

Brandon Bray — Thu, 22 Sep 2011 04:08:00 GMT

Last week at the BUILD conference, we had the pleasure of announcing the next version of the .NET Framework and releasing a developer preview at the same time. We have so many new things in .NET 4.5 to discuss with our developer community – we’re excited to have this opportunity to begin a discussion about each of them.

At the BUILD conference, many talks provided a glimpse of what .NET 4.5 has to offer. These are some of our favorite talks:

The .NET platform is crucial to so many developers’ day to day lives. We’ve been working for the past two years to build the best platform yet. We started the last release by listening to what customer’s wanted, and worked hard to make improvements where we could while supporting important new Microsoft releases like Windows 8. Over this release, a few themes drove a lot of our decisions:

First class support for writing Metro style applications with .NET
Improve performance with little to no effort from application developers
Write code more easily for data access and web services
Make it possible to write portable libraries across platforms, and support a new processor
Address many top developer requests for .NET APIs and tools

We want to dive into each of these areas. Blogging allows us to have a two-way discussion about the data and considerations that went into the design-decisions. We’re very excited to talk about what we’ve been building.

Who are we

Together, Layla Driscoll and I will be your hosts on this blog series as we tour what you can expect from .NET 4.5. We’ll invite others from across all the .NET teams to give an inside look at the features and scenarios now available in the preview. While we both are on the CLR team, together we have driven a great deal of the planning and experiences for this release of .NET.

It doesn’t take long before developers see how much .NET has to offer. That’s because so many teams from all over Microsoft contribute to the framework. We have much to share, and we’ll do our best to bring the work of many teams into the discussion. All of the posts will be written by engineers on the .NET team.

Starting a new discussion

Keeping a line of discussion open with the developer community is a critical factor in how we make decisions for the .NET Framework. That’s one of the reasons why we keep the Connect site open for suggestions and bug submissions year round. Then we recently joined UserVoice to give the community a way to vote on requests and wishes important to you. Lastly, the MSDN Forums give the community a place to ask questions and provide answers.

Here on the .NET blog, we hope to spark new discussions about specific work. We want to share with you the context and decisions that went into the features and scenarios we built. We hope that the community will share their opinions and participate in helping us make this an outstanding developer product. Admittedly, we’ll make mistakes and we’ll recognize when we do. We’ll work hard to make the conversation constructive and learn from what you all have to say.

We do want comments, and many of us on the .NET teams will be watching. Like other Microsoft blogs, we hope to see in comments:

On topic discussions about .NET and the posts on this site
Respect for everyone in the conversation
Focus on the content of the post; do ask for more details

And please refrain from comments that would be considered abusive (according to the community standard), misrepresentation, repeatedly posting the same comment, or blog-spam.

With that, I want to say thank you for reading and I look forward to joining us in the discussion about .NET.

--Brandon Bray

Changes coming to ClickOnce applications running in the Internet Zone

Abhishek_Mondal — Tue, 07 Jun 2011 00:02:40 GMT

We wanted to let you know that we’ll be changing the signature verification process for ClickOnce applications and WPF XAML Browser Applications (XBAPs) in an upcoming update. This change will help users recognize when they’re running untrusted applications from the Internet Zone, but may require you to make changes to your applications. Currently, applications configured to run “online-only” will not display notifications requesting that the user make a decision whether to run the application or not. After the update, users will see an Application Run dialog commonly encountered when accessing applications over the Internet using Internet Explorer.

In some situations you may want to retain the current application launch behavior. There are two ways to achieve this:

1. Add the application publisher to the Trusted Publisher list. (http://msdn.microsoft.com/en-us/library/ms172241.aspx)

2. Add the application URL to the Trusted Sites list (http://windows.microsoft.com/en-US/windows-vista/Security-zones-adding-or-removing-websites)

Hardening Server Applications

Abhishek_Mondal — Fri, 01 Apr 2011 16:09:14 GMT

The Base Class Libraries team, the wider CLR team and Microsoft Research worked hard on this new technology and we are very proud to be able to share the news. Read the full article for details.

Is That the Right Version of Mscoreei.dll?

Abhishek_Mondal — Mon, 13 Sep 2010 22:39:10 GMT

After installing .NET 4.0 or later you may notice something a little unusual about your .NET processes. Here is a partial list of the loaded modules of a simple “Hello World” executable compiled with the .NET 2.0 compiler.

start end module name

60f00000 61491000 mscorwks C:\Windows\Microsoft.NET\Framework\v2.0.50727\mscorwks.dll

6c650000 6c6b6000 mscoreei C:\Windows\Microsoft.NET\Framework\v4.0.30319\mscoreei.dll

6d420000 6d46a000 MSCOREE C:\Windows\SYSTEM32\MSCOREE.DLL

75a80000 75aca000 KERNELBASE C:\Windows\system32\KERNELBASE.dll

Something here looks out of place – the mscoreei.dll file is loaded from the v4.0.30319 folder. What is it doing there next to the main clr dll (mscorwks.dll) from v2.0.50727? Actually, that’s the expected behavior. We call mscoreei.dll the “shim implementation”, or “shim impl” for short, and it’s new for .NET 4.0. The third dll in the list above, previously known as the “shim”, is now more accurately known as the “shell shim”. The two are tightly paired to do the main job that was previously done by mscoree alone – providing the interfaces for loading the runtime. Generally speaking, the shell shim now consists of thin wrapper functions, each of which delegates its functionality to a corresponding function in the shim implementation.

Why the split? We’ve seen significant numbers of machine reboots as part of .NET redist installation, typically due to needing to update a file that is in use. The most common file in use has been mscoree.dll, which is loaded by every single .NET application, and even some services such as MSI. As a result, we undertook the mscoree “split” to avoid machine reboots. By moving the shim implementation into a version-specific file, we are able to deploy a new version of mscoreei.dll (say in the v5.0 .NET folder, as part of .NET 5.0 installation), without touching the machine-wide mscoree.dll file. The next time a managed app is run, mscoree will dynamically find the new mscoreei and defer each of its function calls to it. That way we can deploy new versions of the framework on a machine running existing managed applications without requiring a restart.

Note that mscoree always uses the newest mscoreei it can find, but the actual runtime that is loaded is a completely different question. Therefore it shouldn’t be a surprise to see a newer version of mscoreei loaded with older versions of runtime dlls in the same process. In fact, CLR versions have been running with the latest shim for many years now. If I examine file versions when running a .NET 1.1 app on a machine that has both .NET 1.1 and .NET 2.0 installed I can see that I am using a shim newer than the runtime.

start end module name

79000000 79045000 mscoree C:\WINDOWS\system32\mscoree.dll

File version: 2.0.50727.42

791b0000 79412000 mscorwks C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\mscorwks.dll

File version: 1.1.4322.573

The difference now is that the split of the shim makes it more obvious that the shim version doesn’t always match the runtime version because the version number is right there in the path of the shim impl.

This brings to mind an aphorism from David Wheeler: All problems in computer science can be solved by another level of indirection.

In-Proc SxS and Migration Quick Start

Abhishek_Mondal — Wed, 23 Jun 2010 21:41:06 GMT

This post is meant to help you understand what runtime in-process side-by-side is, how to think about it, how to use it, and how it affects application and component migration to the .NET 4 Runtime. This post is relevant to you if you use native runtime activation APIs, depend on specific runtime activation behaviors, or use mixed mode assemblies built with Visual Studio Managed Extensions for C++ v8 or v9, or if you’re just interested in how we handle pre-.NET 4 and .NET 4 code coexisting on a machine.

The problem

Prior to the .NET 4 Runtime, any given process was limited to loading only one runtime version, and was bound to that runtime for remainder of that process lifetime. In environments where independently authored components built and tested against different .NET Runtime versions can be loaded into a single process, this sometimes caused compatibility problems for the components that did not target the runtime that was loaded into the process.

Our Solution

A new feature in the .NET 4 Runtime, “in-process runtime side-by-side” (or “in-proc SxS” for short) describes the ability to load more than one .NET Runtime version into the same process and have them run “side by side”. This ability to load multiple runtimes into the same process gives us with the ability to provide the highest level of compatibility possible for environments in which multiple independent managed components are loaded by, and communicate with, a native layer. COM is the biggest example of a native layer through which managed code must communicate, and this means that in an environment in which multiple independently-authored managed COM components are activated, each may be loaded into the runtime for which it was built and tested, which maximizes compatibility. The upcoming release of Visual Studio Tools for Office (VSTO) will take advantage of this by enabling all managed office addins to be loaded within their targeted runtime.

Our Solution Is Not…

In-proc SxS does not affect managed assembly loading, such as Assembly.Load or loads due to assembly references – these scenarios continue to load the target assembly in the runtime where the load request was made. To take advantage of in-proc SxS, a managed component must be activated by a native host and interact with its environment through a native interop layer such as COM interop and P/Invoke.

This may be easier to understand if you consider that that two runtimes loaded into a process operate completely independently; i.e., runtime X has no more knowledge about runtime Y than it does about any other native DLL loaded into a process. Each runtime has its own GC; each runtime creates, owns, and manages its own Application Domains (never the other way around), including Shared and Default domains for each; and each runtime interacts with unmanaged DLLs through interop layers.

How to Think About In-Proc SxS

Thinking about multiple runtimes in the same process without additional context isn’t all that helpful; in fact, it just leads to more questions: “How and when does this happen?”; and most importantly, “Does this affect me?”

Does In-Proc SxS Affect Me?

In-proc SxS was designed so that existing applications, components and libraries should be completely unaffected by an installation of the .NET 4 Framework - applications that were running against a pre-v4 framework will remain blissfully unaware of the install.

In addition, migrating a product to .NET 4 is unlikely to be affected by in-proc SxS if:

· The product is a pure managed application;

· The product is a pure managed library; or

· The product is a pure managed COM component.

However, a product may be affected by in-proc SxS during migration if:

· The product makes use of one or more deprecated APIs; or

· The product contains, or takes a dependency on, a mixed mode assembly built with Visual Studio Managed Extensions for C++ version 8 or 9.

See the migration section below for more information.

Runtime Activation

For the small number of you who may be affected by in-proc SxS and need to gain a better understanding of it, the easiest way to understand how and where in-proc SxS fits into the managed world is to think about it as a fundamental change to our runtime activation model. Runtime activation describes the process by which the most appropriate runtime version is found, located, and loaded. A runtime request consists of a set of inputs, which can include (among others) an assembly (often the application’s EXE), an application configuration file (often found next to the application’s EXE), a version string, or any combination thereof, and runtime activation uses these inputs to determine the most suitable match from among the set of installed runtimes. Common examples are: managed EXE launch, CoCreateInstanceInstance of COM-visible managed types, and mixed-mode assemblies loaded from native code.

Old Activation Model

In prior releases, when in-proc SxS did not exist, our runtime activation policy had to take into consideration the reality that only one runtime could ever be loaded into a process. For self-contained managed applications the obvious choice was invariably the right one: activate the runtime that the application targets (derived either from metadata in the managed executable itself or from an application configuration file if one exists). And since virtually all managed application installers out there list the application’s target runtime as a pre-requisite, the application could count on the target runtime being installed. Unfortunately, runtime activation was not always as clear cut in other important scenarios, the most prevalent of which is the activation of managed COM components. In such scenarios, the chosen policy has always been to activate in the latest installed runtime when no runtime has yet been loaded – a policy we call “bind to latest.”

Consider the following example to help understand why this policy was chosen for managed COM class activation. Let’s say that a native application X exposes an extensibility mechanism through COM, and two authors separately write managed extensions A and B that target .NET Frameworks v1.1 and v2.0 respectively, and that application X and both extensions are installed on machine Z with both .NET Frameworks v1.1 and v2.0 installed.

Say that, in the course of normal execution, application X CoCreateInstances component A. In this case it may be tempting to use the same activation policy as that of managed executables and pick the runtime that the component targets (v1.1). However, this choice would cause a later attempt to CoCreateInstance component B to fail, because the .NET Runtime v1.1 (to which the process has been locked) cannot load a v2.0 assembly.

To avoid this problem, the policy for managed COM component activation has always been to activate and load the component into the latest installed runtime, if a runtime has not yet been loaded. However, this policy created an impossibly high compatibility burden: we would need to provide 100% release-to-release compatibility, and to do so we would need to test every possible usage scenario for every managed application ever built on any previous .NET Runtime! So, while the entire .NET team will always set very high release-to-release compatibility requirements, we can all sleep a bit easier knowing that in-process side-by-side enables us to provide a much better compatibility model for our customers, insulating them from the small but inevitable accidental breaking changes that occur in a major product release.

New Activation Model

I’m sure you’ve guessed the fundamental change to our activation policy by now: starting with version 4, the .NET Runtime activation policy may now load the most appropriate runtime version for a given activation request, without regard for the set of runtime version(s) that may already be loaded into the process. The activation policy will no longer target the latest runtime on the machine as it did in previous versions (more details on this below). Again, there are some restrictions with when dealing with pre-v4 runtimes, but the above statement will hold true, without restriction, for all .NET Runtimes version 4 and higher.

Using the previous example, but with the small change that component A has been updated to target .NET 4, runtime activation policy is now free to activate component A in the v4 runtime and later activate component B in the v2 runtime. This provides the greatest compatibility by allowing components to run within their target runtime, without regard for the process environment into which it is loaded.

Now that we’ve covered the basics of the new in-proc SxS feature and activation model, let’s look at the necessary considerations when migrating an application, component or library to .NET 4 from an earlier runtime.

Migration

In the vast majority of cases it is simple to migrate an application to run on .NET Framework 4: if you are migrating an application at the source level, import your solution into Visual Studio 2010 and choose “.NET Framework 4” for the Target Framework in each project’s properties page, then recompile; if you are migrating an application without recompiling, create or update the application configuration file to target .NET 4.For example, the following is the recommended <startup> section to use when migrating an application that targets .NET 3.5 to also target .NET 4:

<?xml version ="1.0"?>

</startup>

</configuration>

Note that starting with .NET Framework 4, only a two-part version string is required, though three-part version strings are still accepted.

A migration decision tree is provided later in this section.

Limitations and Exceptions

When transitioning code originally targeting an in-proc-SxS-unaware runtime to an in-proc-SxS-aware runtime, some additional steps will need to be taken if your application contains one or more of the following:

· calls to any of the hosting global static functions, from native and/or managed code;

· calls to any of the Strong Naming Global Static Functions, from native and/or managed code;

· native code that calls CoCreateInstance using any of the hosting CoClasses (CLSID_CorRuntimeHost, CLSID_CLRRuntimeHost, CLSID_TypeNameFactory, or ComCallUnmarshal), or is managed code that uses COM interop to access interfaces implemented by any of the same CLSIDs;

· calls to CoCreateInstance, from native code, using any of the metadata CoClasses (CLSID_CorMetaDataDispenser, or CLSID_CorMetaDataDispenserRuntime), or managed code that uses COM interop to access interfaces implemented by any of the same CLSIDs;

· calls to CoCreateInstance, from native code, using the CoClass of any COM-registered .NET Framework type;

· a managed COM component (in the binary migration case only);

· an assembly produced with the Visual Studio Managed Extensions for C++ compiler (otherwise known as a managed C++ assembly), or a dependency, either direct or indirect, on such an assembly.

Figure 1

The functionality in the above list was created for prior releases, and each lacks the ability to specify the target framework that is needed for in-proc SxS, and could not be retrofitted to do so. For the most part, all of these chose the same “bind to latest” activation policy as managed COM component activation.

Non-Impactful Install and “Latest Runtime” Activation Model

Why am I even talking about our “bind to latest” activation policy? Well, there’s a very good reason: one of the primary compatibility-related goals of the .NET 4 Framework is to ensure that its installation is non-impactful to pre-existing managed application or component. To achieve this, we have had to modify the “bind to latest” activation policy’s semantics in the .NET 4 release.

The “bind to latest” runtime activation model kicks in when no target runtime version information is provided with the activation request, such as through a managed assembly’s built-in version, a configuration file’s list of <supportedRuntime> entries, and/or an explicit version string passed into any of the native hosting APIs that accept them. In these cases, in previous versions of the runtime, “bind to latest” activation would pick the very latest .NET Runtime installed on the machine. This is also the activation model used for managed COM classes built against pre-v4 runtimes, for precisely the reasons stated in the earlier example.

Starting with the .NET 4 Runtime, this “bind to latest” activation policy has been modified so that, by default, it will only consider pre-v4 runtimes. By making this the default behavior, all pre-existing applications and components using this activation model will be unaffected by the installation of .NET 4.

Source Migration

If you intend to retarget your application or component to .NET 4 and rebuild, and it uses any of the functionality listed above, then you need to migrate to the equivalent functionality provided in the new .NET 4 Framework Hosting interfaces:

· for calls to any of the hosting global static functions, use the corresponding methods provided by the new ICLRMetaHost, ICLRMetaHostPolicy and ICLRRuntimeInfo interfaces;

· for calls to any of the Strong Naming Global Static Functions, use the corresponding methods provided by the new ICLRStrongName interface;

· for instance creation of any of the hosting CoClasses, metadata CoClasses, or COM-visible Framework types, use ICLRRuntimeInfo::GetInterface.

Figure 2

Each ICLRRuntimeInfo and ICLRStrongName interface instance targets a specific runtime, and each method exposes functionality specific to, or provided by, that targeted runtime.

Binary Migration

If the preferred option of source migration and re-compilation is not possible, and the application or component also depends on any of the functionalities listed above, an application configuration extension is available to make migration possible in many cases. Adding this extension to the application’s configuration file will modify the “bind to latest” policy to cause it to bind to the same runtime as the application itself. The extension takes the form of an attribute named useLegacyV2RuntimeActivationPolicy, to be placed in the <startup> element of an application configuration file. The attribute’s valid values are “true” and “false”, of which “false” is the default. When set to “true”, this attribute instructs “bind to latest” runtime activation policy to consider all <supportedRuntime> entries in the application configuration file; when set to false, the policy considers only those <supportedRuntime> entries with a major version number of 2 or less. When a configuration file does not contain any <supportedRuntime> entries (or a deprecated <requiredRuntime> entry), then the version number contained in the executable image, which represents the runtime version against which the executable was built, is treated as an implicit <supportedRuntime> entry (roughly speaking).

Thus, if you were migrating a .NET 3.5 application to .NET 4, your configuration file might look like this:

Figure 3

Managed COM Component

To accommodate managed COM component authors (who in most cases cannot modify application configuration files), a separate mechanism is available: a semicolon delimited list of supported runtimes may be added to the component’s InprocServer32 registration. Specifically, add the string value “SupportedRuntimeVersions” to “HKCR\CLSID\<COM component clsid>\InprocServer32\<COM component version>”.

It is important to note that if a managed COM component depends on any of the items in Figure 1, then it must be migrated at the source level. This is because the COM component would need functionality equivalent to the new useLegacyV2RuntimeActivationPolicy configuration extension to work, which is a process-wide policy decision that must be made by the application author rather than by extensibility components.

If a managed COM component is activated through a reg-free manifest (which cannot be extended to contain a SupportedRuntimes entry), runtime activation will look for a configuration file next to the assembly containing the COM visible class. Note that a useLegacyV2RuntimeActivationPolicy entry will be ignored in this scenario.

Assemblies Built with Managed Extensions for Visual C++ Compiler

There are two additional limitations with managed C++ assemblies.

First, any dependency on a mixed-mode assembly built with a previous release of Visual C++ must be migrated using the useLegacyV2RuntimeActivationPolicy configuration file extension. This is because all such assemblies depend implicitly on the managed CRT, which in turn contains dependencies on functionality listed in Figure 1.This dependency has been removed in the upcoming release.

Second, all mixed-mode assemblies (including those built using this release of Visual C++) and all assemblies built with a previous release of Visual C++ may be loaded into at most one runtime per process, because they may contain process-global static image data that cannot be virtualized across runtimes, and/or because they depend on a prior release of the managed CRT. This limitation is virtually identical in nature to the current limitation restricting these types of assemblies to no more than one application domain at once.

Migration Decision Tree

Use the following decision tree to decide the correct migration path for your component in the most common scenarios.

Binary Migration Examples

Application: Target v2 and v4

Create an application configuration file with the following contents, where the highlighted portion is required only if the application depends on functionality from Figure 1:

<configuration>

<startup useLegacyV2RuntimeActivationPolicy="true">

<supportedRuntime version="v4.0"/>

<supportedRuntime version="v2.0.50727"/>

</startup>

</configuration>

As always, supportedRuntime entries are considered to be listed in decreasing preference; if the .NET 2 Runtime is preferred, the entries should be reversed.

Application: Target v4 Only

Use the same application configuration file as above, but remove the v2-specific supportedRuntime entry.

Managed COM Component: Target v2 and v4

If you authored a managed COM component built against the .NET 2 Runtime, with a CLSID of {DFEEABC2-17FF-4ce6-B25B-A35F7A156C68} and a version of 6.2.0.0, and you wanted the component to run within either the .NET 2 Runtime or the .NET 4 Runtime (in that order of preference), the registry update (in RegEdit syntax) would be:

[HKCR\CLSID\{DFEEABC2-17FF-4ce6-B25B-A35F7A156C68}\InprocServer32\6.2.0.0]

"SupportedRuntimeVersions"="v2.0.50727;v4.0"

If your component was registered without the assembly version portion of the registry key name, you would remove “\6.2.0.0” from the key name.

Or, if your managed COM component is activated through registry-free COM, you would place an application configuration file (should we coin the term component configuration file?) next to the managed COM assembly with contents similar to the example above (without the highlighted portion, and with the supportedRuntime entries reversed).

Troubleshooting

Instantiating Managed COM Components From Managed Code

If your application instantiates and uses COM components through the .NET Runtime’s managed interop layer, and one of those COM components happens to be managed, then there is a small chance that after migrating your application you might run into an error similar to this:

Error: Unable to cast COM object of type 'System.__ComObject' to interface type 'X'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{12345678-1234-1234-1234-123456789012}' failed due to the following error: No such interface supported (Exception from HRESULT: 0x80004002 (E_NOINTERFACE)).

This error can occur as a result of the following two combined conditions:

1. The managed COM component is registered against a different .NET Runtime version from that of the instantiating caller, causing it to be activated in another runtime than that of the instantiating caller.

2. The interface is not marked with System.Runtime.InteropServices.ComVisibleAttribute, with a value of true.

This scenario worked in the past because of a runtime optimization: when the .NET Runtime notices that a COM type it instantiates has a managed implementation, and that it was instantiated into the same Application Domain as the instantiating caller, it bypasses the managed interop layer and returns to the caller the managed object itself. In prior .NET Framework releases (where in-proc SxS was not available), all managed COM types and their managed clients were commonly loaded into the same Application Domain, allowing the implementing managed object to be returned, and in turn allowing the interface cast succeed without being marked as ComVisible.

Migrating an application can easily create condition 1; if the application also relies on condition 2 (which is much less common), then the above error will be encountered. There are a couple of solutions available:

1. Get an updated version of the managed COM type that properly declares the interface as ComVisible.

2. Update the application configuration file to use the useLegacyV2RuntimeActivationPolicy extension.

Loading Managed C++ Assemblies Built Against Previous .NET Runtime Versions

Attempting to load a managed C++ assembly that was built with an earlier compiler into the .NET 4 Runtime when the application has not been configured with the useLegacyV2RuntimeActivationPolicy configuration file extension will result in the following error:

System.IO.FileLoadException: Mixed mode assembly is built against version 'v2.0.50727' of the runtime and cannot be loaded in the 4.0 runtime without additional configuration information.

Note: the error is not 100% correct, as it specifies that only mixed mode assemblies are subject to this problem; it should say “Managed C++” instead of “Mixed mode”, as managed C++ assemblies compiled with /clr:pure will also encounter this error.

It is possible to correct this issue with either of the following:

1. Recompile the assembly with the .NET 4 managed C++ compiler.

2. Update the application to use the useLegacyV2RuntimeActivationPolicy configuration file extension.

NGen: Walk-through Series

Abhishek_Mondal — Tue, 04 May 2010 01:22:51 GMT

The CLR Code Generation team blog has a series of new posts featuring hands-on style content around how to use the NGen technology and how to measure performance benefits from it. You can find the series here.

Connect

Abhishek_Mondal — Tue, 13 Apr 2010 23:42:00 GMT

As hopefully most of you know, Microsoft has a site called Connect where customers can log bugs and suggestions (http://connect.microsoft.com/). Issues for .NET can be found in the Visual Studio and .NET Framework feedback category (http://connect.microsoft.com/VisualStudio/Feedback) and can be logged from the main Visual Studio Connect page (http://connect.microsoft.com/VisualStudio/).

One of the complaints we often hear from customers is that they don’t know what happens to a Connect issue after they file it. So I wanted to take this chance to tell you a little about how the process works on the CLR team.

Once the issue is opened, there’s a first level of screening just to figure out which team it should be sent to. It usually ends up assigned to someone in the product within a few days. If the initial assignment was incorrect, though, it could take a couple more days to get it to the right person.

If the issue is a code defect, we treat it like any other product bug. We make sure it’s reproducible and not by design and something we can fix for the current release depending on where we are in the product cycle. We don’t necessarily do this right away, though. There are times in our product cycle when we’re more focused on bugs than others, and the bar gets higher toward the end of the release when all changes are riskier. Any change can introduce regressions in compatibility, performance, or behavior, and taking changes toward the end gives us less time for stabilization. We might not be as good about delivering status updates on Connect as we could be, but we do try to comment on each issue.

When deciding whether to fix a bug, there are several criteria we look at. One of them is whether anyone might have taken a hard dependency on the existing behavior. We don’t want to solve one problem but end up with a bunch of customers whose applications no longer work after upgrading to the newest version or applying the latest patch. There are occasionally security reasons to make breaking changes, but we try to avoid them when we can. We also look at things like how hard it is to work around the bug and how many votes there are on the issue. The severity of the issue also affects the priority: Is this a bug that will crash your machine, or is it just a minor annoyance?

Another thing we look at when deciding to fix a bug is how many people are likely impacted. If you find a new bug in an older product, like .NET 2.0 that’s been out for 5 years and used by hundreds of thousands of developers, you might be one of only a few people who is impacted by that bug. But if you file a bug on a newer product, like .NET 4, that’s has just been released, there’s a good chance that hundreds or even thousands of people are impacted. Because of this, we take bugs on products in Beta, or that have just been released, particularly seriously.

Once we do decide to fix a bug, we have to decide where to fix it. Our default answer is generally to fix it in the next major or service pack release (which is usually the one we’re working on at the time). Only fixes for particularly severe bugs might be included in an update to an existing release. If you are blocked on a bug for an existing release that’s been out for longer, e.g. .NET 2.0, and you need a fix, Connect is not the place to file a bug. You should be contacting Microsoft customer support to request a QFE.

We tend to get a high number of suggestions, both from Connect and other sources, and we don’t always get a chance to look at them and respond right away. But we do look at all the suggestions that come in on Connect. When suggestions come in, we consider whether it’s something we might do in this release or a future release. We also check to see if it’s a duplicate of an existing suggestion

Aside: please search before you file. One suggestion with 20 votes has more impact than 20 suggestions with 1 vote. You can sort by number of votes on the site, and that helps us focus our attention when it comes to suggestions. For example, we recently fixed a SerialPort issue and a serialization issue that had a high number of votes.

We try to close issues if there isn’t a chance we’d ever do them, but we leave ones we might consider open so that the community has a chance to vote and comment on them. Again, these can go for a while without comment, until we get to a point where we can act on them. However, we do have a mechanism to be alerted when someone comments on an issue. It’s opt-in, so it may not be universally used, but it may get someone’s attention. If suggestions come in while we’re working on a release and already have plans, it’s less likely we’ll be able to implement anything large for that release.

One thing that really helps us with suggestions is when the poster or commenter includes details on what problem this would help them solve. With a lot of suggestions, people jump straight to the suggested solution. You may think this is a shortcut for us, but it’s not always. We see those and realize that maybe we could do that, but don’t necessarily know why we should. What would you be able to do if you had this that you can’t do now? Often there are several ways to solve a problem (at least). If we know what you’re trying to accomplish, it helps us measure whether our solution really met the goal. Or if we see lots of customers with similar problems, we might be able to come up with a slightly different solution that meets lots of needs. And it’s a lot easier to reason about that from the problem descriptions than from the often-diverse solution descriptions.

If you’re wondering about the kinds of suggestions we’ve acted on, in .NET 4 we took suggestions ranging from new numeric types to adding single APIs or overloads that helped customers out.

For those of you wondering when we’ll get to these suggestions, the answer is now. With work on .NET Framework 4 coming to an end, we’re starting to look at the open suggestions in more detail. Through May we’ll be trying to close ones we don’t think we’ll be able to do in our next cycle. We’ll also be trying to work through any bugs that might still be open. The fact that a suggestion is still open in June doesn’t mean you’ll necessarily be seeing it any time soon, but only that we’ll take the time to look into it more closely.

If you’ve been thinking about voting or commenting on a suggestion, or opening a new one, now would be a good time. If we close one that you feel passionately is more important than some of that other stuff we’ve left open, let us know that, too.

Announcing a new CLR Team Codeplex Project: CLR Runtime Activation

Abhishek_Mondal — Tue, 16 Feb 2010 21:04:23 GMT

This Codeplex site will feature demos, samples, and helper libraries related to the selection and activation of the CLR: http://clractivation.codeplex.com/

With .Net Framework 4 and VS 2010 the new ability to load multiple versions of the runtime has changed the way a version is chosen for any particular piece of managed code and gives developers much more control over the runtime their code will end up running against. For a detailed introduction to this new functionality you can see this MSDN magazine article. This CodePlex project will focus on the various ways to start up the runtime and how to influence these choices using the new runtime activation APIs introduced in .Net Framework 4. Our first two projects include a helper library to ease migration from the old to the new APIs and a managed wrapper of our new native activation APIs.

Sharing Silverlight Assemblies with .NET Apps

Abhishek_Mondal — Tue, 01 Dec 2009 23:08:00 GMT

At the recent PDC, Scott Guthrie announced in his Silverlight 4 keynote that we had implemented a new feature, to enable developers to share certain assemblies between Silverlight and .NET. There are many differences between Silverlight and full .NET including WPF, and this new feature doesn’t solve those differences – in those cases, you’ll still need to compile your code twice. But in some cases, developers will write code that only uses features whose behavior is identical between Silverlight and full .NET, and in those cases, we want to enable that code to be shared. This post provides more detail on that sharing, and explains how developers can target it, and what the restrictions are.

Developer Scenario

Today, many developers write code that is intended to run on both Silverlight and .NET. A good example would be validation code: when writing a client-server application, you want to validate the data at the client (to make sure the user gets quick feedback) and then re-validate it again at the server (to make sure a rogue client implementation can’t send back bad data.) Today, in order to get that scenario to work, developers need to compile their source for Silverlight and also for .NET. In addition to dual-compilation, developers need to manage and deploy those assemblies separately, ensuring that they end up being deployed to the right places and consumed by the right platform (Silverlight or .NET). Today’s model is certainly workable, but it is not ideal.

We’ve heard from many developers that you want something better. We’ve also read quite a number of blog and forum posts stating the same. We’ve even seen some interesting workarounds to try and make the situation better for themselves and other developers. Cool! Specifically, developers want to write and compile their code once and deploy it as part of both their Silverlight and .NET apps, without having to dual-compile or worry about paying attention to the compilation target. This ability has the obvious benefit of avoiding duplication of effort for a number of the steps in your development and deployment processes.

Portability Explained

We named this new feature “assembly portability”, given that the feature allows your code to be “ported” between both Silverlight and .NET. Portability provides the ability to compile your source with the Silverlight tools, and run your built assemblies on both the Silverlight and .NET runtimes. This feature doesn’t change the underlying implementation of either the Silverlight or .NET runtimes; instead, if you write code that only uses APIs that have the same behavior across Silverlight and .NET, it allows you to use one set of binaries to target both. But how do you know if the APIs you are using are compatible? We have identified five key assemblies that are compatible between Silverlight and .NET. (The Silverlight UI layer is of course not one of these – there are some important differences between Silverlight UI and WPF.)

Like most features we build, we had a lot of choice about how to design the feature, and which scenarios we would enable. At the heart of the feature, a major design choice was to enable portability from Silverlight to .NET. The motivation for this choice was that Silverlight exposes an API subset of .NET, and so assemblies built with Silverlight should “just work” on .NET, whereas the reverse direction (.NET assemblies running on Silverlight) would have proven more technically challenging (for us all).

Another important design choice was identifying the set of Silverlight/.NET assemblies that developers could safely use, while still maintaining binary comparability. We looked at the scenarios that would both most benefit from portability, and would be straightforward for developers to use. We looked at a bunch of scenarios, including: the most low-level, typical business logic, networking and also UI. We decided we would start with the most fundamental and requested scenarios for this release. As a result, for .NET 4 and SL4, we have enabled portability for a significant set of low-level assemblies that we believe will enable a variety of interesting scenarios.

For SL 4 and .NET 4, we have made the following assemblies portable:

Mscorlib
System
System.Core
System.ComponentModel.Composition
Microsoft.VisualBasic

Note again that the Silverlight surface area for these assemblies is what we made portable. There are a great number of types and members in the .NET surface area that cannot run on Silverlight. As a result, you need to write code that targets the Silverlight versions of these assemblies in order to get this scenario to work.

Also note that there may be some behaviors that are not strictly identical across both platforms. We’ve worked to avoid those behaviors, however, those differences are sometimes very subtle and difficult to identify. Please do contact us if you do see behavioral differences between Silverlight and .NET, for portable APIs, which are bothering you.

Visual Studio Experience

Like all .NET features, it is important that we provide good tooling support in Visual Studio. This feature can be used in Visual Studio, in just the way that one might imagine. Developers should write their portable logic in Silverlight Class Library projects, and then are free to reference such projects from both Silverlight and .NET application projects. To clarify, on the .NET side, you can reference such a library from any kind of .NET project (Ex: WPF, winforms, WCF, WF, ASP.NET, …).

The only caveat is that the support in Visual Studio was implemented in such a way that you need to click a few extra times to get a reference to a portable project correct setup. For VS 2010, you must rely on browsing to the binary – AKA “binary reference” – that is built from the Silverlight class library to setup a reference. You cannot just reference the project itself – AKA “project to project reference”. Note that this restriction only exists for .NET apps, and not Silverlight ones.

The following are the basic steps to follow to enable the use of portable code in Visual Studio 2010, followed by a set of screen caps that hopefully make it super clear what to do.

Steps:

Setup projects
1. Create .NET application
2. Create or add existing Silverlight class library project
3. Build Silverlight project
Establish portable code reference
1. Reference Silverlight class library binary from the .NET project
Code
Run App!

Screen Caps:

Step 1 – Setup projects

Step 2 -- Establish portable code reference to the other project (required going up and back down the directory structure to the Silverlight library project).

Step 3 – Code

Step 4 – Run App!

Note: My app is a WPF app that is intended to display how many days there are until the Vancouver 2010 Olympics start, from today. The Silverlight library is the one that does the actual countdown calculation, and is (naturally) usable in both .NET and Silverlight apps. Cool!

Looking Forward and Feedback

It is great to see so much energy around both .NET and Silverlight, and particularly around the maximal sharing of code across the platforms. I encourage you to use this new scenario to its fullest extent, and to communicate back to us where you’d like to see the feature expanded in the future.

We are aware of some scenarios that are not currently enabled to be portable, but that developers will probably run up against pretty quickly. The most obvious ones are XML, networking, calling WCF web-services and UI code. Another interesting one is portable XAML. It would be really useful to hear from you on the specific scenarios that you would like to implement in portable code, but cannot implement due to the limitations that I’ve mentioned. I’d also like to hear why you saw being able to implement that scenario in portable code as being such a benefit. Your feedback will help guide us with future changes in the portable code space.

How to Make the Most of Your .NET Server Code

Abhishek_Mondal — Fri, 13 Nov 2009 18:32:00 GMT

One of our team’s field engineers recently sent a link to a Channel 9 video: Steve Michelotti of e.magination on High Performance Web Solutions. This company built a 64-bit web server that handles over 3 billion transactions a day and guarantees a 250 ms response time. And it goes without saying that they built it on top of .NET.

The team optimized their code to avoid implicit allocations, pool and reuse objects and allocate large collections directly on the Large Object Heap. But they still saw some latencies that were greater than their contract allowed. So they worked with their field engineer and the GC team to get a new feature implemented in .NET 3.5 SP1 that let them work around the problem.

The problem comes from the size of their managed heaps. When you try to manage memory on a 16 GB server it can take a few seconds to run through the heaps. There’s not much you can do on a single machine besides keeping your heaps reasonably sized. Having millions of objects in memory is expensive. But their architecture is such that they can direct requests from the web server to any of a set of identical application servers. This means that they can take any individual server offline for the few seconds it takes to do a full collection. All they needed is to know when a GC is going to happen and when it’s completed. Full GC notifications is a new feature in 3.5 SP1 that gave e.magination necessary insight into the GC’s behavior that lets them redirect requests away from servers that are doing full GCs.

This video is only about 20 minutes long but it’s packed with great insights into how to make the most of your .NET server code. It shows the team at e.magination doing a fantastic job of analyzing and tuning their system for optimum performance. I found it interesting and inspiring. I hope you do too.

Automatically Capturing a Dump When a Process Crashes

Abhishek_Mondal — Fri, 16 Oct 2009 03:35:00 GMT

I recently received the following question from a customer:

“During our test runs (which might run for hours), if a process crashes, we’d like to create full memory dumps for later diagnosis. Can I configure the machine to do this automatically?”

I’ve actually gotten this question numerous times over the past couple of years. It comes in various forms but the general scenario is: A user wants to be able to diagnose application crashes and the environment does not lend itself to live debugging. The latter criterion could be for a number of reasons, but the most common I see are:

· We have an intermittent failure in production and want to gather a dump to debug the issue offline.

· I’m running a bunch of tests and when one crashes I don’t want to interrupt the whole run to diagnose the issue at failure time. Let’s just gather some information for triaging.

· Our issue isn’t reproducible under a debugger. E.g. a stress bug.

These essentially reduce to: You want to get as much data as you can, while minimizing the impact to the environment. Given these requirements, the solution I find meeting most people’s needs is to configure a just-in-time (JIT) debugger to launch, grab a dump, and exit whenever a process crashes.

Support for just-in-time debugging has been in the CLR since V1x and in the OS for as long as I can remember. The basic idea behind a JIT debugger is: when a process crashes, launch and attach a debugger so you can figure out why.

There are registry keys which provide this general ability for both managed and native code (support for the latter might actually be in Win.ini for 9x/Me). If your application is written in managed code (which is the case I’m often presented) you might ask, “My application is managed code, why do I care about native code?” Given that even the most simple managed applications run native code (e.g. the runtime itself), if your requirement is to gather data for any crash, you’ll need to set the keys for both types of code. In CLR V4 we’ve actually unified the key which controls the managed JIT debugger with the native one. However, that change does not change my guidance since, for the time being, we’ll be living in a world where V2 managed code exists alongside V4.

How do I configure the debugger?

1. Download and install the latest “Debugging Tools for Windows.”

a. If you’re running a 64-bit OS, you’ll want both the 32- and 64-bit versions*

b. You can either install the entire set of tools on the machine (it’s a quick, small install) or you can install to one machine and copy “cdb.exe” from the install directory to any number of target machines.

*Note: My sample .reg files below assume you install the 32-bit debugger to c:\debuggers\x86\ and the 64-bit version to c:\debuggers\x64\.

2. Create/set the following registry keys and values (If you’re working on a 64-bit version of Windows, you’ll need to set these keys under the Wow6432node as well.†):

a. Key: HKLM\Software\Microsoft\Windows NT\Current Version\AeDebug:

i. Value: “Debugger”

1. Type: String

2. Value data: <path to cdb> -pv -p %ld -c “.dump /u /ma <dump file path\name.dmp>;.kill;qd"

ii. Value: “Auto”

1. Type: String

2. Value data: “1”

b. Key: HKLM\Software\Microsoft\.NETFramework

i. Value: “DbgManagedDebugger

1. Type: String

2. Value data: <path to cdb> -pv -p %ld -c ".dump /u /ma <dump file path\name.dmp>;.kill;qd"

ii. Value: DbgJITDebugLaunchSetting

1. Type: DWORD (32-bit)

2. Value data: 2

†Note: You should set the keys to point to the appropriate “bitness” debugger. I.e. you want the OS/CLR to launch the 64-bit debugger for 64-bit process crashes and the 32-bit version for 32-bit crashes. Make sure your debugger paths are set accordingly.

The following sample .reg file will set cdb.exe to auto-launch and generate a crash dump for every process crash on the machine. Note the assumptions the file makes about debugger paths and the dump-file-placement path.

Windows Registry Editor Version 5.00

;This reg file installs just-in-time debuggers to capture a dump of all process
;crashes for the machine.

;

;Assumes 32-bit debugger is cdb.exe and is installed to c:\debuggers\x86\.

;Assumes 64-bit debugger is cdb.exe and is installed to c:\debuggers\x64\.

;

;Assumes crash dumps can be written to c:\crash_dumps\.

;Make sure all users have write access to this directory.

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework]

"DbgManagedDebugger"="\"c:\\debuggers\\x64\\cdb.exe\" -pv -p %ld -c \".dump /u /ma c:\\crash_dumps\\crash.dmp;.kill;qd\""

"DbgJITDebugLaunchSetting"=dword:00000002

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug]

"Debugger"="\"c:\\debuggers\\x64\\cdb.exe\" -pv -p %ld -c \".dump /u /ma c:\\crash_dumps\\crash.dmp;.kill;qd\""

"Auto"="1"

;The following keys are only used on 64-bit versions of Windows (note Wow6432Node).

;They can be safely created with no side-effects on 32-bit versions of Windows.

;Alternatively, you can delete the remainder of this file if you’re running a

;32-bit version of Windows.

[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Windows NT\CurrentVersion\AeDebug]

"Debugger"="\"c:\\debuggers\\x86\\cdb.exe\" -pv -p %ld -c \".dump /u /ma c:\\crash_dumps\\crash.dmp;.kill;qd\""

"Auto"="1"

[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\.NETFramework]

"DbgManagedDebugger"="\"c:\\debuggers\\x86\\cdb.exe\" -pv -p %ld -c \".dump /u /ma c:\\crash_dumps\\crash.dmp;.kill;qd\""

"DbgJITDebugLaunchSetting"=dword:00000002

What do these keys do?

The “Debugger” and “DbgManagedDebugger” value data are basically command lines (printf format strings) that are run when a process crashes. The OS or CLR will substitute values for the format specifiers (e.g. in the case above it substitutes the crashing process ID for the “%ld”) and run the command in the user context of the crashing process. The command line I’ve supplied:

· launches cdb.exe, the debugger.

o Obviously, you must specify the correct path to the debugger.

· “-pv %ld” : Attaches non-invasively (just suspends the threads) to the crashing process (the OS or the CLR will actually fill in the PID for you).

· “.dump /u /ma <dump file path\name.dmp>”: Takes a full memory dump with a unique name (appends the date, time and process ID) and stores it in the path defined.

o The path and file name can be whatever you want them to be. Since the debugger is launched in the context of the crashing process, make sure the path points to a location all accounts can write to.

· “.kill”: Kills the target process (you’ve gotten the data you need).

· “qd”: Quits the debugger.

The “Auto” and “DbgJITDebugLaunchSetting” values set the policy for when to launch the debugger. As I wrote above, we want to get our data as quickly as possible and move on, so we don’t want to require any user intervention. For example, in the server scenario, there may be no one logged into the machine to click some “OK” button. The settings I described will automatically launch the registered debugger for all processes on the machine, without prompting (see Enabling JIT-attach Debugging for more details on the settings). Note that when these settings are in place, the debugger will be launched for crashes in all processes running on the machine and you will not have the option of submitting the crash through “Windows Error Reporting.” In future posts I’ll discuss ways of enabling both.

I don’t care about most processes on a machine. Can I just target specific ones?

The answer depends on the version of the OS and the version of CLR. Here are the rules:

· For native code: your OS must be Vista/Server 2008 or higher.

· For managed code: Your version of the CLR must be V4 (or higher, if you’re reading this from the future).

And here is how you configure:

1. Set the debugger keys (AeDebug\Debugger and .NETFramework\DbgManagedDebugger) just as you did in 2.a.i and 2.b.i, above.

2. Ensure AeDebug\Auto and .NETFramework\DbgJITDebugLaunchSetting are not set to auto-launch (again, see Enabling JIT-attach Debugging for more details on the settings).

a. Or you can delete them.

3. Create the following registry keys and values:

a. HKLM\Software\Microsoft\Windows\Windows Error Reporting\DebugApplications

i. Value: <Name of application executable> (e.g. “myapp.exe”)

1. Type: DWORD (32-bit)

2. Value data: 1

b. Repeat this for each application you want the debugger to be auto-launched.

You can set the DebugApplications key and values in HKCU if you prefer per-user control. When these settings are in effect the debugger will be launched for only the processes you specify and normal error handling will occur for all other processes (e.g. for most default configurations, a prompt to submit an error report to Microsoft).

The following sample .reg file will set cdb.exe to be auto-launched only for HelloWorld.exe. Replace HelloWorld.exe with the name(s) of the application(s) for which you’re interested in generating crash dumps.

Windows Registry Editor Version 5.00

;This reg file installs just-in-time debuggers to capture a dump of only the
;processes listed under the [DebugApplications] key, below.

;

;Assumes 32-bit debugger is cdb.exe and is installed to c:\debuggers\x86\.

;Assumes 64-bit debugger is cdb.exe and is installed to c:\debuggers\x64\.

;

;Assumes crash dumps can be written to c:\crash_dumps\.

;Make sure all users have write access to this directory.

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework]

"DbgManagedDebugger"="\"c:\\debuggers\\x64\\cdb.exe\" -pv -p %ld -c \".dump /u /ma c:\\crash_dumps\\crash.dmp;.kill;qd\""

"DbgJITDebugLaunchSetting"=dword:00000000

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug]

"Debugger"="\"c:\\debuggers\\x64\\cdb.exe\" -pv -p %ld -c \".dump /u /ma c:\\crash_dumps\\crash.dmp;.kill;qd\""

"Auto"="0"

;The following keys are only used on 64-bit versions of Windows (note Wow6432Node).

;They can be safely created with no side-effects on 32-bit versions of Windows.

;Alternatively, you can delete the remainder of this file if you’re running a

;32-bit version of Windows.

[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Windows NT\CurrentVersion\AeDebug]

"Debugger"="\"c:\\debuggers\\x86\\cdb.exe\" -pv -p %ld -c \".dump /u /ma c:\\crash_dumps\\crash.dmp;.kill;qd\""

"Auto"="0"

[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\.NETFramework]

"DbgManagedDebugger"="\"c:\\debuggers\\x86\\cdb.exe\" -pv -p %ld -c \".dump /u /ma c:\\crash_dumps\\crash.dmp;.kill;qd\""

"DbgJITDebugLaunchSetting"=dword:00000000

;For each application you want the debugger to be auto-launched, add a row below

;similar to “HelloWorld.exe"=dword:00000001 but replacing HelloWorld.exe with

;your application .exe name.

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\DebugApplications]

"HelloWorld.exe"=dword:00000001

Can I do more than just capture a dump?

More information on the command-line switches for the Windows Debuggers can be found in the documentation included with the Debugging Tools for Windows. You’re not limited just to dumps. You can perform quite a bit of debugger automation if you choose.

So what happened?

Now that you have your dump, it’s time to figure out why your application crashed. For those of you familiar with dump debugging, at this point you’re about to fire up WinDbg + SOS (the managed debugging extension) and dive in. But wait! If your application runs on V4 of the CLR (.Net Framework 4.0) you can debug the dump in Visual Studio 2010. Our desire for the experience in VS 2010 is that it feels very similar to stopped-state live debugging (like you’ve hit a breakpoint). The fact that you’re often debugging optimized code can make this a bit trickier, but that’s a topic for another day.

For more information about how to debug crashes in managed code, a search for “managed dump debugging” yields quite a few helpful results. A great place to start is Tess Ferrandez’s blog (a support engineer at Microsoft). She has a number of great posts on the topic, including a post on dump debugging in VS 2010, and some labs/walkthroughs as well.

Jon Langdon,

PM, CLR

Five Myths about Managed Code

Abhishek_Mondal — Sat, 03 Oct 2009 03:41:40 GMT

My name is Immo Landwerth and I was a Program Manager intern this year in the CLR team. In this blog post I am not going to showcase any of the fantastic features that will ship with .NET 4.0 – my colleagues in the CLR team know them much better and already did a fabulous job discussing them here, over there and on Channel 9.

Instead I want to discuss the following five myths about managed code and in particular about the CLR:

· Managed code is always JIT compiled

· Generic co- and contra variance are new in .NET 4.0

· Everything is an object

· .NET only supports statically typed languages

· Microsoft is not using Managed Code

Myth Five – Managed code is always JIT compiled

Having a JIT compiler has many advantages because a lot of things are becoming much easier when a JIT compiler is available:

1. On-the-fly code generation (System.Reflection.Emit) is much easier because you only have to target one virtual machine architecture (IL) instead all the processor architectures the runtime supports (such as x86 and x64).

2. To some degree it solves the fragile base class library problem. That means we can share class definitions across modules without having the problem that changes such as adding fields or adding virtual methods crashes dependent code.

3. The working set can be improved because the JIT only compiles methods that are actually executed.

4. Theoretically, you could take situational facts into consideration, such as which processor-architecture is actually used (e.g. is it SSE2 capable), the application usage patterns etc. and optimize differently for them.

However, JIT compilation also has downsides such as:

1. It takes time. That means JIT compilation always has to trade-off time vs. code quality.

2. The code is stored on private pages so the compiled code is not shared across processes.

Therefore we created a tool called NGEN that allows you to pre-create native images during the setup. You could call this ahead-of-time compilation (as opposed to just-in-time). Certain special conditions left aside (such as some hosting scenarios or profiling), the runtime will now pick up the native images instead of JIT-compiling the code.

Why did we not allow you to pre-create the native images during build time and let you ship the native images directly? Well, because we then run into the fragile base class library problem mentioned above. In that case, your native images would get invalid every time the .NET Framework is updated. Today we solve this problem by re-running NGEN on the customer’s machine when the framework is serviced. In .NET Framework 4 we ship a new feature called targeted patching, that allows us for method-body only changes to minimize or to even to fully avoid recompilation. For more details about NGEN in general see here and for more details about NGEN in .NET Framework 4 see here.

Even if you are not using NGEN for you application code: for desktop CLR applications all the assemblies that are part of the .NET Framework itself are not JIT compiled – instead the runtime will bind to the native images. So even in these cases only your application code will be JIT compiled and therefore both ahead-of-time as well as just-in-time technologies are used simultaneously. Thus, stating that all code is JITted is simply wrong.

Myth Four – Generic co- and contra variance are new in .NET 4.0

The short answer is ‘no’. The longer answer is ‘well, sort of’.

But I am getting ahead of myself. Let’s first see what co- and contravariance actually means. Generic covariance allows you to call a method that takes an IEnumerable<Shape> with an IEnumerable<Circle> (if Circle is derived from Shape). This is useful if Shape contains, e.g. a method that allows you to compute the area. This way you can write a method that computes the area for any collection of shapes. Contravariance on the other hand allows you to call a method that takes an IComparer<Circle> with an IComparer<Shape>. This is handy if someone wants to compare circles and you already have created a general comparer for any shape (this works because if your comparer knows how to compare two instances of Shape it certainly is also able to compare two instances of Circle).

The support for co- and contra variance has always been in the CLR since generics came up in the .NET Framework 2.0. However, as Rick Byers pointed out you would have to use ILASM for creating covariant and contravariant type definitions:

In IL, covariant type parameters are indicated by a ‘+’, and contravariant type parameters are indicated by a ‘-‘ (non-variant type parameters are the default, and can be used anywhere).

What has been added in the .NET 4.0 release is language support for C# and Visual Basic. For example, the following uses the C# syntax (in and out modifiers for the generic type declaration) to create some covariant and contra variant types:

// Covariant parameters can be used as result types

interface IEnumerator<out T> {

T Current { get; }

bool MoveNext();

}

// Covariant parameters can be used in covariant result types

interface IEnumerable<out T> {

IEnumerator<T> GetEnumerator();

}

// Contravariant parameters can be used as argument types

interface IComparer<in T> {

bool Compare(T x, T y);

}

Myth Three – Everything is an object

“Wait a minute – this is the number one programming promise everyone was making about .NET!” you might say now. Yes, and yet it is false. Many .NET or C# books make this mistake in one form or the other. “Everything is an object”. Although we believe there is a lot of value in simplifying things for didactic reasons (and hence many authors just claim it that way) we would like to take this opportunity to tell you “sorry, it is not completely true”.

Before we discuss this issue we should first define what the sentence “everything is an object” is supposed to mean. The interpretation we will use here is this:

Every type is derived from a single root (System.Object). This means, that every value can be implicitly casted to System.Object. More precisely, this means that every value is representable as an instance of System.Object.

So why is this not true for the CLR? The counter example is a whole class of types that are not derived from System.Object: pointers (such as int*). So you cannot pass a pointer to a method that takes an object. In addition you cannot call the ToString or GetHashCode methods on a pointer.

We could also use a different interpretation of “everything is an object” such as:

Every type is derived from a single root (System.Object). This means, that every value is an object at all times.

Why is this different? Simple values (i.e.. values that have types derived from System.ValueType) are not objects by the definition of an object (they lack identity). But every value can be casted implicitly to System.Object (because System.ValueType is derived from System.Object). However, in that case an object instance that contains the value is created. This process is called boxing. The resulting object instance (the “box”, not to be confused with Don Box) has indeed a notion of identity (which is in particular also true for Don Box).

As you can see, the CLR uses the first interpretation and yet it is still not completely true as pointers do not derive from System.Object.

Myth Two – .NET only supports statically typed languages

It is true that the CLR uses a static type system. But this does not necessarily mean that it is only suited for programming languages that use a static type system. At the end, the programming language is implemented using the CLR but it is not identical with the CLR. So do not be fooled by the fact that the type system and mechanics of C# almost map directly to first class CLR-concepts. In fact, there are many concepts in C# that the CLR is not aware of:

1. Namespaces. As far as the CLR is concerned namespaces do not even exist. They are just implemented as type prefixes separated by dots (so instead of saying ‘the class Console is contained in the namespace System’ the CLR would just say ‘there is a class called System.Console’).

2. Iterators. The CLR does not provide any support for it. All the magic is done by the compiler (if you want to know, the compiler turns your method into a new type that internally uses a state-machine to track the current point of execution. Details can be found here).

3. Lambdas. They are just syntactic sugar. For the runtime these are just delegates, which in turn can also be considered syntactic sugar. In fact, a delegate is nothing more than a class derived from System.MulticastDelegate that provides an Invoke, BeginInvoke and EndInvoke method with the appropriate parameters.

Please note that this list is not complete. Instead it is only used to show you that even C# has to implement itself on top of the CLR and hence it is not a 1:1 mapping of the concepts the runtime provides. What does this have to do with static typing vs. dynamic typing? The answer is simply: you can implement a dynamically typed system on top of a statically typed system.

If you know see a huge business opportunity here, we have to disappoint you. Some smart people already had the same idea. This effort is called the Dynamic Language Runtime, or DLR for short. If you are like me then you immediately think of native code when someone mentions the term ‘runtime’. However, the DLR is completely implemented in C# and is just a class library that can be used by programming languages to implement dynamic systems on top of the CLR. The DLR shares the fundamental design principle of the CLR, i.e. it provides a platform for more than one language. That means you can share IronPython objects with IronRuby objects because they are implemented with the same underlying runtime (the DLR).

With .NET 4.0 the DLR ships as part of the box. So while .NET has first-class support for statically typed languages through the CLR it also provides first-class support for dynamically typed languages through the DLR.

Myth One – Microsoft is not using Managed Code

We often hear this (“Office and Windows are still not built on top of managed code!”) when customers ask about performance and future investments of Microsoft in managed code. The reasoning goes like this:

Since Microsoft is not implementing Windows and Office in managed code that means that it must be significantly flawed/runs much slower than native code and therefore their long term strategy will still be C++. This in turn means that we should not use managed code either.

In fact Microsoft has a huge investment in managed code (although it is still true that Office and Windows are not implemented in managed code). However, there are a bunch of products that are significantly (if not completely) implemented in managed code:

1. Windows components, such as

a. PowerShell

b. System Center

2. Office components, such as

a. Exchange

b. SharePoint/Office Server

3. Developer Tools, such as

a. Visual Studio and Visual Studio Team System

b. Expression

4. Dynamics

This list if by far not complete but it should be large enough to convince you that we are in fact ‘eating our own dog food’.

The reason that not all products are written in managed is not only related to performance. Sometimes the wins of re-implementing working native code in managed code do not outweigh its costs. On the other hand, there are still scenarios in which managed code simply cannot be used today (such as building the CLR itself or the debugger).

However, we will not deny that there are scenarios in which we cannot compete with the performance of native code today. But this does not mean that we have given up on this. In fact, projects like Singularity should show you that we are really very ambitious about redefining the limits of the managed world.

The last thing to keep in mind is that manually optimized assembler code is also faster than plain C-code. But this does not mean that all operating systems are completely written in assembler.

Thus our vision is more like this: native code where it makes sense, managed code where it makes sense with the bigger portion being managed.

IL Stub Diagnostic Tool

Abhishek_Mondal — Thu, 27 Aug 2009 09:08:50 GMT

The IL Stub Diagnostic Tool enables real-time inspection of the contents of IL stubs. Developers now have a powerful tool to troubleshoot issues in interop marshalling,

Introduction

Jesse posted a great blog talking about the concept, history, and improvements of Intermediate Language (IL) stubs for CLR v4. He mentions

The remainder of this post continues the discussion that Jesse started in the “Easier Debugging” section of his blog.

Get the tool

You are encouraged to download the tool from CodePlex.

Usage of the IL Stubs Diagnostics Tool

To begin using the tool, follow these basic steps:

1) Launch the IL Stub Diagnostic tool. Note: you need to be an administrator of the machine because the tool utilizes the Event Tracing for Windows (ETW) feature to receive ETW events.

There are three key features of the tool:

· Starting and Controlling the IL stubs monitoring session

The upper-left button, Start, is the most important aspect of starting a session. After clicking the Start button, the tool begins monitoring all the stubs generated by CLR on the machine. It will keep monitoring the IL stubs until you click it again (i.e., Start button will turn to Stop after it is clicked).

· Filtering to find a specific IL Stub

The IL Stub List section shows all the IL stubs collected by the tool. Sometimes you will receive a huge number of stubs. In this case, you can set several filters using the Filters feature to find the exact stub.

· Navigating and Inspecting the contents of an IL Stub

Once you find the IL Stub, the contents of it (the IL code) will be shown in the IL Code section. We provide some features like navigation to help you read and understand the IL code.

2) Start monitoring IL stubs by clicking the Start button.

3) Run a program involving interop. For example, the sample program sample\pinvoke.exe, which is included with the tool on CodePlex, generates a p/invoke from managed code to native code. Running this sample from the command line will generate several entries in the IL Stub List section. We will use this for illustration purposes below.

In this example, each item represents an IL Stub. Select an entry and notice its corresponding IL code shown in the IL Code Section.

Features which Enable Inspection and Navigation of IL Stubs

IL Stubs are generated on the fly by the CLR for Interop, and bridge the gap between managed and native methods. Since they are dynamically generated, it is difficult for developers to investigate or troubleshoot issues because the stubs are further JIT-compiled into machine code. The IL Stub Diagnostic tool eases the interop debugging experience by enabling the inspection of every stub. Let’s take a look at some of the key features which enable the inspection and navigation of IL stubs.

Detailed information of IL Stub

By default, the tool shows the following categories for each IL stub: MethodName, CreateTime, ProcessName, Category, ManagedSignature and NativeSignature. These categories are important for developers to identify an IL stub. In special cases (e.g., debugging with windbg), developers may also be interested in information like ModuleId and ManagedMethodToken. These are available by right clicking the table header of the IL Stub list.

IL Stub Filters

You may find that the CLR generates a lot of IL Stubs during the execution of a managed program. To quickly locate the stub we’re seeking, filters can be set to reduce the number of IL stub entries in the IL Stub list. Using the pinvoke.exe sample (included with the tool on CodePlex), you can use the following filter:

Once the filter has been entered, click the Refresh button to obtain the filtered list.

To remove a filter, you can right click on the row header of the first filter condition and select Delete Filter. It will take effect after clicking Refresh button.

Navigation buttons for IL Stub code

The code may not be well-formatted for first time display. To make the code more reader-friendly, the tool provides navigation buttons that allow you to view the code block by block. As you can see, each IL stub contains the following color-coded blocks: Initialize, Marshal, the calling method, and Unmarshal blocks. In the IL code, StubHelpers are used to do some complex work. Most of the functionality of each stub helper can be figured out by its name. The tool also provides a Stub API button to look up its usage.

Please note that stub helpers are internal to the runtime and their names/signatures/description should only be used for informational purposes. We may change any of these in the next version of the CLR so no one should establish any dependencies on them.

Troubleshooting marshalling issues with IL Stub Diagnostic tool

As mentioned earlier, developers can use the tool to investigate and debug marshalling issues. Here is an example.

Case description

There is an issue with the following p/invoke scenario. The signature of the native method is:

void DumpString(WCHAR *pStr)

{

printf("%ws", pStr);

}

In managed code, we declare it as:

[DllImport("testNative.dll")]

public static extern void DumpString(string str);

When invoking from managed side with the following code, it shows nothing on the console.

DumpString("Hello World");

Our job is to discover what happens during the p/invoke call.

Troubleshooting

Since the native code simply dumps the string, the problem must be occuring during the marshaling phase.

First, locate the stub by following these steps:

1. Launch IL Stub Diagnostic tool.

2. Click Start Button.

3. Run the program again.

4. Set the following filter as shown below and click the Refresh button.

Next, navigate to the block of code which handles the marshaling by following these steps:

1. Click the “IL Code” button to get a bigger view.

2. Click the “Next Block” and navigate to the Marshal block.

3. Usually, StubHelpers APIs are the most informational code here. Move the mouse over it to see the explanation of the method (shown in the green box below):

4. The description tells us that the StubHelpers method is used to convert an Ansi string. However, our native method expects a WCHAR string, which results in the marshaling issue. The fix is to specify Unicode, as the highlighted code below.

[DllImport("testNative.dll", CharSet=CharSet.Unicode)]

public static extern void DumpString(string str);

Summary

As you can see, the IL Stubs Diagnostic Tool provides a window into what used to be a black box, and allows you to look into the very core of interop marshaling – the IL Stub itself.

Acknowledgements

We would like to specially thank Yifeng Fu for being one of the key developers of IL Stub Diagnostic tool during his internship with us.

See Also

The MSDN topic “MSIL Stub Customization” contains detailed reference and usage information.

Yongtai Zhu,

Developer, CLR.

The good and the bad of exception filters

Abhishek_Mondal — Wed, 26 Aug 2009 03:04:00 GMT

Every so often we get asked questions about the CLR’s support for exception filters. Why do some languages support them and others do not? Is it a good idea to add them to my new language? When should I use a filter vs. catch/rethrow? Etc. I’ll try to answer some of these questions for you here, and while I won’t go into all of them hopefully you’ll walk away with enough info to form your own opinion on the rest. Like so many things there’s good things and bad things about exception filters…

So what’s a filter?

The CLR provides a number of exception handling primitives that higher level languages can build upon. Some are fairly obvious, and map readily to language constructs that most of us know and love: try/catch and try/finally, for instance. I’d hazard to guess that everyone knows what those do, but just in case, let’s consider a quick example in C#:

try

{

try

{

Console.Write(“1”);

if (P) throw new ArgumentException();

}

finally

{

Console.Write(“2”);

}

catch (ArgumentException e)

{

Console.Write(“3”);

}

Console.Write(“4”);

If P is true, then that will print out “1234”, of course. If P is false, then it will print “124”. Groovy.

But the CLR also provides two more EH primitives: fault and filter. A fault clause is much like a finally clause; it runs when an exception escapes its associated try block. The difference is that a finally clause also runs when control leaves the try block normally, whereas the fault clause will only run when control leaves the try block due to an exception. In the case above, if we replaced the “finally” with “fault” (there’s no C# syntax for that, but suspend your disbelief for a moment) then it would print “1234” if P is true, and “14” is P is false. See the difference? Most languages don’t expose this as a first-class language construct, but we do have a few that use fault clauses under the covers for specific scenarios.

So that leaves us with filters. I suppose the simplest definition of a filter is that it is a construct that allows one to build a conditional catch clause. In fact, that’s exactly what VB uses filters for. Let’s consider a more complicated example in VB:

Function Foo() As Boolean

Console.Write("3")

Return True

End Function

Sub Main()

Dim P As Boolean = True

Try

Console.Write("1")

If (P) Then

Throw New ArgumentNullException()

End If

Console.Write(“P was False!”)

Finally

Console.Write("2")

End Try

Catch ex As ArgumentNullException When Foo()

Console.Write("4")

Catch ex As ArgumentException

Console.Write("5")

End Try

Console.Write("6")

End Sub

Here you’ll note the “Catch ex As ArgumentNullException When Foo()” line is a conditional catch statement. The catch handler will only execute and print “4” when the exception is an ArgumentNullException and when Foo() returns true. If Foo() returns false, then the catch clause doesn’t execute, and the system continues to search for a catch clause that can handle the exception. In this case, the very next clause would handle the exception, and print “5”.

So, what do you think this program prints? Don’t cheat by attempting to compile and run it! Using what you know about exception handling and looking at the program structure and syntax, what would you imagine this program prints? I suspect most people would guess “12346”. I even gave you a clue with the numbering.

I think most of us would look at the example above and conclude that the result should be “12346” because when we look at the syntax above we, quite rightly, see lexically scoped language constructs. We expect that when the code in the inner finally clause starts executing that no more code anywhere in the associated try block will execute. For instance, in the example above, if P is true then when we enter the Finally we know that no more code will execute in the try block, and that we’ll never print “P was False!”. Likewise, when we evaluate one of the catch clauses, we expect that all the code in the associated try block is done executing.

And here comes the bad…

It turns out that program actually prints “13246”. My clue with the numbering was an evil ruse. After the throw, Foo() is executed first as part of evaluating the first catch clause, and then the finally within the associated try block executes. And that’s just freaky… what happened to our lexically scoped language constructs?!

This is a surprising result for most. It breaks our intuitive reasoning about the language based on the lexically scope exception handling constructs provided. Here, when we evaluate the conditional catch clause, all the code in the associated try block has not, in fact, finished executing.

Why does this happen?

The reason we see “3” before “2” is subtle, and founded in the CLR’s implementation of exception handling. The CLR’s exception handling model is actually a “two pass” model. When an exception is thrown, the runtime searches the call stack for a handler for the exception. The goal of the first pass is to simply determine if a handler for the exception is present on the stack. If it sees finally (or fault) clauses, it ignores them for a moment.

The handler may be in the form of a typed catch clause, i.e., “Catch ex as ArgumentException)”. When the runtime sees a typed catch clause, it can determine itself if this clause will handle the exception by performing a simple check to determine if the exception object is of a type that inherits from (or is) the type in the clause.

But when the runtime sees a filter, it must execute the filter in order to determine if the associated handler will handle the exception. If the filter evaluates to true, then a handler has been found. If it evaluates to false, then the runtime will keep searching for a handler.

Once a handler has been found, the first pass is over and the second pass begins. On the second pass, the runtime again runs the call stack from the point of throw, but this time it executes all finally (or fault) clauses it finds on the way to the handler it identified on the first pass. When the handler is reached, it is executed, and the exception has finally been handled.

But why is that bad?

“Okay”, you say, “I get it. I understand that filters run during the first pass. I can deal with that… what’s the big deal?” Well, let’s first consider what a finally clause is for. We typically use finally clauses to ensure that our program state remains consistent when exiting a function even in the face of an exception. We put back temporarily broken invariants. Consider that the C# “using” statement is built using try/finally, and then consider all the things one might do with that.

But when your filter runs, none of those finally clauses have executed. If you called into a library within the associated try block, you may have not actually completed the call when your filter executes. Can you call back into the same library in that case? I don’t know. It might work. Or it might yield an assert, or an exception, or, well, your guess is as good as mine. The point is that you can’t tell.

Using filters wisely (or, “the good”)

But the notion of a conditional catch clause really is quite appealing, and there are ways to use these without getting caught by the problems of when the filter actually executes. The key is to only read information from either the exception object itself, or from immutable global state, and to not change any global state. If you limit your actions in a filter to just those, then it doesn’t matter when the filter runs, and no one will be able to tell that the filter ran out of order.

For instance, if you have a fairly general exception, like COMException, you typically only want to catch that when it represents a certain HRESULT. For instance, you want to let it go unhanded when it represents E_FAIL, but you want to catch it when it represents E_ACCESSDEINED because you have an alternative for that case. Here, this is a perfectly reasonable conditional catch clause:

Catch ex As System.Runtime.InteropServices.COMException When ex.ErrorCode() = &H80070005

The alternative is to place the condition within the catch block, and rethrow the exception if it doesn’t meet your criteria. For instance:

Catch ex As System.Runtime.InteropServices.COMException

If (ex.ErrorCode != &H80070005) Then Throw

Logically, this “catch/rethrow” pattern does the same thing as the filter did, but there is a subtle and important difference. If the exception goes unhandled, then the program state is quite different between the two. In the catch/rethrow case, the unhandled exception will appear to come from the Throw statement within the catch block. There will be no call stack beyond that, and any finally blocks up to the catch clause will have been executed. Both make debugging more difficult. In the filter case, the exception goes unhandled from the point of the original throw, and no program state has been changed by finally clauses.

The problem is that we rely on programmer discipline in order to use filters correctly, but it’s easy to use them incorrectly and end up with infrequently executed code (exceptions, after all, are for exceptional circumstances) that has subtle and hard to diagnose bugs due to inconsistent program state that should have been cleaned up by finally clauses further down the stack.

Why does the CLR use a two-pass exception handling model?

The CLR implements a two-pass exception handling system in order to better interoperate with unmanaged exception handling systems, like Win32 Structured Exception Handling (SEH) or C++ Exception Handling. We must run finally (and fault) clauses on the second pass so they run in order with unmanaged equivalents. Likewise, we must not execute filters later (say, on the second pass) because one of those unmanaged systems may have remembered that it was supposed to be responsible for handling the exception. If we were to run a filter late on the second pass and decide that a managed clause really should catch the exception after previously having not declared that on the first pass, then we would violate our contract with those unmanaged mechanisms with unpredictable results.

So, in short, it’s for interop, and like many things involving interop, we have a compatibility burden that we can’t ignore.

Would a one-pass model be better?

Many have wondered over the years if perhaps the two-pass model in general is bad, and if a one-pass model would be better. Like so many things in the world, it’s just not that clear. A one-pass model would simplify the exception handling implementation, and it would make more sense in the cases shown above. However, there are advantages to the two-pass model that can’t be ignored. Perhaps the most important one is that if the search for a handler fails on the first-pass the exception goes unhandled and in general no program state has changed, even though filters are run since filters tend not to change things. The call stack is still intact, and all values that lead to the exception are still present on the stack and on the heap (assuming no race conditions.) This is frequently essential when debugging an unhandled exception. In a one-pass model, all of the finally clauses would have been run before the exception goes unhandled.

Wrapping up

Of the languages that MS ships, only VB and F# support filters, and both do so via conditional catch clauses. In F#, you have to really go out of your way to attempt to inspect mutable global state, or to actually have a side effect, so you’re fairly safe there. In VB, though, you can call a function from the “when” clause of their catch statement, and in there you have free reign to do whatever you please. You can, without a doubt, get yourself into trouble attempting to do too much complicated work within such a filter. To keep your world safe and simple try to limit yourself to expressions that only access the exception object, and don’t modify anything. If you go beyond that, you need to consider carefully all the code executed in the try block, and if your actions in the filter will work if the backout code below has not finished executing yet.

Mike Magruder,

Developer, CLR.

Improvements to Interop Marshaling in V4: IL Stubs Everywhere

Abhishek_Mondal — Mon, 10 Aug 2009 23:27:15 GMT

When the CLR needs to transition between managed and native code – usually because of P\Invoke or COM interop – we need to generate marshaling stubs (little chunks of code) to handle that specific call and transform the data from managed to native format and back again. . These stubs are little pieces of code that are usually generated at runtime to do their work behind the scenes and, hopefully, without making developers or users aware of their presence. In .Net 4 we’ve improved our marshaling infrastructure substantially with a feature we call “IL Stubs Everywhere.” For those who have a deep interest in interop marshaling we get into the details of this feature below. For everyone else, here are a few key benefits of this feature:

· Faster interop marshaling: the more complex the signature the greater the speed-up

· x86 and x64 behavior matches: we’ve updated the x64 marshaling to behave exactly as x86 always has and mostly without impact to x64

· Better debugging: when something goes wrong in marshaling we now give you the ability, and specialized tools, to find the problem

History

The 1.0 and 1.1 versions of the CLR had several different techniques for creating and executing these stubs that were each designed for marshaling different types of signatures. These techniques ranged from directly generated x86 assembly instructions for simple signatures to generating specialized ML (an internal marshaling language)* and running them through an internal interpreter for the most complicated signatures. This system worked well enough – although not without difficulties – in 1.0 and 1.1 but presented us with a serious maintenance problem when 2.0, and its support for multiple processor architectures, came around.

We realized early in the process of adding 64 bit support to 2.0 that this approach was not sustainable across multiple architectures. Had we continued with the same strategy we would have had to create parallel marshaling infrastructures for each new architecture we supported (remember in 2.0 we introduced support for both x64 and IA64) which would, in addition to the initial cost, at least triple the cost of every new marshaling feature or bug fix. We needed one marshaling stub technology that would work on multiple processor architectures and could be efficiently executed on each one: enter IL stubs.

With x64 and IA64 bit support in v2 we introduced IL stubs to perform the marshaling on these platforms. With IL stubs we generate actual IL rather than assembly code or the specialized language in our ML stubs. On these platforms we only used the IL stubs and were thus able to have a single implementation, for all stubs, across both processor architectures. These IL stubs were much faster than the ML stubs but were also much slower than x86 stubs. Because of the performance regress and our resource constraints, we left the x86 platform with its original marshaling stubs implementation.

New in V4: IL Stubs Everywhere

One of the largest features the interop team worked on in the v4 product was the “IL Stubs Everywhere” feature. With this work we moved to uniformly using the same IL stubs infrastructure for all marshaling on all platforms. Sometimes when we spend a lot of time developing, or reworking, a feature there is a concern that it will be expensive for developers to start taking advantage of it: the good news with this one is that this will change will occur automatically and the majority of the benefits will happen just by running your app on v4.0.

A few of the benefits you will see with this feature are:

Increased performance:

The original x86 stubs for the simplest signatures were pretty fast – often a simple copy operation -- but as soon as you needed marshaling for more complex signatures you were forced onto the slow path with the ML interpreter. We heavily optimized our IL stub infrastructure and now IL stubs are faster across the board (even compared to x86 stubs) and are orders of magnitude faster than the interpreted stubs you used to get for the more interesting signatures. Additionally, for those that use NGEN you’ll find that we include most of your IL stubs in the NGEN images to improve this scenario even further.

Consistent behavior:

One of the common problems people ran into in interop in the v2 days is that with different marshaling technologies used for 32 and 64 bit the behaviors didn’t line up exactly the same. There were a few places where the same code behaved differently based on platform. With every platform running off of IL stubs we now have the same behavior across all platforms.

Note: when we did this we had a choice between giving the 4.0 IL stubs the behavior of the v2.0 64 bit IL stubs or the v2.0 32 bit stubs. We chose to use the 2.0 32 bit behavior for the 4.0 because of the much larger percentage of interop apps that have been built on 32bit to date.

This means that you should not see any behavioral differences moving a 32 bit app to the v4.0 NetFX but you may see some for 64 bit interop apps. Because of the nature of these changes we expect the impact of them to 64 bit apps to be very minimal: most of the changes involve relaxing behavior that was stricter in 64 bit than 32 bit. The biggest change in reverting to 32 bit behavior was to stop clearing ‘out’ parameters during certain error conditions and should be completely non-breaking to 64 bit applications.

Easier Debugging:

One of the other problems with the old 32 bit marshaling infrastructure was that it worked essentially as a black box that was very difficult to debug if something went wrong. It was very difficult to figure out how/what the runtime was trying to marshal when something went wrong. By moving to IL stubs it is actually possible to see the IL instructions that we’re executing and determine where your problem is.

Since Visual Studio does not support debugging IL, the move to IL stubs alone wouldn’t help debugging very much if that was all we did. Instead we updated our IL stubs infrastructure to fire an ETW event detailing each stub as they get generated and built a tool, which we recently released on codeplex, which lets you easily see all the stubs being generated and quickly search for the ones for the methods you are most interested in. In an upcoming post we will have more details and a walkthrough of the tool, but in the meantime you can find it here: http://clrinterop.codeplex.com/Release/ProjectReleases.aspx?ReleaseId=29745

*ML Stubs: it is a little known fact that up until v4 the CLR, at runtime, converted complex signatures into an internal CLR language called ML (marshaling language) which it then ran through the CLR’s internal ML interpreter each time the method was invoked. This little interpreter is almost another separate runtime just for interop marshaling. You can imagine how replacing this with IL and thus getting all the advantages of the perf work that goes into NGEN and the JIT allowed us to massively speed up these scenarios. There was much rejoicing on our team when we removed this interpreter in v4.

Jesse Kaplan,

PM, CLR

AlwaysOn support in SqlClient	New 4.0.2 feature
SQL Server Express Local Database Runtime support in SqlClient	New 4.0.2 feature
StateMachine (including designer support)	Existing 4.0.1 feature
SqlWorkflowInstanceStore on SQL Azure	Existing 4.0.1 feature
Compensation for custom hosts	Existing 4.0.1 feature