BCL Team Blog

MEF and TPL Dataflow NuGet Packages for .NET Framework 4.5 RC [Nick]

BCL Team — Wed, 30 May 2012 16:09:00 GMT

If you’ve installed the .NET Framework 4.5 RC, you may have noticed some differences between this prerelease and the .NET Framework 4.5 Beta. One of the changes we made was to move two libraries to an independent distribution model via NuGet.

Many of you sent us feedback about the frustration of waiting for a full product cycle to get small changes. By changing the distribution model, we are taking our first steps toward meeting this challenge. We expect this will enable us to invest in these libraries and deliver improvements to you faster.

The BCL is broad and varied, and not every library in it has the same lifecycle requirements. NuGet today provides an alternative to the full .NET Framework redistributable, and we want to take advantage of it where the model is a good fit.

The packages we are releasing on NuGet – TPL Dataflow and MEF for Windows Metro style apps – were included in the base installation of the .NET Framework 4.5 Beta, but are now available as NuGet packages. These packages are fully supported, first-class members of the BCL and offer additional functionality over their .NET Framework 4.5 Beta versions.

TPL Dataflow

TPL Dataflow, which is part of the Task Parallel Library, is an exciting new library for building highly concurrent systems based on actor/agent-oriented techniques.

TPL Dataflow was implemented in the assembly System.Threading.Tasks.Dataflow.dll in the .NET Framework 4.5 Beta and was not available for use by Windows Metro style apps. In the .NET Framework 4.5 RC, TPL Dataflow is delivered in the Microsoft.Tpl.Dataflow NuGet package and is supported everywhere, including desktop, server and .NET for Metro style apps.

The simplest way to install the package in Visual Studio is to highlight the target project, choose Tools > Library Package Manager > Package Manager Console, and at the prompt type:

Install-Package Microsoft.Tpl.Dataflow -Pre

The -Pre flag is needed because these packages are marked as prerelease versions. You can use the GUI Manage NuGet Packages dialog box for this task as well, but when searching make sure to select Include Prerelease from the appropriate drop-down list:

MEF for Metro style apps

The Managed Extensibility Framework (MEF), which was introduced in the .NET Framework 4, provides a “part model” for application extensibility and to keep applications flexible as they grow in size and complexity.

In the .NET Framework 4.5 Beta, Windows Metro style apps could use a subset of MEF functionality. Feedback on this approach led us to conclude that Metro style apps have different requirements of MEF than desktop applications do. For instance, extensibility by third parties is not a focus of Metro style apps. Instead, MEF use focuses on application decoupling scenarios in conjunction with patterns like Model-View-ViewModel (MVVM). By releasing MEF for Metro style apps as a NuGet package, we are enabling the .NET Framework to better adapt to the requirements of this application model.

The package ID for MEF in Metro style apps is Microsoft.Composition. To install the package from the Package Manager Console, type:

Install-Package Microsoft.Composition -Pre

You can alternatively install the package from the Manage NuGet Packages dialog box.

MEF for Metro style apps is built on the lightweight composition engine previously previewed for use with ASP.NET MVC. This composition engine enables us to tailor the experience better for Metro style apps, but also means that this release is not fully backward-compatible with the beta release. Documentation of the differences between the Microsoft.Composition package and the version of MEF in the full .NET Framework can be found here.

MEF is still included and supported in the .NET Framework 4.5 RC. Furthermore, we are continuing our development of MEF as an open source project. These changes affect Metro style apps only.

Working with NuGet

NuGet is a developer-deployed, developer-serviced distribution mechanism. This means that applications carry their own versions of NuGet-packaged assemblies. This greatly simplifies side-by-side deployment and often allows you to use new versions of libraries without needing a .NET Framework update on your computer.

Developers need to update and redeploy an application to take advantage of a new version of a NuGet package: new versions won’t be pushed out automatically with .NET Framework updates. This model doesn’t suit every kind of library, but we’re confident that for MEF and for TPL Dataflow this is the right approach.

NuGet is integrated into Visual Studio 2012 RC. The package repository (including Microsoft and third-party packages) can be viewed at http://nuget.org. To use NuGet packages in a disconnected environment, you can set up a local file share as a package repository using the Package Sources page of the Visual Studio Options dialog:

Final thoughts

We’re very excited to be opening up new channels to deliver BCL functionality. We are deeply grateful to our early adopters and we've considered this direction carefully, although we realize that making this change in RC involves additional work for some of you.

We expect that our approach will evolve with the help of your feedback and experiences. To provide feedback on the TPL Dataflow release, please visit our UserVoice site. For MEF feedback, the MEF CodePlex project discussion forum is the best place to engage with our team.

Announcing Portable Library Tools 2 Beta for Visual Studio 2010 [Mircea]

BCL Team — Wed, 09 May 2012 18:07:00 GMT

Today we are announcing the Beta release of the Portable Library Tools 2 for Visual Studio 2010. This is an update to the first version we released in June 2011. It brings parity with Portable Library Tools in Visual Studio 11 Beta for those platforms supported by Visual Studio 2010. It also adds support for building portable libraries targeting more recent platforms, scoped to those features supported by Visual Studio 2010.

Portable Library Tools allow you to create C# and Visual Basic libraries that may be reused, without recompilation, from .NET projects for a variety of different platforms.

Download the Portable Library Tools Beta 2 today! (requires Visual Studio 2010 SP1)

What’s new?

New Supported APIs

Since the release of the initial CTP, we have received numerous requests for more API feature areas to be supported by the Portable Library Tools. In response to your feedback, we have been able to add support for a number of new feature areas for this release. Here is a summary of what Beta 2 supports, highlighting the additions:

Feature Area	Silverlight		Windows Phone		Metro Style Apps	.NET Framework			XBOX 360
	4	5	7	7.1		4	4.0.3	4.5
Core BCL	√	√	√	√	√	√	√	√	√
Core XML	√	√	√	√	√	√	√	√	√
LINQ	√	√	√	√	√	√	√	√
IQueryable	√	√		√	√	√	√	√
dynamic keyword support	√	√			√		√	√
Core WCF	√	√	√	√	√	√	√	√
Core Networking	√	√	√	√	√	√	√	√
View Models	√	√	√	√	√			√
Data Annotations	√	√			√		√	√
XLINQ	√	√	√	√	√		√	√	√
MEF	√	√			√	√	√	√
Data Contract Serialization	√	√	√	√	√	√	√	√
XML Serialization	√	√	√	√	√	√	√	√
JSon Serialization	√	√	√	√	√	√	√	√
System.Numerics	√	√			√	√	√	√

User interface changes

To accommodate for the increased number of platforms and versions supported, we have updated the target platform selection dialog.

The workflow for creating a Portable Library changed slightly, as well. As before, you create a new portable library project by selecting the “Portable Library” template:

However, upon clicking the OK button, you are now prompted with a dialog to select the platforms to target:

The check boxes allow you to enable or disable support for a given platform, while the dropdowns allow you to select the minimum version of that platform you wish to support.

The platform selection dialog is still available In the project property pane after the project is created by clicking the “Change…” button:

In conclusion...

Please let us know what you think - we’d love to hear your feedback!

Async Targeting Pack for Visual Studio 11 now available for .NET 4 and Silverlight 5 [Greg]

BCL Team — Fri, 27 Apr 2012 00:23:00 GMT

Async Targeting Pack for Visual Studio 11 now available for .NET 4 and Silverlight 5

We would like to announce that you can now download an Async Targeting Pack for Visual Studio 11 that lets you use Visual Studio 11 Beta to create projects that target .NET 4.0 and Silverlight 5.

The Async Targeting Pack includes DLLs that address an incompatibility between Visual Studio 11 Beta compilers and the DLLs that shipped with the Async CTP for Visual Studio 2010. This issue has been previously discussed on the PFX team blog.

The Async Targeting Pack is available as a NuGet package, with installation details available here.

Enjoy!

Multi-Targeting Guidelines for Tools for Managed Code [Mircea]

BCL Team — Thu, 12 Apr 2012 00:58:00 GMT

This article provides guidelines for using and developing compilers, metadata readers, build extensions, and other tools that work with .NET Framework metadata in multi-targeting scenarios. Multi-targeting is the ability to target a specific combination of .NET Framework versions and platforms, or to target a subset of the .NET Framework (such as the Portable Class Library) that is common to a set of version/platform combinations.

Tools are often written in managed code. For multi-targeting scenarios, this means they must run on one version of the .NET Framework while processing binaries (compiling, building, reading metadata, manipulating metadata, and so on) for another version. For example, MSBuild is a managed tool, and it may run on the .NET Framework 4.5 while building managed binaries for Windows Phone 7.5.

Multi-targeting packs

A multi-targeting pack, or MT pack, is a set of reference assemblies that corresponds to a particular .NET Framework platform and version. A reference assembly is a .NET Framework assembly that typically has no method bodies and no internal or private APIs. Reference assemblies contain just the information a compiler needs.

For example, there are multi-targeting packs for the .NET Framework 3.5, the .NET Framework 4, Update 4.0.x for the .NET Framework 4, the .NET Framework 4.5, Windows Phone 7.5, Windows Metro style apps, Portable Class Library, and so on.

Visual Studio installs a set of multi-targeting packs, and so do SDKs such as the Visual Studio SDK, the Silverlight SDK, and the Windows Phone SDK. Typically, multi-targeting packs are installed under "%ProgramFiles(x86)%\Reference Assemblies\Microsoft\Framework" ("%ProgramFiles%\Reference Assemblies\Microsoft\Framework" on a 32-bit computer). For more information about multi-targeting packs, including other packs that are available, see .NET Multi-Targeting Packs on the MSDN Dev Center.

Guidelines for Using Compilers

DO specify explicitly every reference assembly that is required for compilation, including the core library. The core library is the assembly that defines the primitive types, such as System.Object and System.Int32, that get special handling in metadata. Originally the core library was mscorlib.dll. In more recent versions, it's System.Runtime.dll.

By default, the C# and Visual Basic compilers use the reference assemblies of the runtime version they shipped with. These assemblies are located in one of the version folders under %FrameworkDir%. We recommend that you avoid this default behavior, even if you're compiling for the default runtime version. For example, installing a .NET Framework service pack makes that service pack the new default version, which may not be what you intended.

DO NOT mix and match reference assemblies from different multi-targeting packs; this is not supported. Mixing assemblies can occur by accident, for example, by forgetting to specify /nostdlib when you compile a C# or Visual Basic program (as discussed in the next guideline), or by failing to explicitly specify a reference assembly.

DO use the necessary multi-targeting compiler options.

For both C# and Visual Basic, specify the /nostdlib option. Otherwise, the compiler uses the core library of the runtime the compiler shipped with, even if you use the /reference option to specify a different version.
For C#, additionally specify the /noconfig option to prevent the compiler from using the csc.rsp file, which references all the assemblies that ship with the .NET Framework. Using the csc.rsp file means that if you forget to explicitly specify a reference assembly, you will not get a warning from the compiler; instead, you will reference the wrong version of the reference assembly. (This option is not required by Visual Basic, because the Visual Basic /nostdlib compiler option suppresses vbc.rsp.)
For Visual Basic, specify the path to the multi-targeting pack folder, by using the /sdkpath option.
For both Visual Basic and C#, use the /reference option (short form /r) to specify the necessary reference assemblies. For C#, you must specify the exact paths. For Visual Basic, you only need to specify the assembly; the path is provided by the /sdkpath option.
If you use the Code Document Object Model (CodeDOM), set the CompilerParameters.CoreAssemblyFileName property to the full path to the reference assemblies. The Visual Basic and C# CodeDOM providers will enable the correct compiler settings as shown above. For other compiler providers, check with the vendor to ensure that the property is correctly honored.

Other compilers may have other command-line options for enabling correct multi-targeting. Check with your compiler vendor for details.

Examples:

Csc.exe /target:library /nostdlib /noconfig /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.5\mscorlib.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.5\system.dll" mytest.cs

Vbc.exe /target:library /nostdlib /sdkpath:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.5" /r:system.dll mytest.vb

Note that, in the VB case, you don't need to pass /r:mscorlib.dll. The VB compiler will pick it from the sdk path.

Guidelines for authoring tools

DO NOT assume a specific name (such as mscorlib.dll) for the core library.

As mentioned earlier, the core library is the assembly that defines primitive types such as System.Object and System.Int32. With multi-targeting packs, you cannot assume a name for the core library. In the .NET Framework 4.5, the core library is mscorlib.dll when compiling for the desktop, but System.Runtime.dll when compiling a Windows Metro style app. Microsoft doesn't guarantee a particular name for this library in the future.

We recommend that you provide your tool with an option that enables the user to specify the core library, or that you use name-based heuristics to identify the core library. (For example, your tool might assume that the first reference assembly that defines System.Object is the core library.)

DO handle and follow type forwarding links.

Microsoft may change the location of types, and preserve run-time and compile-time compatibility by using type forwarders at run time or as part of multi-targeting packs. Therefore, a compiler or a tool that uses metadata must be able to handle type forwarders.

For example, consider the compilation of MyApp.cs for the .NET Framework 4.5. MyApp.cs uses two third-party libraries: UtilityA.dll, which is compiled with the .NET Framework 4.5, and UtilityB.dll, which is compiled with the .NET Framework 4. MyApp and the two libraries exchange objects that implement the ICommand interface. That is, UtilityA.dll might define a type that implements ICommand, while MyApp and UtilityB.dll might have methods with parameters of type ICommand.

In the .NET Framework 4, ICommand was defined in PresentationCore.dll, but it was moved to System.dll in the .NET Framework 4.5. Therefore, UtilityB.dll references ICommand from PresentationCore.dll, while UtilityA.dll references it from System.dll. This is shown in the following illustration.

The thin arrows are references. MyApp has references to ICommand in System.dll, and references to various APIs in UtilityA.dll and UtilityB.dll. UtilityA.dll and UtilityB.dll have references to ICommand in the reference assemblies that correspond to the assemblies they were compiled to use. The thick arrow in the multi-targeting pack depicts a type forwarder.

The compiler uses the reference assemblies in the .NET Framework 4.5 multi-targeting pack because MyApp.cs targets the .NET Framework 4.5. When the compiler encounters an API in UtilityB.dll that references ICommand from PresentationCore.dll, the compiler discovers the type forwarder in the reference assembly for PresentationCore.dll and follows it to the definition of the type in System.dll. When the compiler compares the references to ICommand in the APIs from UtilityA.dll and UtilityB.dll, it resolves them to the same type, and compilation succeeds.

DO create assembly references to the location of the type definition, regardless of type forwarders.

In the previous example, the compiled MyApp.exe references System.dll as the location of ICommand because System.dll contains the definition of the type in the specified multi-targeting pack. PresentationCore.dll contains only a forwarding link.

In fact, if MyApp.cs doesn't use any types from PresentationCore.dll, MyApp.exe won't have an assembly reference entry for PresentationCore.dll in its manifest.

DO NOT design your tool to require the core library (or any other assembly) to contain a particular type. A compiler or other tool should not search for a type unless the code or metadata it is processing explicitly refers to that type.

For example, a compiler should not try to find System.Char if a program it compiles does not use this type. Similarly, a compiler should not try to find the LINQ infrastructure (like the Enumerable extension methods) unless the program it compiles is using LINQ. (If the program refers explicitly to the Enumerable extension methods, which are part of LINQ, search for the methods the program actually uses.)

DO NOT design language features that assume that APIs are located in a particular assembly. Use naming patterns instead.

For example, the extension method mechanism in C# only requires a method to be static and marked with the System.Runtime.CompilerServicing.ExtensionAttribute attribute. It doesn’t make assumptions about where that attribute is defined. Similarly, async support requires the presence of certain members and signature patterns, but doesn't make assumptions about where those members are defined.

Introducing MEF Lightweight Composition and an Updated Composition Provider for ASP.NET MVC [Nick]

BCL Team — Fri, 02 Mar 2012 22:57:00 GMT

With the second major release of MEF the team has significantly streamlined support for web applications. In November we previewed the new Composition Provider for ASP.NET MVC, and we’ve had great feedback supporting this direction. In this post we share some exciting new progress. — The BCL Team.

We’ve had many positive comments from people who have used the MEF Composition Provider for ASP.NET MVC, so we spent additional time looking at how we can improve this component and apply the same model to a wider range of applications.

There are two opportunities we identified in this process. We are setting out to address them today with a new preview, MEF 2 Lightweight Composition, which is available via NuGet and CodePlex (package details at the end of this post):

By optimizing the scenarios appropriate for web applications, a much higher throughput can be achieved (composed graphs/second)
Some tweaks to the lifetime model used in the Composition Provider make it easier to share parts from a web application with alternative hosts (back-end processes, worker roles, etc.)

Let’s take a look at these in detail.

High-Throughput/Concurrent Composition

The previous release of the Composition Provider used nested CompositionContainers to drive the creation of parts. Those familiar with MEF on the desktop will immediately recognize CompositionContainer, which is the core of the MEF hosting API.

Our V1 goals called for CompositionContainer to be optimized heavily for startup rather than throughput, given the requirements of large, extensible desktop applications like Visual Studio. CompositionContainer also supports a wide range of advanced composition scenarios that are not necessary for web applications. These include recomposition, stable composition AKA rejection, and other features that assume a low throughput, low contention execution environment. The architecture that supports these features incurs an overhead, particularly in the way concurrency is handled.

This caused us to stop and think, because it is important that we set a good foundation for MEF on the server. Web applications are highly concurrent and will continue to become more so as servers handle ever-greater volumes of requests. While composition throughput is irrelevant in many real-world applications, we would like to keep it off the performance radar entirely, so that MEF can be utilized broadly in frameworks targeting the web and cloud.

MEF 2 Lightweight Composition includes a stripped-down ‘lightweight’ version of CompositionContainer built specifically for high-throughput, concurrent composition. The Composition Provider for ASP.NET MVC uses this container to provide the same user experience as in previous versions, but with many times the throughput of the full CompositionContainer and nearly linear scaling with the addition of CPUs.

Broader Lifetime Model

The lifetime model used in the previous Composition Provider relied on a per-HTTP-request default lifetime for parts, and the [ApplicationShared] attribute to mark ‘singletons’. These together make an excellent environment for rapid web development, but they make deep assumptions that the parts being written are for web applications only.

Very few large projects produce only web entry points. From message handlers, job and batch processes to administrative console tools, large applications (the kind targeted by MEF) are frequently supported by a suite of different hosts. We want the lifetime defaults used in the Composition Provider to enable reliable sharing of parts between all of these environments (when using Lightweight Composition).

So, the updated Composition Provider employs a new, simplified lifetime model that is optimized for transactional environments. How to best take advantage of this is the topic of upcoming blog posts, but the basics are very simple to explain.

By default, parts created by the Composition Provider are non-shared
Adding a [Shared] attribute will make the part shared at the application-level, i.e. with singleton semantics
The [Shared] attribute accepts a parameter describing the ‘boundary’ within which the part will be shared, so [Shared(Boundaries.HttpRequest)] will cause an instance of the part to be shared/released along with the lifecycle of processing a web request

To see the new lifetime model in action, you can view the demo and samples in the MEF CodePlex repository. These additions apply to the lightweight model only, we are still providing CompositionScopeDefinition with the full CompositionContainer.

How Lightweight Composition Works

Apart from initialization/bootstrapping, there are very few visible differences between applications using Lightweight Composition and those using the full CompositionContainer.

For example, the following part works with Lightweight Composition:

[Export(typeof(IMessageHandler))]
public class MessageHandler : IMessageHandler
{
    [ImportingConstructor]
    public MessageHandler(IDatabaseConnection connection)
    {
    }
}

We see the familiar [Export] and [ImportingConstructor] attributes with their usual meanings. Lightweight Composition covers:

The [Export], [Import], [ImportingConstructor], [ImportMany], [PartMetadata] and [MetadataAttribute] attributes, including custom attributes
Class, interface and property exports
Constructor and property imports
IPartImportsSatisfiedNotification for OnImportsSatisfied()
IDisposable

To make a highly-optimized implementation possible, however, some features provided by the full CompositionContainer are not supported:

Imports or exports on private fields, properties or constructors – these limit our ability to generate MSIL for composition
Inherited exports, field exports, static exports or method exports – these slow down the part discovery/startup process
Importing into custom collections – this is complicated to implement efficiently but can be revisited in the future
Imports with RequiredCreationPolicy or ImportSource – these are incompatible with the simplified lifetime model
ICompositionService – this is inefficient in server scenarios; IExportProvider is used as a substitute with similar functionality

Perhaps not by coincidence, we have found ourselves recommending against using some of these features, even in environments where they are supported. Private imports and exports, for example, cause issues for testability, and field imports/exports prevent proper encapsulation.

In this preview, there may be further subtle differences, but for the most part these will be addressed with fixes and updates. Notably, the current preview may have issues running in partial trust.

On the hosting side, Lightweight Composition is a lot more minimalistic than the full Composition Container. Without rejection or recomposition, very few of the catalogs or CompositionContainer APIs are useful. Rather than modify these classes, we have opted instead to provide a simple configuration class, ContainerConfiguration. The configuration API is type/assembly-based, leaving discovery (e.g. directory scanning) to the developer for the time being.

Constructing a simple container looks like this:

var configuration = new ContainerConfiguration()
    .WithAssembly(typeof(Program).Assembly);

using (var container = configuration.CreateContainer())
{
    var greeter = container.Value.GetExport<Greeter>();
    greeter.Greet();
}

ContainerConfiguration provides the methods WithPart() and WithAssembly() as well as a few more simple helper methods that cover the majority of scenarios. The return type of CreateContainer() is ExportLifetimeContext<IExportProvider>, a clue that the new hosting code relies very strongly on ExportFactory<T> for controlling lifetime.

If you’re using ASP.NET MVC, all of this is hidden within the Composition Provider and doesn’t need to be set up manually.

Over the coming months we’ll post more detail on how to configure and use the Lightweight Composition feature; at the moment, the source code provided on CodePlex is the best place to find examples: the /oob/demos folder has code showing the new lifetime model and extension points.

Getting Started

The Lightweight Composition feature is being released via NuGet in conjunction with the updated Composition Provider for ASP.NET MVC. These two additional components will work on top of the MEF 2 assemblies that are shipping with the .NET Framework 4.5 Beta.

The two packages we're releasing are:

Important:These packages are currently marked as 'alpha' releases. The Library Package Manager dialog does not currently show prerelease packages, so to install them you need to use the Package Manager Console:

    PM> Install-Package -Pre Microsoft.Mef.MvcCompositionProvider

Installing the Microsoft.Mef.MvcCompositionProvider package into your ASP.NET MVC application will automatically pull in the other package as a dependency.

Note, we currently sign the assemblies in these packages with our public CodePlex key, and not the private framework key.

You can find instructions for getting started on MSDN. There are some minor changes between the article and the current version of the NuGet packages – we'll reconcile these as soon as possible.

The Future

The release being made today is a technology preview and will change as we deepen our understanding of the approach and how best to utilize it in web and cloud apps.

By shipping out-of-band with NuGet, we are enabling a much more responsive lifecycle for these features, with follow-up releases possible on a timeline of weeks and months, rather than years. We hope this demonstrates our commitment to making MEF the best it can be in the cloud and on the web.

If you’re using MEF in one of the XAML-based environments, don’t worry – we haven’t forgotten you! Releasing Lightweight Composition in conjunction with the Composition Provider for ASP.NET MVC is just a first step in making MEF a more responsive, community-driven project and we will ensure that these benefits span all of the application models in which MEF is supported.

We hope you find some interest and use in this functionality. We’d love to hear about your experiences, good and bad, on this blog or over at the MEF community site on CodePlex.

What’s new in MEF 2 Preview 5? [Alok, Nick]

BCL Team — Mon, 19 Dec 2011 19:18:00 GMT

Nearly two months since we published MEF 2 Preview 4, we have just posted our fifth preview on the MEF CodePlex site.

The new preview has been shaped by your feedback; everyone who took the time to download, evaluate or comment on the last version, thank you!

So, what has changed since Preview 4?

CompositionException.RootCauses, previously only visible in the debugger, can now be accessed directly in code
Using RegistrationBuilder, all interfaces on a part can now be exported using the ExportInterfaces method
IDisposable and IPartImportsSatisfiedNotification will now be filtered out by RegistrationBuilder automatically (these should never be exported)
CompositionProvider in the ASP.NET MVC integration now exposes ApplicationSharedPartMetadataKey for those wishing to create their own sharing conventions compatible with the provider
Some additional performance tuning has been done

As we get closer to making our next supported release as part of the .NET Framework, the bar is now high for changes to be included in MEF 2. Even so, we’d love to hear about your experiences with the new preview, and planning for MEF 3 is just around the corner. You can get in touch via this blog or through the discussion forum and issue tracker on CodePlex.

Sharing with CompositionScopeDefinition in MEF 2 [Alok]

BCL Team — Mon, 19 Dec 2011 17:23:00 GMT

This post discusses features in the preview version of MEF, and some details may change between now and the full release.

In the previous post, we introduced ExportFactory<T> and demonstrated how it can be used to control the lifetimes of MEF parts. In this article we will talk about a new type MEF called CompositionScopeDefinition that can be used in conjunction with ExportFactory<T> to control sharing. Let’s take a scenario that we could not achieve using what we know currently and how CompositionScopeDefinition provides the needed control.

Important note: CompositionScopeDefinition is geared towards more advanced, desktop application scenarios. If you're using MEF in an ASP.NET MVC web application and want to control sharing and lifetime on a per-HTTP-request level, you should use the pre-defined ASP.NET MVC integration.

Lets go back to our application block diagram that we built in Part 1 of this article.

Now let’s add a few details to this architecture. As our application has grown more complex, we have decided to factor out the data access part into a separate component called DataAccessLayer, which handles manages the connection to the database and reading and writing to it. In addition to this we also want to add a logger, which also logs some tracking information to the same database. So our slightly expanded functional diagram looks like the following.

In the context of our scenario, all data access within a single request should be made through the same database connection. In the default configuration, MEF will share a single DatabaseConnection across the application. How do we instruct MEF to share the database connection per request? Lets start by laying out the class definitions for the different parts.

    [Export]
    public class RequestListener
    {
        ExportFactory<RequestHandler> _handlerFactory;

        public RequestListener(ExportFactory<RequestHandler> handlerFactory)
        {
            _handlerFactory = handlerFactory;
        }

        public void HandleRequest()
        {
            using (var request = _handlerFactory.CreateExport())
            {
                request.Value.Process();
            }
        }
    }

    [Export]
    public class RequestHandler
    {
        [ImportingConstructor]
        public RequestHandler(DataAccessLayer dataAccess, Logger logger)
        {
        }

        public void Process()
        {
            //Do stuff using DataAccess and Logger
        }
    }

    [Export]
    public class DataAccessLayer
    {
        [ImportingConstructor]
        public DataAccessLayer(DatabaseConnection conn)
        {
        }
    }

    [Export]
    public class Logger
    {
        [ImportingConstructor]
        public Logger(DatabaseConnection conn)
        {
        }
    }

    [Export]
    public class DatabaseConnection
    {
        public DatabaseConnection()
        {
        }
    }

Introducing CompositionScopeDefinition

Scenario 5: Sharing within the scope of an ExportFactory<T> instantiation

As we can observe as we had in the previous article the RequestListener class has an ExportFactory to create instances of the RequestHandler class. The RequestHandler now depends on two classes the DataAccess and the Logger class, both of which depend on the DatabaseConnection class. What we want to achieve looks like the following figure.

If we look from the DatabaseConnection down to the root of the composition at RequestListener, we can see that from a composition beginning at the RequestHandler (the highlighted portion), we can achieve the sharing using standard MEF we have discussed in part 1 of this article. This is one way to figure out how to divide your catalog into scopes, when you hit a sharing pattern that cannot be achieved by using the simple MEF options. We would then need to write the following code to obtain the desired sharing.

var listenerLevel = new TypeCatalog(typeof(RequestListener));
var requestLevel = new TypeCatalog(
    typeof(RequestHandler), 
    typeof(Logger), 
    typeof(DataAccessLayer), 
    typeof(DatabaseConnection));

var container = new CompositionContainer(
     new CompositionScopeDefinition(listenerLevel, 
         new [] { new CompositionScopeDefinition(requestLevel, null) }));

var requestListener = container.GetExportedValue<RequestListener>();
requestListener.HandleRequest();
requestListener.HandleRequest();

The types making up the scenario are distributed between two catalogs — those that should be created and shared at the request listener level, and those that need to be created and shared at the per-request level. When an ExportFactory<RequestHandler> is imported into the request listener, the container sees that the RequestHandler type is within a child scope, so the child scope definition is used to determine the components (including the DatabaseConnection) that should be shared in the child scope.

In order for a parent scope to be able to discover a part in a child scope, it has to be imported as an ExportFactory<T>. If the RequestListener imported the RequestHandler type and not an ExportFactory<RequestHandler>, a CompositionException would be thrown.

How have you used CompositionScopeDefintion? Did you find it easy to use? Are there still sharing scenarios that you cannot achieve? Leave us a message in the comments or in the dicussion forum on the MEF CodePlex site.

ExportFactory in MEF 2 [Alok]

BCL Team — Fri, 18 Nov 2011 01:12:00 GMT

This post discusses features in the preview version of MEF, and some details may change between now and the full release.

In the first version of MEF there are only two notions of lifetime: A shared global lifetime or a per instance lifetime. In MEF 2 Preview 4 new support has been added to enable a finer grained control over the lifetime and sharing of parts. Before we can cover those changes in detail, we need to introduce ExportFactory<T>.

Let us consider a simple application scenario of a RequestListener, which spawns a RequestHandler, which uses a DatabaseConnection to connect to a data source (Fig 1):

Let us also consider the following parts that implement the RequestHandler and DatabaseConnection.

    [Export]
    public class RequestHandler
    {
        [ImportingConstructor]
        public RequestHandler(DatabaseConnection connection)
        {
        }
    }

    [Export]
    public class DatabaseConnection
    {
        public DatabaseConnection()
        {
        }
    }

In MEF version 1...

To better illustrate the new capabilities we’ve added in MEF 2, some of the scenarios that could be achieved in MEF version 1 are discussed below.

Scenario 1: RequestHandler and DatabaseConnection instances are shared

When we first start building our app, we are being very conservative about resources and not knowing our load we spin up a single instance of RequestHandler and DatabaseConnection

We could instantiate an instance of the RequestHandler using the following code.

TypeCatalog global = new TypeCatalog(
    typeof(RequestHandler), typeof(DatabaseConnection));
var container = new CompositionContainer(global);
var requestHandler = container.GetExportedValue<RequestHandler>();

Regardless of the number of times GetExportedValue() is called, the same instance of the RequestHandler is always returned (Fig 2).

Scenario 2: Separate instances of RequestHandler and DatabaseConnection

Now we realize that the throughput for our system is really poor, since we can only process one request at a time. So in order to increase our throughput, we spin up new instance of RequestHandler and DatabaseConnection for every request that comes in. In order to achieve this with MEF we can put a PartCreationPolicyAttribute on RequestHandler and DatabaseConnection:

    [PartCreationPolicy(CreationPolicy.NonShared)]
    public class RequestHandler { … }

    [PartCreationPolicy(CreationPolicy.NonShared)]
    public class DatabaseConnection { … }

Now, with the same invoking code to GetExportedValue(), we now get new instances for RequestHandler and DatabaseConnection every time.

Scenario 3: RequestHandler instances created and DatabaseConnection instance is shared

Our throughput is now up, but creating a new database connection per request is really taking a toll on our database server which can handle only a few open connections at a time. To ease the load on our database server we hence decide, that we will share a single instance of a DatabaseConnection, among multiple RequestHandlers. This will give us our throughput improvements without overloading our database server. We can achieve the following by getting rid of the PartCreationPolicy we have on the DatabaseConnection class, so it would now look like the following:

    public class DatabaseConnection { … }

With the removal of this attribute, we now get only a new instance of the RequestHandler every time GetExportedValue() is called, which all share a single instance of the DatabaseConnection.

In MEF 2... introducing ExportFactory<T>

The above scenarios that we have seen are the sharing scenarios that could be achieved using MEF in the past releases. However there is one glaring omission in the matrix that we presented above. What if I wanted a multiple instance of a dependency from a single instance of what it was depending on? To put it in the context of our example, how does the RequestListener class which was the first block in our diagram create multiple instances of the RequestHandler.

This brings us to:

Scenario 4: Single instance of RequestListener and multiple (dynamic) instances of RequestHandler

In order to enable this scenario, in the latest version of MEF, we introduce a new class called the ExportFactory<T>. ExportFactory<T> may be familiar to Silverlight 4 developers, since it was shipped as a part of the Silverlight 4 SDK. The export factory allows us to create new instances of dependencies and control the lifetime of the created parts.

Let us go back to our initial block diagram and write out the code for the RequestListener class.

    [Export]
    public class RequestListener
    {
        ExportFactory<RequestHandler> _factory;

        [ImportingConstructor]
        public RequestListener(ExportFactory<RequestHandler> factory)
        {
            _factory = factory;
        }

        public void HandleRequest()
        {
            using (var instance = _factory.CreateExport())
            {
                instance.Value.Process();
            }
        }
    }

Now we see in the code that instead of importing a RequestHandler as a depedency, we import an ExportFactory<RequestHandler>. MEF treats ExportFactory<T> as a special class, and will automatically hook it up to the part providing T. We then added a HandleRequest() member, which does the work of instantiating the RequestHandler. The CreateExport() method creates an ExportLifetimeContext<T>, which implements IDisposable, which is used to control the lifetime of the objects created by the factory. Since the listener is creating the parts, we can get also get rid of the PartCreationPolicy attribute on the RequestHandler. Putting all the pieces together a call to:

TypeCatalog global = new TypeCatalog(typeof(RequestHandler), typeof(DatabaseConnection) , typeof(RequestListener));
var container = new CompositionContainer(global);
var requestListener = container.GetExportedValue<RequestListener>();
requestListener.HandleRequest();

will result in the following composition graph:

Stay tuned for part two of this post where we talk about some more sharing scenarios and how we can use the new scoping enhancements in MEF in conjunction with ExportFactory<T> to accomplish these.

Using MEF 2 with ASP.NET MVC 3 [Nick]

BCL Team — Fri, 11 Nov 2011 23:47:00 GMT

This post discusses features in the preview version of MEF, and some details may change between now and the full release.

The latest MEF preview includes a new assembly, System.ComponentModel.Composition.Web.Mvc.CodePlex.dll, that simplifies the task of building flexible, testable and maitainable ASP.NET MVC web applications.

To achieve this it:

provides dependency injection for Controller classes,
defines a simple convention for identifying and configuring MEF parts,
maps the lifetime of part instances to the ASP.NET request processing lifecycle, and
simplifies the use of dependency injection in action filter attributes and model binders.

The .NET 4.0 version of the 'composition provider' assembly that can be downloaded from CodePlex only supports the preview version of MEF 2 with which it is bundled, and requires ASP.NET MVC 3. We expect to release the final version as a NuGet package targeting the MEF 2 RTM.

Update: the Composition Provider packages targeting the .NET Framework 4.5 Beta are now available on NuGet!. See this post for details including updated instructions.

In this article, we'll walk through the process of setting up basic composition for controllers and parts. We'll then examine some of the deeper support provided for other ASP.NET MVC features. You'll need some familiarity with both managed composition (MEF/IoC) and ASP.NET MVC.

This post is accompanied by a sample application called 'Fabrikam Books.' You may wish to download the sample solution now so that you can explore and experiment with runnable code.

Enabling the composition provider in an ASP.NET MVC application

Setting up the composition provider is super simple:

If you haven't already, create your ASP.NET MVC 3 web application.
Add assembly references to:
- System.ComponentModel.Composition.CodePlex.dll
- System.ComponentModel.Composition.Web.Mvc.CodePlex.dll
Use Visual Studio to create a folder in the root of your web application project called Parts

The last step — creating a Parts folder — is a good way to get started. In the long run, you may reorganise and have several parts folders in your project, but the important thing is that parts for composition live in a namespace that includes a 'Parts' element.

The composition provider from this point on is configuration-free. You can press F5 and watch your site run — so far nothing obvious should have changed, but under the covers, ASP.NET will be using MEF to construct the controllers used to handle requests! Before you can take advantage of this, you'll need to create some parts.

Creating a part

ASP.NET MVC already defines some important roles for application classes – there are controllers, models, views, filters and so-on. There are many other roles that make up a complete application; classes for:

Accessing data
Performing calculations
Communicating with other sites and web services
The secret sauce that makes your application special!

When using the composition provider for ASP.NET MVC, these classes can be implemented as parts.

The Parts folder

By convention, classes created in the Parts namespace are assumed to be parts for managed composition. Create a regular class by right-clicking on the Parts folder and selecting “Add New” → “Class”. Call the class “TraceLogger”.

    public class TraceLogger
    {
        public void Write(string text)
        {
            System.Diagnostics.Trace.WriteLine(text);
        }
    }

Exporting contracts

The goal of managed composition is to decouple the parts in an application, usually through the use of interfaces as ‘contracts’ that a part provides to other parts.

In the Parts folder, create a new interface called “ILogger” and declare a “Write” method on it:

    public interface ILogger
    {
        void Write(string text);
    }

Now go back to the TraceLogger part and implement the ILogger interface:

    public class TraceLogger : ILogger { … }

By convention, the composition provider assumes that any interfaces implemented by a part are the exports that it provides to other parts in the system.

In the included sample application, you can find a number of parts under its Parts folder. The simplest example is TraceLogger, under Parts/Tracing. The FabrikamBooksDbContext part, under Parts/Data, exports the IDbContext contract. The constructor of this part imports an ILogger, showing how parts can consume the contracts provided by other parts.

Using a part from a controller

If your ASP.NET MVC application was created from the default template, you will have a controller class called “HomeController” in the application’s Controllers folder.

Controllers are the primary place to consume parts – by adding a constructor to HomeController that accepts an ILogger, the HomeController can import the ILogger contract supplied by the TraceLogger part:

    public class HomeController : Controller
    {
        ILogger _logger;

        public HomeController(ILogger logger)
        {
            _logger = logger;
        }
        
        public ActionResult Index()
        {
            _logger.Write("Executing the Index() action method.");
            return View();
        }
    }

Whenever the HomeController.Index() action is executed, the message from the _logger.Write() call will be written to the application’s debug output window (try it!).

In the same way that HomeController can import the ILogger contract by declaring a constructor parameter, other parts can import instances of each other. This is how, from small pieces, the managed composition system can build up complex applications to do interesting things.

In the sample application, HomeController can be found under the Controllers folder.

Property imports

Although the composition provider will automatically supply constructor parameters to controllers and other parts, it will not automatically set properties, even when those properties are of a type supplied by another part. To mark a property as being an import, apply the MEF Import attribute to the property declaration.

The composition scopes

Scope or 'lifetime' determines how long a part created by the composition provider will live, and how it will be shared between other parts.

Request scope

When instances of parts are created during the processing of a web request, those parts will be shared within the scope of that web request.

So, if CatalogController requires both a ProductRepository part and a CategoryRepository part, and each repository requires a DataContext, then the same DataContext part instance will be shared by both repositories within a single web request. Within different web requests, different part instances will be created and shared.

Once processing of a request completes, the parts that were created during the web request are no longer required. If the parts implement IDisposable, their Dispose() methods will be called automatically. This is a great way to make sure that resources like database connections, files and service proxies are always released correctly.

In the Fabrikam Books sample, FabrikamDbContext is an example of a request-scoped part. It can be found under Parts/Data.

The application scope

Occasionally, creating a part instance per-request is not the desired behavior. For example, a part implementing an in-memory cache of category names might be created once and then shared by all parts in all requests served by a web application.

To mark a part as being shared at the application level, apply the ApplicationShared attribute:

    [ApplicationShared]
    public class CategoryCache : ICache<Category> { … }

Application-shared parts can only depend on contracts supplied by other application-shared parts.

Be careful when creating such parts, they will need to be thread-safe.

In the sample, TraceLogger is implemented as an application-shared part. It can be found under Parts/Tracing.

Non-shared parts, contract names and wiring

MEF comes with many additional options for wiring parts together. By applying attributes to part types, the defaults for sharing and contract matching can be overridden.

Adding parts from additional assemblies

Sometimes, perhaps because code needs to be shared between multiple applications, a web application will need to be implemented with more than one assembly.

To find parts in assemblies other than the main web application, first, the parts must be in a namespace that includes a segment ‘Parts’ such as “MyApp.OtherAssembly.Parts” or even “MyApp.Parts.OtherAssembly”.

Then, to ensure it is found by the composition provider, the assembly must be explicitly added during application startup in Global.asax.

    protected void Application_Start()
    {
        Assembly otherAssembly = ...;
        CompositionProvider.AddPartsAssembly(otherAssembly);
    }

Having to use a Parts namespace for all parts may seem a little excessive, but it is a good reminder that not every class should be a part. Parts are the ‘coarse-grained’ chunks that make up an application. Assemblies that add non-trivial functionality will almost always have a large number of additional supporting classes that the parts in the assembly use, but that are not parts themselves.

Action filter attributes

Action filter attributes can be applied to ASP.NET MVC controllers or action methods in order to change the way the action is executed. Action filter attributes are not created by the composition provider, but can be provided with contracts via properties marked with the Import attribute.

    public class SaveChangesAttribute : ActionFilterAttribute
    {
        [Import]
        public IDbContext DbContext { get; set; }

        public override void OnActionExecuted(ActionExecutedContext filterContext)
        {
            if (filterContext.Exception == null)
                DbContext.SaveChanges();
        }
    }

The sample includes Parts/Data/SaveChangesAttribute.cs. This is applied to methods on Controllers/HomeController.cs.

Model binders

The final place in ASP.NET MVC where the composition provider can be used is to create model binders. These parts transform HTTP request information into strongly-typed objects that can be used in action methods and elsewhere.

Model binders are regular parts and appear under the Parts folder, but must also be marked with the ModelBinderExport attribute so that the binder is associated with the type of model that it can bind.

    [ModelBinderExport(typeof(Book))]
    public class BookModelBinder : IModelBinder { … }

A full model binder example is given in the Fabrikam Books sample, under Parts/Binding/BookModelBinder.cs.

Summary

The composition provider for ASP.NET MVC is a great step towards making your ASP.NET MVC applications more flexible, testable and maintainable.

Using managed composition to construct parts at runtime is only a small part of the picture. The real value in this technique is that it promotes good design by separating the design of contracts from the parts that implement them.

Some good sources for discussion of these concepts are:

Mark Seemann's book on dependency injection in .NET
The SOLID principles described by Bob Martin

It is our hope that the combination of MEF and ASP.NET MVC 3 makes for an enjoyable and rewarding development experience. We'd love to hear your feedback and suggestions!

Download MEF 2 Preview 4 from the CodePlex site.

Overriding part registration conventions with the MEF attributes [Nick]

BCL Team — Thu, 03 Nov 2011 23:49:00 GMT

This post discusses features in the preview version of MEF, and some details may change between now and the full release.

One of the big advances made in the new MEF preview is a convention-driven programming model aimed at eliminating repetitive attribute usage.

In this new model, the MEF attributes like Export and Import have a new role as the clean and powerful mechanism for overriding the registration conventions.

Configuration by exception

The power in convention-driven rules is that they require 'constant effort' to handle an arbitrary number of parts.

    var registrationBuilder = new RegistrationBuilder();

    registrationBuilder.ForTypesDerivedFrom<IMessageHandler>()
        .Export<IMessageHandler>()
        .SetCreationPolicy(CreationPolicy.NonShared);

The rule above remains unchanged whether there are 10 or 10,000 message handlers in an application, while using explicit configuration for each hander would result in extra effort expended for every handler added.

Inevitably, there are edge cases and parts for which a rule doesn’t quite fit. How these cases are handled has an impact on the effectiveness and maintainability of a convention-based system.

Handling non-conforming parts (“exceptions”) via rules

It is possible to embed exceptions to part discovery/configuration rules into the rules themselves. This approach does not scale well to many parts or parts distributed between loosely-coupled assemblies.

In an application with 50 message handler parts, where 10% have some need of special handling, conventions in MEF's syntax would look like:

    registrationBuilder.ForTypesMatching(t =>
           t != typeof(OrderPlacedMessageHandler) &&
           t != typeof(NewCustomerMessageHandler) &&
           t != typeof(CheckoutMessageHandler) &&
           t != typeof(CheckBouncedMessageHandler) &&
           t != typeof(OutOfStockMessageHandler) &&
           typeof(IMessageHandler).IsAssignableFrom(t))
        .Export<IMessageHandler>()
        .SetCreationPolicy(CreationPolicy.NonShared);

    // Special contract name
    registrationBuilder.ForType<OrderPlacedMessageHandler>()
        .Export<IMessageHandler>(x => x.Named("orders"))
        .SetCreationPolicy(CreationPolicy.NonShared);

    // Default creation policy
    registrationBuilder.ForType<NewCustomerMessageHandler>()
        .Export<IMessageHandler>();

    // Similar single-component rules for remaining three parts above.
    // ...

This has the following issues:

Applying an exception requires tinkering with the host, which is more complicated than changing an individual part
As soon as there is even one exception, the declarative 'syntactic sugar' of ForTypesDerivedFrom() becomes inapplicable
All exceptional types must be visible from the code defining the rule - this does not work well with dynamic discovery e.g. DirectoryCatalog
The rule becomes unreadable and verbose
Keeping the 'shared' aspects of the conventions in sync between the single-component registrations and the general case may introduce errors
The rule text may be a point of source control contention on large product teams

Handling exceptions via attributes

Fortunately, MEF has deep support for attributed configuration, which provides a way to specify exceptions to the rules without the issues discussed above.

In a MEF application, the rule can remain unchanged.

    registrationBuilder.ForTypesDerivedFrom<IMessageHandler>()
        .Export<IMessageHandler>()
        .SetCreationPolicy(CreationPolicy.NonShared);

Then, for example OrderPlacedMessageHandler can have its particular contract name expressed by the addition of an export attribute.

    [Export("orders", typeof(IMessageHandler))]
    class OrderPlacedMessageHandler : IMessageHandler {
    }

The NewCustomerMessageHandler can modify its creation policy.

    [PartCreationPolicy(CreationPolicy.NonShared)]
    class NewCustomerMessageHandler : IMessageHandler {
    }

In contrast with the rule-based approach preceding it, the attributed approach has the following benefits:

A developer applying an exception only needs to deal with the aspect of the part they want to configure, rather than unrelated hosting code
The rule does not change in the presence of exceptions
There is no duplication of the parts of the rule that are not affected by the exception
The rule is not coupled to the implementation types
Developers viewing the 'exceptional' part source code can see that it is treated differently, for example, avoiding threading bugs caused by variations in the sharing model

How attributes and rules interact

Where a rule and an applied attribute overlap, the attribute will always take precendence over the rule.

MEF attributes in a part’s source file always override any rules that affect the same aspect of the same member.

The groups of attributes that are considered ‘aspects’ of a member are:

Export - the Export attribute, ExportMetadata, InheritedExport, custom export and metadata attributes
Import - the Import and ImportMany attributes
Constructor selection - the ImportingConstructor attribute
Part creation policy - the PartCreationPolicy attribute
Part metadata - the PartMetadata attribute

The source code items considered as a single member for the purpose of convention overrides are:

The class declaration
All constructors, considered as one group
A single constructor parameter within a constructor
A single property
A single method

Examples

Let’s take a look at some common situations where attributes may be used to override conventions.

Implementing decorators by changing the exported contract name

In the Decorator Design Pattern, one part 'wraps' another part, providing the same contract but adding funtionality to what the wrapped part provides.

For this scenario, an example convention specifies that public interfaces on a certain set of types are exported:

    registrationBuilder.ForTypesMatching(t => t.Namespace.EndsWith(".Parts"))
        .ExportInterfaces(i => i.IsPublic);

The RemoteProductRepository part provides the IProductRepository contract, while a second part, ProductCache, is introduced in order to improve product retrieval performance.

These are hooked up in a chain, so that other parts importing IProductRepository will be given the cache, while the cache itself imports the underlying repository.

The convention that applies to these types is that they export their implemented interfaces.

    public interface IProductRepository {
        Product GetById(int id);
    }

    [Export("impl", typeof(IProductRepository))]
    class RemoteProductRepository : IProductRepository { ... }

    class ProductCache : IProductRepository {
        public ProductCache([Import("impl")] IProductRepository impl) { ... }
    }

The named export and import ensure that only the ProductCache sees the decorated repository.

Changing export metadata to customise behavior

The TaskRunner component imports task components that it runs periodically. Metadata is used to determine the frequency at which the task runs.

    public interface ITask {
        void Run();
    }

    class TaskRunner : ITaskRunner {
        public TaskRunner(IEnumerable<Lazy<ITask,ITaskFrequency>> tasks) { ... }
    }

Most tasks run hourly, so this is the default by convention.

    registrationBuilder.ForTypesDerivedFrom<ITask>()
        .Export<ITask>(x => x.AddMetadata(“Frequency”, 60));

The SendMailTask runs every 5 minutes and uses an override to specify this.

     [Export(typeof(ITask)), ExportMetadata("Frequency", 5)]
    class SendMailTask : ITask { ... }

Selecting a shorter constructor

By default a part configured with RegistrationBuilder will use the constructor with the most parameters. In this example, the longer constructor is for unit testing purposes only, so the shorter constructor is marked with an attribute.

    class MyPart {
        public MyPart(IFoo foo, IBarTestStub bar) { }

        [ImportingConstructor]
        public MyPart(IFoo foo) { }
    }

Changing the contract name of an import for explicit wiring

If a property is selected as an import by convention, an Import attribute can be applied to the property to override the contract name or other facet of the ImportDefinition. This is useful to implement explicit wiring.

    class MyPart {
        [Import("impl")]
        public IFoo Foo { get; set; }
    }

    class Foo1 : IFoo { }

    [Export("impl", typeof(IFoo))]
    class Foo2 : IFoo { }

Here the export and import attributes specify names so that the Foo import on MyPart will be provided with a Foo2 instance.

Opting out

If a type that would otherwise be matched by a rule should not be used as a part, it can be marked with the PartNotDiscoverable attribute:

    [PartNotDiscoverable]
    class MyPart {… }

Diagnostics

Whenever a rule is overridden by a source-level attribute, you can find a diagnostic message in the debugger trace output. This helps keep track of where the exceptions to the rules are. The message looks like:

    A convention that would apply to member A of type B has been overridden
    by attributes applied in the source file.

Conclusions

When using conventions to register parts in MEF, rules can be kept clean and simple by using the standard MEF attributes to override rules where necessary.

We hope you enjoy working with the new preview; we deeply value your feedback and would love to hear about your experiences via the Discussion Forum or Issue Tracker on the MEF developer preview site.

Coming up on this blog we’ll explore the other major addition to MEF in version 2 – composition scoping.

Download MEF 2 Preview 4 from the CodePlex site.

Getting started with convention-based part registration in MEF 2 [Nick]

BCL Team — Tue, 01 Nov 2011 18:14:00 GMT

This post discusses features in the preview version of MEF, and some details may change between now and the full release.

MEF ‘version 1’ uses attributes to describe how a type should be handled for composition purposes.

    [Export, PartCreationPolicy(CreationPolicy.NonShared)]
    public class HomeController :  Controller
    {
        [ImportingConstructor]
        public HomeController(/* Dependencies here */) { … }
        public ActionResult Index() { … }
    }

Attributes are a good fit for plug-in development when building extensible applications, and we haven't changed any of this experience in the new version. Attributes are still a great way to unambiguously mark-up types for composition.

Planning MEF ‘version 2’ we observed that in large MEF-based applications, only a small proportion of imports and exports are actually extensibility related. A large amount of composition happens to support the internal architecture of the application itself. In many cases, applications that use MEF don’t provide a third-party plug-in model at all, and use MEF purely as a way to bring the benefits of loosely-coupled architecture to monolithic apps.

In the context of a single application, it is much easier to set and communicate rules about how parts should be written, loosening up the need to explicitly specify every detail of every part.

In the example above, if all controllers are exported, all have the NonShared creation policy, and all use a single constructor to receive dependencies, then it is in some ways harmful to repeat that information using attributes on every single controller class. Given these ‘norms’ for a controller, a programmer working in a structured application should be able to write:

    public class HomeController :  Controller
    {
        public HomeController(/* Dependencies here */) { … }
        public ActionResult Index() { … }
    }

…and have the composition aspects inferred automatically. Less repetition leads to more readable code, less overall lines of code and fewer opportunities to make mistakes.

RegistrationBuilder is a new MEF feature coming in .NET 4.5 that provides exactly this kind of experience. Using RegistrationBuilder, the rules defining controller parts look like:

    var conventions = new RegistrationBuilder();
    conventions.ForTypesDerivedFrom<Controller>()
        .Export()
        .SetCreationPolicy(CreationPolicy.NonShared);

Notice that the rule doesn’t say anything about how to find types that derive from Controller. Catalogs still play the role of locating types, while RegistrationBuilder rules replace the attribute syntax. So, to apply these rules to all types in an assembly, an AssemblyCatalog is used:

    Assembly controllersAssembly = // …
    var catalog = new AssemblyCatalog(controllersAssembly, conventions);
    var container = new CompositionContainer(catalog);

Notice the additional constructor parameter accepted by the catalog. All of the familiar MEF catalog types have been updated in the new version to support RegistrationBuilder conventions.

Building Conventions

Using RegistrationBuilder has two aspects:

Identifying types that should be considered parts; and,
Configuring types for use as parts.

Identifying Parts

Each rule is defined by calling one of the three For*() methods on RegistrationBuilder to specify which types the rule will apply to.

ForType<T>() — selects the single type T. For example ForType<HomeController>() would set up a rule for the concrete HomeController type.
ForTypesDerivedFrom<T>() — selects types assignable, but not equal to, a contract type T. This may be a base class or an interface. The example rule for identifying controllers earlier in this article uses this mechanism.
ForTypesMatching(Predicate<Type> predicate) — selects types that match a Boolean selector. For example, ForTypesMatching(t => t.Name.EndsWith("Repository")) will select all types whose name ends with the word “Repository”. This is the most flexible option.

Multiple rules may match the same type. In such cases, as long as the rules don’t overlap in an incompatible way, all matching rules are applied.

The selector methods above return a PartBuilder that can then be used to configure the matching types for use as parts.

Configuring Parts

Rules can be used to configure all aspects of a MEF part:

Exported interfaces and properties, with metadata
Importing constructors, constructor parameters and properties
Part creation policy
Part metadata

Exports

Exports can be configured at the type declaration/interface level, or via properties in the case of property exports.

At the level of the part type itself, the concrete type of the part can be exported (in this case MainWindow):

    conventions.ForType<MainWindow>()
        .Export()

Or, more commonly, one or more of its interfaces will be exported:

    conventions.ForType<MainWindow>()
        .Export<IView>();

Multiple interfaces can be exported using a filter:

    conventions.ForType<MainWindow>()
        .ExportInterfaces(i => i.IsPublic);

Overloads exist for configuring the export with a contract name or with metadata:

    conventions.ForType<MainWindow>()
        .Export<IView>(x => x.AddMetadata("Name", "Main"))

Properties can be exported using a similar syntax. If the concrete type is specified, properties can be selected using a Linq expression:

    conventions.ForType<MainWindow>()
        .ExportProperty(mw => mw.Notifier);

When the concrete type is not known, for example with ForTypesMatching(), properties can be exported by selecting them:

    conventions.ForTypesMatching(t => t.Namespace == "MyApp.Windows")
        .ExportProperties(pi => pi.IsPublic);

Multiple export-related configuration rules on the same part are additive, so a type can export several contracts:

    conventions.ForType<MainWindow>()
        .Export<IView>()
        .Export<ICloseable>();

Imports

Imports are specified either at the constructor or at the property level.

By default, the RegistrationBuilder will select the public constructor with the most parameters as the importing constructor.

To select a different constructor, the SelectConstructor() method is used.

    conventions.ForTypesDerivedFrom<IView>()
        .SelectConstructor(ctors =>
            ctors.Min(ctor => ctor.GetParameters().Length));

When the concrete type of the part is known at configuration time, the constructor can be selected using strongly-typed syntax:

    conventions.ForType<MainWindow>()
        .SelectConstructor(pb => new MainWindow(pb.Import<ILogger>()));

By default, any constructor parameters or imported properties of an enumerable or collection type will be considered to “Import Many”.

The syntax for performing property imports mirrors the syntax described for property exports previously.

Part Creation Policy

By default, parts configured by RegistrationBuilder will use the standard MEF CreationPolicy.Any setting. In normal use, this will boil down to ‘single instance per container’ sharing, unless the part is created using an ExportFactory<T>.

To select the creation policy, use the SetCreationPolicy() method:

    conventions.ForTypesDerivedFrom<Controller>()
        .SetCreationPolicy(CreationPolicy.NonShared);

(Note, the ASP.NET MVC integration also included in the preview interprets CreationPolicy.Any and CreationPolicy.Shared to mean ‘single instance per HTTP request.’ To make a part use ‘single instance per application’ sharing under ASP.NET MVC, you need to apply the ApplicationShared attribute to the part.)

Part Metadata

Part metadata is useful when filtering parts in customized catalogs, or when setting up a CompositionScopeDefinition. Part metadata can be added explicitly, as in:

    conventions.ForType<OutputWindow>()
        .AddMetadata("Mode", "Debug");

Or, it can be calculated from the type:

    conventions.ForType<OutputWindow>()
        .AddMetadata("Mode", t => GetMode(t));

Overriding Rules

Inevitably, there are exceptions to every rule. In these situations, the MEF attributes can often be used to override the rules on a case-by-case basis. This is the topic of an upcoming post – stay tuned!

Download MEF 2 Preview 4 from the CodePlex site.

What’s new in MEF 2 Preview 4? [Alok, Nick]

BCL Team — Thu, 27 Oct 2011 23:15:00 GMT

The MEF team has been quietly at work for some time getting the finishing touches on another developer preview. You can download the new source and .NET 4.0-compatible binaries from the Codeplex site.

This preview is another step towards the next supported release, which will be made as part of .NET 4.5.

Highlights of the new version are:

Support for generic parts
A new convention-based registration API (attributes optional!)
New scoping features that use ExportFactory and CompositionScopeDefinition
Integrated support for MEF composition in ASP.NET MVC
Diagnostics improvements

Open generic parts

Open generic types can now be used as parts. For example, if you are using the Generic Repository pattern, you may use IRepository<T> as a contract:

public interface IRepository<T> where T : class
{
    T FindById(int id);
}

Other parts then import this contract using closed types like IRepository<Customer>:

[Export]
public class CustomersWindow :  Window
{
    [ImportingConstructor]
    public CustomerWindow(IRepository<Customer> customers)
    {
        // … use customers

Rather than create a specific CustomerRepository class, and a similar implementation for other kinds of repository, the repository interface can be exported on a generic type:

[Export(typeof(IRepository<>))]
public class EntityFrameworkRepository<T> : IRepository<T> where T : class
{
    // … generic repository implementation

Open generic parts reduce the effort required to implement many common scenarios.

Convention-based part registration

The previous MEF release was optimized for third-party extensibility scenarios, where explicit annotation of parts with attributes makes a lot of sense. When the entire application is under the control of one group of developers, applying attributes to describe parts can be redundant and repetitive.

For example, every class that derives from Window in an application might have an Export attribute applied:

[Export]
class CustomersWindow : Window { … }

RegistrationBuilder is a class added in the new release that allows a set of rules (‘conventions’) to be defined that map types to parts, without the need for attributes.

var rb = new RegistrationBuilder();
rb.ForTypesDerivedFrom<Window>().Export();

The standard MEF catalogs accept a RegistrationBuilder as a constructor parameter, and will apply the rules defined in the registration builder to the types included in the catalog:

var catalog = new AssemblyCatalog(Assembly.GetExecutingAssembly(), rb);

RegistrationBuilder is available in System.ComponentModel.Composition.Registration.

Composition scoping enhancements

In the first version of MEF, part instances could be either shared or per-importer, but it was much harder to control scope and lifetime in a more fine-grained way. Often it is desirable to configure and manage parts as a group to handle some kind of unit of work – a web request, processing a message or managing a single visual element like a WPF view and ViewModel.

The new MEF release makes it easier to control the lifetime and sharing of parts. This example shows how the new scoping support can be used to isolate the parts used for processing 'messages' — the basic pattern applies to many scenarios.

[Export]
public class DataConnection
{
}
 
[Export]
public class MessageHandler
{
    [ImportingConstructor]
    public MessageHandler(DataConnection dataConnection) { … }

    // HandleMessage(Message m) omitted
}

The two 'skeleton' parts above work together to process a network message of some kind. Each time a message handler is created, it will be initialized with a DataConnection that it will use to make changes in response to the message.

More importantly, other dependencies of MessageHandler (not shown) should share the same DataConnection. MessageHandlers processing different messages should get different DataConnections. Any of these parts may also have dependencies that are global, i.e. common to all scopes.

[Export]
public class MessageListener
{
    [ImportingConstructor]
    public MessageListener(ExportFactory<MessageHandler> handlerFactory) { … }
 
    public void ProcessRequest()
    {
        var message = GetNextMessage();
        using (var handler = _handlerFactory.CreateExport())
            handler.HandleMessage(message);
    }
}

The implementation of MessageListener may be familiar to users of MEF in Silverlight 4. The ExportFactory<T> type allows the MessageListener to create a new message handler for each message.

Now, by default, this won’t enable any kind of scoping around the individual message listeners — they’ll all share the same scope (and DataConnection) as the MessageListener.

To describe the scoping we want, the container needs to be created with a CompositionScopeDefinition in place of a ComposablePartCatalog:

var globalParts = new TypeCatalog(typeof(MessageListener));
var perMessageParts = new TypeCatalog(typeof(MessageHandler), typeof(DbConnection));
var scopeDefinition = new CompositionScopeDefinition(globalParts,
            new[] { new CompositionScopeDefinition(perMessageParts, null)});

var container = new CompositionContainer(scopeDefinition);
var listener = container.GetExportedValue<GlobalRequestManager>();
listener.HandleMessage();
listener.HandleMessage();

Every time HandleMessage() is called, new scoped instances of MessageHandler and DataConnection are created and released.

CompositionScopeDefinition is available in the System.ComponentModel.Composition.Hosting namespace.

Composition provider for ASP.NET MVC

Simplifying the use of MEF in web applications is a core goal of the new release. To further this aim, we are supplying functionality that integrates MEF into ASP.NET MVC, to allow controllers and other MVC types to participate in composition.

To use this feature, reference the System.ComponentModel.Composition.Web.Mvc assembly in your ASP.NET MVC project. All configuration is handled automatically — to create parts, add them to a Parts namespace (folder) under the MVC site:

namespace FabrikamBooks.Parts
{
    class TraceLogger : ILogger { … }

Just by being under the Parts folder, the class will automatically ‘export’ all of its implemented interfaces. The longest constructor on the class will be treated as its importing constructor. No attributes are required, unless you want to override the inbuilt defaults.

Controllers simply take dependencies on parts through their constructors:

class HomeController : Controller
{
    // MEF will automatically supply TraceLogger
    public HomeController(ILogger logger) { … }

The composition provider implements a set of defaults that we feel are sensible for most web applications. These defaults can be overridden by applying attributes to parts. Advanced users can still set up MEF ‘by hand’ to meet more specific goals.

The downloadable binaries for this feature are built against ASP.NET MVC version 3; if you want to rebuild the assemblies from source, you'll need to get ASP.NET MVC separately. To maintain a flexible release schedule as this component evolves, we anticipate delivering binaries via NuGet rather than the .NET Framework SDK.

Diagnostics improvements

Providing good diagnostic support in a framework as dynamic and flexible as MEF is a challenge, and is something that we expect to go on improving over time.

In this release, two key improvements have been made to diagnostics in MEF.

CompositionOptions.DisableSilentRejection

MEF’s roots in extensibility make it more tolerant of composition problems than other similar frameworks. Parts with missing dependencies are identified and excluded from composition, which makes diagnostics hard in some circumstances.

When constructing a CompositionContainer in the new MEF version, flags can be passed controlling this and other behaviors. Specifying CompositionOptions.DisableSilentRejection will cause exceptions to be thrown when any composition problem occurs.

var container = new CompositionContainer(
    catalog, CompositionOptions.DisableSilentRejection);

It is recommended that most users building non-third-party-extensible applications specify this flag. It is set by default when using the ASP.NET MVC composition provider described previously.

Better exception messages

Finally, the exception message text in the first version of MEF can be complex and verbose in some situations. The new release improves the way exception messages are formatted, presenting the most important information up-front.

Moving forward

Subscribe to this blog for examples and more detailed documentation on the new features.

We’re looking forward to your feedback on this release. As always, you can get in touch with the team through this blog or at the MEF Codeplex site’s issue list and discussion forum.

Porting existing .NET code to Metro style apps [Wes]

BCL Team — Tue, 04 Oct 2011 04:25:00 GMT

As many of you know at //build we announced that there will be a new .NET target framework for Metro style apps. To learn more about the goals of this new target framework I encourage you to watch Krzystzof Cwalina’s build talk "A .NET developer's view of Windows 8 app development".

One of the things we know people want to do is reuse existing .NET code but due to the constraints on Metro style apps it is not possible to directly reuse existing .NET class libraries nor is it always possible to simply recompile existing code. We realize this is going to be a pain point for many current .NET developers so we are working on a guide at the .NET for Metro style apps overview page which will assist developers in translating their existing .NET skill set into the Metro style apps world. Sometimes that means explaining why something was removed and other times it means giving another .NET or WinRT API to call instead. Today the page is limited but we are actively working on it internally to update it with more data for Beta and RTM.

For a demo of the kind of things you are likely to run into while porting code I suggest watching Daniel Plaisted’s build talk "Bringing existing managed code into Metro style apps".

While we are looking at many forms of feedback we would greatly appreciate any feedback on issues you run into while porting existing .NET code to Metro style apps. Please leave feedback in the comments or on the forums.

Memory Mapped File Quirks [Greg]

BCL Team — Mon, 06 Jun 2011 20:56:00 GMT

Memory mapped files are segments of virtual memory that are directly mapped to a physical file on disk, byte-by-byte. This technology has a number of benefits over traditional stream based I/O, such as performance during random access to large files, or the ability to share the mapped memory between different threads and processes. The .NET framework provides built-in support for memory mapped files starting from version 4. In most cases using memory mapped files in .NET is as easy as using traditional files streams: just map a file into the memory, open a seekable memory stream and use it as if it was a normal file while enjoying all the performance benefits. However, there are some subtle, but important differences. One of them is that memory mapped file views must always be aligned at the boundaries of the system’s virtual memory pages. An important consequence is that:

Compared to the size of the underlying file on disk, the size of a memory mapped view stream is always rounded up towards a multiple of the system’s virtual memory page size. You must take care to differentiate between useful data and junk bytes used to fill up the space.

A little background

A memory mapped file is a continuous chunk of memory in a process address space that has a direct byte-by-byte mapping to an external resource. Typically, such resource is a data file on the physical hard drive. However, it is also possible to represent device memory or any other OS object that can be accessed via a file handle as a memory mapped file structure. In Windows, memory mapped files can also be backed by the system page file rather than by a separate file system object.

In Windows, managing memory mapped files has been supported for a while (see the function reference). The .NET framework natively supports this technology starting from version 4.0 through the types in the System.IO.MemoryMappedFiles namespace.

In .NET you can create a memory mapping either for an existing file on the physical hard drive by specifying its path, or for an existing mapping by referring to the map name. Alternatively, you can create a new memory mapped file and have the system automatically manage a temporary file on disk to back it up. The temp file will be automatically deleted when you dispose of the memory mapped file; this is particularly useful when using memory mapped files for inter-process communication rather than for data persistence. In any case you need to use one of the static methods in the MemoryMappedFile class to create a memory map instance.

On the OS level, in order to actually read data from or write data to a file mapping you need to first create a view (projection) of that map into your process. Luckily, .NET handles this for you and all you need to do is to create either a MemoryMappedViewAccessor or a MemoryMappedViewStream using the CreateViewAccessor or CreateViewStream methods respectively. You can then use the accessor or the stream for I/O on the memory mapped file. Particularly when using the stream, the approach is not much different from using a traditional FileStream.

Subtle differences (A): The map is not quite the same as the source

Every technology must be fully understood in order to fully realise the improvements it brings over older technologies, and this is no different for memory mapped files. Although a memory map can be usually understood as a byte-by-byte mapping of the underlying resource (file) there are some subtle, but important cases where this does not apply. It is important to be aware of these cases when working with memory mapped files.

For instance, recall that NTFS optionally supports both, compression and encryption of files. Even if compression has not been activated, NTFS can also support sparse files. The idea behind sparse files is, in a nutshell, that if a file contains a very large number of consecutive zeros, the file system performs a very simple form of compression that avoids writing out all those zeros to the disk. Instead, just the number of the zeros is written, thus saving disk space without a measurable performance cost. When the zeros are overwritten with other data, the file system replaces those file chunks with normal data segments. All this happens transparently to the layers above the file system. So, when you create a memory mapping of an encrypted, compressed or sparse file, the memory map holds the respectively decrypted, decompressed or inflated content of the file. As a result, the memory page does not correspond to the physical disk file content on a byte-by-byte level. In most cases you do not need to worry about it. However, in some cases this difference may become important, for instance when you want to copy the map contents to a file manually rather than flushing the view, or when you want to use the memory map contents to reason about the size of the file on disk.

Subtle differences (B): Where are all those zeros coming from?

Another important difference relates to the size or length of the file on disk vs. the size of the memory map. To emphasize the problem, let us consider an example program that reads and displays the contents of a text file that has been mapped to memory:

 
using System;
using System.IO;
using System.IO.MemoryMappedFiles;
using System.Text;
 
 
namespace Microsoft.NetFx.BCL.MSDNBlog {
class MemoryMappedViewStreamLengthDemo {
 
  const String dataFilePath = @"data01.txt";
 
 
  public void Exec() {
 
    using (MemoryMappedFile mmFile = MemoryMappedFile.CreateFromFile(dataFilePath)) {
 
      Stream mmvStream = mmFile.CreateViewStream();
      Console.WriteLine("Stream length: {0}", mmvStream.Length);
            
      using (StreamReader sr = new StreamReader(mmvStream, ASCIIEncoding.ASCII)) {
 
        while (!sr.EndOfStream) {
 
          String line = sr.ReadLine();
          Console.WriteLine("[{0:D4}] \"{1}\"", line.Length, line);
        }
      }            
    }
  }
 
 
  public static void Main(String[] unusedArgs) {
 
    (new MemoryMappedViewStreamLengthDemo()).Exec();
  }
 
}
}

The input file data01.txt looks like this:

This file is 13 bytes long: 9 bytes are taken up by the digit characters, and 4 more by the invisible carriage return (\r) and new line (\n) characters in lines 1 and 2 (line 3 does not have them because there is no new line 4). Here is a hex-binary view of the file:

 
31  31  31  0D  0A  32  32  32  0D  0A  33  33  33
 1   1   1  \r  \n   2   2   2  \r  \n   3   3   3

So naturally, we expect the following output: (note that ReadLine() discards the training \r and \n characters)

Stream length: 13
[0003] "111"
[0003] "222"
[0003] "333"

(Note: the prefix "[0003]" describes the length of each line. It results from the "[{0:D4}]"-part of the WriteLine command in the inner loop of the above example program.)

However, this is not what we get. Instead, we get this (on my machine):

Stream length: 4096
[0003] "111"
[0003] "222"
[4086] "333

































              "

If we step through our demo we can quickly discover that during the last iteration the string variable line consists of three '3'-characters followed by 4083 null characters:

 
    line = "333\0\0\0\0\0 ... \0\0\0";

So what is the reason for this unexpected behaviour and how can we prevent it from happening

Recall that when you create a view of a memory mapped file, Windows aligns the view to virtual memory page boundaries. E.g., if your page size is, say, 4096 bytes, a memory-map view can only be 4096, 8192, 12288, ..., N × 4096 bytes large.

As our data file is smaller than a memory page, we get a view that is as large as a memory page (if the size of virtual memory pages on your machine is not 4096, then the output will also differ accordingly). The memory-map view contains the data supplemented with zeros. When we use a stream reader to read this data, we only reach the EOF mark when we get to the end of the entire view, not to the end of the original data.

One may wish that the .NET APIs were smart enough to notice the size of the underlying file on disk and to use this information to make sure that the stream reader hits an EOF at the end of the original data. However, this is not always possible, even in principle, for three types of reasons:

First, it is not always possible to even determine the path of the backing file on disk. The MemoryMappedFile may have been created not by specifying a physical file, but by referring to a previously memory-mapped file [OS ref]. In such case, it is not possible to determine the backing file path. Recall that it is also possible to create a new MemoryMappedFile without specifying a backing file on disk at all (by either using the system page file of a temp file) [OS ref].

The second reason is that even when the path of the backing disk file is known, it is not in general possible to reliably determine the subset of the memory page that represents actual data versus the subset that is just filling zeros. The memory contents may not directly correspond to the file on disk due to any of the reasons pointed out in the previous section, or simply because the memory has been modified but is not yet flushed.

Finally, the framework APIs do not have the application-specific domain knowledge about whether consecutive zeros could at all be an intended content of the mapped file or must represent filler bytes to the end of the memory page.

Solution: Know your data

On the other hand, application logic may very well be able to easily differentiate between useful and junk data.

For instance, in our particular case we know that the actual data does not contain any null characters, so we could peek ahead and stop reading when we see null characters coming up. Another approach is to define a constant such as

 
    static readonly char[] FillerChars = new char[] { '\0' };

and then to trim the strings as we read them in:

 
    String line = rdr.ReadLine().Trim(FillerChars);

Summary

Memory mapped files provide many performance benefits and can be used for inter-process communication as well as for random file access. When using memory mapped files, developers need to be aware of some quirks. One of these quirks is that the size of a memory mapped file view is always blown up to the next virtual memory page boundary. When accessing data from such memory-map views, it is the responsibility of the application developer to use the domain knowledge in order to determine what memory contents represent original data, and what are junk bytes used to fill the space up to the next virtual memory page boundary.

Aligning Numbers on the Decimal Separator [Ron Petrusha]

BCL Team — Mon, 16 May 2011 01:34:00 GMT

The composite formatting feature in the .NET Framework makes it easy to left-align or right-align a value in a fixed-width field. If the alignment component of a format item in a composite format string is negative, its corresponding argument is left-aligned. If the alignment component is positive, its corresponding argument is right-aligned. For instance, the following example assigns random numbers to an eleven-element array and displays their values. In the output, the array index is left-aligned in a five-character field, and the array value is right-aligned in a twelve-character field.

[Visual Basic]
Dim rnd As New Random()
Dim values(10) As Double
Console.WriteLine("{0,-5}{1,12}", "Index", "Value")
For ctr As Integer = values.GetLowerBound(0) To values.GetUpperBound(0)
    values(ctr) = rnd.NextDouble()
    Console.WriteLine(" {0,-5} {1,12:F6}", ctr, values(ctr))
Next

[C#]
Random rnd = new Random();
double[] values = new double[11];
Console.WriteLine("{0,-5}{1,12}", "Index", "Value");
for (int ctr = values.GetLowerBound(0); ctr <= values.GetUpperBound(0); ctr++)
{
   values[ctr] = rnd.NextDouble();
   Console.WriteLine(" {0,-5} {1,12:F6}", ctr, values[ctr]);
}

The example produces output similar to the following:

Index      Value

 0         0.682236
 1         0.827866
 2         0.927542
 3         0.670535
 4         0.023612
 5         0.353050
 6         0.685052
 7         0.938842
 8         0.494007
 9         0.532631
 10        0.978260

The .NET Framework composite formatting feature doesn’t support additional forms of alignment such as centering a value or aligning numeric values in a field on their decimal points. However, such forms of alignment can be easily implemented by dynamically building a composite format string or by modifying the string representation of the value to be formatted. This blog post will illustrate both approaches by showing how to align a series of numeric values on their decimal point.

The first step is to determine the maximum size of the field that is to display the aligned values. In some cases, the field size is static and is known at compile time. In other cases, the precise range of the values to be formatted is based on the format of data that is known only at runtime. However, the alignment component of a format string must consist of a literal numeric value; you cannot supply a variable that represents that value. For example, the composite format string "The value is {0,12}." indicates that the first argument is to be right-aligned in a twelve-character field. But the composite format string "The value is {0,width}." throws a FormatException.

This means that if we cannot supply a numeric value to the alignment component of a composite format string at design time, we must use string concatenation to dynamically create the format string at runtime. This, in turn, requires an initial pass through the data to be formatted. To align numeric data on a decimal point, we need two items of information:

The number of characters in the data item that has the largest number of integral digits.
The number of characters in the data item that has the largest number of fractional digits.

The sum of these two values, along with the number of characters reserved for the decimal separator, any other formatting styles, and white space, defines the total width of the field. The number of characters in the data item with the largest number of integral digits, along with any formatting styles and leading white space, defines the maximum number of characters that you must accommodate for the value before the decimal separator in the formatted string. For example, to align the values 16.509 and 13052.4 on their decimal point, requires a field width of nine (assuming that we do not want to include any additional styles such as white space or group separators) with a maximum of five integral digits.

The following example illustrates this first pass through an array of numeric values to determine the total field size and the maximum number of integral characters. Note that the example determines the number of characters in a particular numeric value by calling the ToString method with a format specifier.

[Visual Basic]
Dim numbers() As Decimal = { 3D, 15.4D, 19.008D, 18.62D, 1093.425D }
Dim intDigits, decDigits, decPosition As Integer
Dim decimalSeparator As String = NumberFormatInfo.CurrentInfo.NumberDecimalSeparator
For Each number As Decimal In numbers
   Dim value As String = number.ToString("G")
   decPosition = value.IndexOf(decimalSeparator)
   If intDigits < decPosition Then intDigits = decPosition
   If decDigits < value.Length - (decPosition + 1) Then decDigits = value.Length - (decPosition + 1)
Next

Dim fieldWidth As Integer = intDigits + decimalSeparator.Length + decDigits

Console.WriteLine("Maximum number of integral digits: {0}", intDigits)
Console.WriteLine("Maximum number of decimal digits: {0}", decDigits)
Console.WriteLine("Total number of characters: {0}", fieldWidth)
Console.WriteLine()

[C#]
decimal[] numbers = { 3m, 15.4m, 19.008m, 18.62m, 1093.425m };
int intDigits = 0, decDigits = 0, decPosition = 0;
string decimalSeparator = NumberFormatInfo.CurrentInfo.NumberDecimalSeparator;
foreach (decimal number in numbers)
{
   string value = number.ToString("G");
   decPosition = value.IndexOf(decimalSeparator);
   if (intDigits < decPosition) intDigits = decPosition;
   if (decDigits < value.Length - (decPosition + 1)) decDigits = value.Length - (decPosition + 1);
}

int fieldWidth = intDigits + decimalSeparator.Length + decDigits;

Console.WriteLine("Maximum number of integral digits: {0}", intDigits);
Console.WriteLine("Maximum number of decimal digits: {0}", decDigits);
Console.WriteLine("Total number of characters: {0}", fieldWidth);
Console.WriteLine();

As the output shows, aligning this data on its decimal separator requires that each data item be displayed in an eight-character field, and that each data item be allocated space for four integral characters.

Maximum number of integral digits: 4
Maximum number of decimal digits: 3
Total number of characters: 8

Once the first pass through the data determines the total field width and the number of integral characters in each field, a second pass can display the data. To align numeric values on their decimal separator, this second pass must dynamically add the field width to the format string. Since we are aligning the numeric values on their decimal points based on the number of integral characters, the string must define a left-aligned field. When iterating the numeric values in the array, each number must be padded with a sufficient number of leading spaces so that the total length of its integral portion equals the maximum number of integral digits. This allows the numbers to be aligned on their decimal separators.

[Visual Basic]
Dim fmt As String = "Value: {0,-" + fieldWidth.ToString() + "}"
For Each number As Decimal In numbers
   decPosition = number.ToString().IndexOf(decimalSeparator)
   If decPosition = -1 Then decPosition = number.ToString().Length

   Dim value As String = String.Format("{0}{1}", _
                         New String(" "c, intDigits - decPosition), number)

   Console.WriteLine(fmt, value)
Next

[C#]
string fmt = "Value: {0,-" + fieldWidth.ToString() + "}";
foreach (decimal number in numbers)
{
   decPosition = number.ToString().IndexOf(decimalSeparator);
   if (decPosition == -1) decPosition = number.ToString().Length;

   string value = String.Format("{0}{1}", 
                  new String(' ', intDigits - decPosition), number);

   Console.WriteLine(fmt, value);
}

The example displays the following output:

Value:    3
Value:   15.4
Value:   19.008
Value:   18.62
Value: 1093.425

Although this example shows how to align numeric values on their decimal point, you can adopt a similar approach to other forms of alignment. To center a numeric value in a field, for example, requires that you know the total field width. Centering the number is then just a matter of determining the length of the string that represents a particular number and padding it with a number of spaces equal to half of the difference between the total field width and the number’s length.

Unexpected effects of the RegEx SET operator [Greg]

BCL Team — Mon, 02 May 2011 23:24:00 GMT

Regular Expressions (RegEx) are a powerful tool for searching for text that matches specific patterns, but it is also a complex tool that requires care and attention to detail. There are many caveats to using RegEx.
Here, I would like to briefly talk about a common caveat when using RegEx: the set operator [ ] and its effect on the contained tokens...

...(read more)

Parsing Non-Standard Date and Time Formats [Ron Petrusha]

BCL Team — Thu, 14 Apr 2011 00:19:00 GMT

Frequently, particularly when dealing with remote data collection devices, an application receives string data containing date and time information that must be converted to either DateTime or DateTimeOffset values. In these cases, the most commonly used parsing methods, the overloads of the DateTime.Parse or DateTimeOffset.Parse methods, all throw a FormatException when they attempt to perform the conversion, and their corresponding TryParse overloads all return false and assign a data and time of MinValue to the method’s date and time argument. However, the .NET Framework provides two alternative overloaded methods, ParseExact and TryParseExact, that can be used to parse the string representations of date and time values.

In parsing date and time strings, both Parse and TryParse use a NumberFormatInfo object that corresponds either to the current culture or, if one of the overloads with an IFormatProvider parameter is called, to the culture that corresponds to the IFormatProvider argument. The properties of this NumberFormatInfo object determine the acceptable date and time formats that a string can have in order to parse successfully. Even though the number of supported formats is typically sizable, the Parse and TryParse methods cannot parse strings in non-standard formats successfully.

On the other hand, the ParseExact or TryParseExact methods are ideal, since they require that a string exactly conform to a particular definable pattern if the parse operation is to succeed. They are useful when either of the following conditions holds true:

The precise format of a string containing date and time information is known in advance, as it is when the data is gathered by some data collection object or device.
User input must have a particular format.

The overloads of the ParseExact and TryParseExact methods of both the DateTime and DateTimeOffset objects allow you to specify either a single pattern or an array of patterns. You to define a pattern by specifying a format string that consists of either a standard format specifier or one or more custom format specifiers. The input string must then conform to that single pattern or to any one of the patterns in the array. Two of the overloads also include a DateTimeStyles parameter that allows you to define style elements (such as white-space characters) that may be included in the input, or to determine how the input string is to be interpreted (for example, as local or universal time).

To see how the ParseExact and TryParseExact methods can help with parsing date and time strings, let’s take three examples. First, in some cases, date and time data is provided to an application as a set of numbers without any separators (for example, “20100501061245500”). The following code uses a custom date and time format string to do this.

[Visual Basic]
Module Example
   Public Sub Main()
      Dim fmt As String = "yyyyMMddhhmmssFFFFFFF"
      Dim value As String = "20100501061245500"
      Dim dt As DateTime = DateTime.ParseExact(value, fmt, Nothing)
      Console.WriteLine("{0}--> {1}", value, dt)
   End Sub
End Module

[C#]
using System;

public class  Example
{
   public static void Main()
   {
      string fmt = "yyyyMMddhhmmssFFFFFFF";
      string value = "20100501061245500";
      DateTime dt = DateTime.ParseExact(value, fmt, null);
      Console.WriteLine("{0} --> {1}", value, dt);
   }
}

The example displays the following output:

20100501061245500 --> 5/1/2010 6:12:45 AM

Second, in some cases, date and time strings include time zone information expressed as the time zone’s offset from Coordinated Universal Time (UTC). The z, zz, or zzz custom format specifiers can be used in a custom format string to handle such time zone offsets. For example, the following code uses both the ParseExact and TryParseExact methods to parse the string representation of a date and time that contains a time zone offset.

[Visual Basic]
Imports System.Globalization

Module Example 
   Sub Main() 
      Dim value As String = "12/15/2009 6:32 PM -05:00"
      Dim fmt As String = "MM/dd/yyyyh:mm tt zzz"
      Dim date1 As DateTime 
      If DateTime.TryParseExact(value, fmt, Nothing, _
                                DateTimeStyles.AllowWhiteSpaces, date1) Then
         Console.WriteLine("Converted '{0}' to {1}, Kind {2}", _
                           value, date1, date1.Kind) 
      Else 
         Console.WriteLine("Unable to convert '{0} to a date.", value)
      End If

      Try 
         Dim date2 As DateTime = DateTime.ParseExact(value, fmt, _
                                 Nothing, DateTimeStyles.AdjustToUniversal) 
         Console.WriteLine("Converted '{0}' to {1}, Kind {2}", _ 
                           value, date2, date2.Kind) 
      Catch e As FormatException 
         Console.WriteLine("Unable to convert '{0} to a date.",
                           value) 
      End Try
   End Sub
End Module

[C#]
using System;
using System.Globalization;

class Example
{
   static void Main()
   {
      string value = @"12/15/2009 6:32 PM -05:00";
      string fmt = @"MM/dd/yyyy h:mm tt zzz";
      DateTime date1;

      if (DateTime.TryParseExact(value, fmt, null,
                   DateTimeStyles.AllowWhiteSpaces, out date1))
         Console.WriteLine("Converted '{0}' to {1}, Kind {2}",
                           value, date1, date1.Kind);
      else
         Console.WriteLine("Unable to convert '{0} to a date.", value);

      try {
         DateTime date2 = DateTime.ParseExact(value, fmt, null,
                          DateTimeStyles.AdjustToUniversal);
         Console.WriteLine("Converted '{0}' to {1}, Kind {2}",
                           value, date2, date2.Kind); 
      }
      catch (FormatException)
      {
         Console.WriteLine("Unable to convert '{0} to a date.", value); 
      } 
   }
}

The example displays the following output:

Converted '12/15/2009 6:32 PM -05:00' to 12/15/2009 3:32:00 PM, Kind Local
Converted '12/15/2009 6:32 PM -05:00' to 12/15/2009 11:32:00 PM, Kind Utc

Note that the parsing method has not only converted a string to a DateTime value, but that in the process it has also performed a time zone conversion. In the first case, the time was adjusted to the time zone of the computer system on which the application was run (which was U.S. Pacific Standard Time in the case of our example). In the second case, because the ParseExact method was called with the styles parameter set to DateTimeStyles.AdjustToUniversal, the string was converted to Coordinated Universal Time.

It is possible to avoid the loss of time zone information in the input string by calling the DateTimeOffset.ParseExact or DateTimeOffset.TryParseExact method rather than the DateTime.ParseExact or DateTime.TryParseExact method, as the following example code shows.

[Visual Basic]
Imports System.Globalization

Module Example
   Public Sub Main()
      Dim value As String = "12/15/2009 6:32 PM -05:00"
      Dim fmt As String = "MM/dd/yyyy h:mm tt zzz"
      Dim date1 As DateTimeOffset

      If DateTimeOffset.TryParseExact(value, fmt, Nothing, _ 
                               DateTimeStyles.AllowWhiteSpaces, date1) Then
         Console.WriteLine("Converted '{0}' to {1}", value, date1)
      Else
         Console.WriteLine("Unable to convert '{0} to a date.", value)
      End If
   End Sub
End Module

[C#]
using System;
using System.Globalization;

public class Example
{
   public static void Main()
   {
      string value = @"12/15/2009 6:32 PM -05:00";
      string fmt = @"MM/dd/yyyy h:mm tt zzz";
      DateTimeOffset date1;

      if (DateTimeOffset.TryParseExact(value, fmt, null,
                         DateTimeStyles.AllowWhiteSpaces, out date1))
         Console.WriteLine("Converted '{0}' to {1}", value, date1);
      else
         Console.WriteLine("Unable to convert '{0} to a date.", value);
   }
}

The example displays the following output:

Converted '12/15/2009 6:32 PM -05:00' to 12/15/2009 6:32:00 PM -05:00

As the third and final example, assume that our application must parse a string in the form “mm/dd/yy hh:mm:ss PST” submitted by a control device. In this case, the time zone is represented by a predefined time zone abbreviation. In this case, the following code successfully parses the string.

[Visual Basic]
Module Example
   Public Sub Main()
      Dim values() As String = {"12/15/2009 6:32 PM PST", _
                                "1/2/2010 4:18 AM PST", _
                                "01/16/2010 11:17 PM PST"}
      Dim fmt As String = "M/d/yyyy h:mm tt PST";
      Dim dt As DateTime

      For Each value As String In values
         Try
            dt = DateTime.ParseExact(value, fmt, Nothing)
            Console.WriteLine("'{0}' --> {1}", value, dt)
         Catch e As FormatException
            Console.WriteLine("'{0}': Bad Format", value)
         End Try
      Next
   End Sub
End Module

[C#]
using System;

public class Example
{
    public static void Main()
    {
        string[] values = { "12/15/2009 6:32 PM PST",
                            "1/2/2010 4:18 AM PST",
                            "01/16/2010 11:17 PM PST" };
        string fmt = "M/d/yyyy h:mm tt PST";
        DateTime dt;

        foreach (string value in values)
        {
            try
            {
                dt = DateTime.ParseExact(value, fmt, null);
                Console.WriteLine("'{0}' --> {1}", value, dt);
            }

            catch (FormatException)
            {
                Console.WriteLine("'{0}': Bad Format", value);
            }
        }
    }
}

The example displays the following output:

'12/15/2009 6:32 PM PST' --> 12/15/2009 6:32:00 PM
'1/2/2010 4:18 AM PST' --> 1/2/2010 4:18:00 AM
'01/16/2010 11:17 PM PST' --> 1/16/2010 11:17:00 PM

In this case, the ParseExact method recognizes only a single time zone abbreviation at the end of the string. By calling another overload that accepts an array of custom format strings, we can extend the example so that it recognizes multiple time zone abbreviations. The following code uses the DateTimeOffset.TryParseExact method for this purpose.

[Visual Basic]
Imports System.Globalization

Module Example
   Public Sub Main()
      Dim values() As String = {"12/15/2009 6:32 PM PST", _
                                 "1/2/2010 4:18 AM PDT", _
                                 "01/16/2010 11:17 PM CST"}
      Dim fmts() As String = {"M/d/yyyy h:mm tt PST", _
                               "M/d/yyyy h:mm tt PDT", _
                               "M/d/yyyy h:mm tt MST", _
                               "M/d/yyyy h:mm tt MDT", _
                               "M/d/yyyy h:mm tt CST", _
                               "M/d/yyyy h:mm tt CDT", _
                               "M/d/yyyy h:mm tt EST", _
                               "M/d/yyyy h:mm tt EDT"}
      Dim dt As DateTimeOffset

      For Each value As String In values
         If DateTimeOffset.TryParseExact(value, fmts, Nothing,
                           DateTimeStyles.None, dt) Then
            Console.WriteLine("'{0}' --> {1}", value, dt)
         Else
            Console.WriteLine("Cannot parse '{0}'", value)
         End If
      Next
   End Sub
End Module

[C#]
using System;
using System.Globalization;

public class Example
{
   public static void  Main()
   {
      string[] values = { "12/15/2009 6:32 PM PST",
                          "1/2/2010 4:18 AM PDT",
                          "01/16/2010 11:17 PM CST" };
      string[] fmts = { "M/d/yyyy h:mm tt PST",
                        "M/d/yyyy h:mm tt PDT",
                        "M/d/yyyy h:mm tt MST",
                        "M/d/yyyy h:mm tt MDT",
                        "M/d/yyyy h:mm tt CST",
                        "M/d/yyyy h:mm tt CDT",
                        "M/d/yyyy h:mm tt EST",
                        "M/d/yyyy h:mm tt EDT" };
      DateTimeOffset dt;

      foreach (string value in values)
      {
         if (DateTimeOffset.TryParseExact(value, fmts, null,
                               DateTimeStyles.None, out dt))
            Console.WriteLine("'{0}' --> {1}", value, dt);
         else
            Console.WriteLine("Cannot parse '{0}'", value);
      }
   }
}

The example displays the following output:

'12/15/2009 6:32 PM PST' --> 12/15/2009 6:32:00 PM -08:00
'1/2/2010 4:18 AM PDT' --> 1/2/2010 4:18:00 AM -08:00
'01/16/2010 11:17 PM CST' --> 1/16/2010 11:17:00 PM -08:00

But since we are now parsing times from different time zones, the output reveals a significant limitation of this example: it doesn’t recognize the time zone abbreviation in our input strings, and so it assumes that the input string represents the data and time in the local system’s time zone (which in this case is in the U.S. Pacific Standard Time zone). We can make our example time zone-aware by using a generic Dictionary object to look up the time zone identifier from its abbreviation, and supplying the identifier as an argument to the TimeZoneInfo.ConvertTimeBySystemTimeZoneId method. The following is the revised, time zone-aware version of our example.

[Visual Basic]
Imports System.Collections.Generic
Imports System.Globalization
Imports System.Text.RegularExpressions

Module Example
   Public Sub Main()
      ' Create dictionary of time zones.
      Dim timeZones As New Dictionary(Of String, String)
      timezones.Add("PST", "Pacific Standard Time")
      timezones.Add("PDT", "Pacific Standard Time")
      timezones.Add("MST", "Mountain Standard Time")
      timezones.Add("MDT", "Mountain Standard Time")
      timezones.Add("CST", "Central Standard Time")
      timezones.Add("CDT", "Central Standard Time")
      timezones.Add("EST", "Eastern Standard Time")
      timezones.Add("EDT", "Eastern Standard Time")

      Dim values() As String = {"12/15/2009 6:32 PM PST", _
                                "1/2/2010 4:18 AM PDT", _
                                "01/16/2010 11:17 PM CST"}

      Dim fmts() As String = {"M/d/yyyy h:mm tt PST", _
                              "M/d/yyyy h:mm tt PDT", _
                              "M/d/yyyy h:mm tt MST", _
                              "M/d/yyyy h:mm tt MDT", _
                              "M/d/yyyy h:mm tt CST", _
                              "M/d/yyyy h:mm tt CDT", _
                              "M/d/yyyy h:mm tt EST", _
                              "M/d/yyyy h:mm tt EDT"}

      Dim rgx As New Regex("\w(D|S)T", RegexOptions.IgnoreCase)
      Dim dt As DateTime
      Dim tz As String = Nothing

      For Each value As String In values
         Try
            tz = rgx.Match(value).Value
            dt = TimeZoneInfo.ConvertTimeBySystemTimeZoneId( _
                             DateTime.ParseExact(value, fmts, Nothing, DateTimeStyles.None), _
                             timeZones.Item(tz), _
                             TimeZoneInfo.Local.Id)
            Console.WriteLine("'{0}' --> {1}", value, dt)
         Catch e As FormatException
            Console.WriteLine("Cannot parse '{0}'", value)
         Catch e As TimeZoneNotFoundException
            Console.WriteLine("Cannot identify time zone {0}", tz)
         End Try
      Next
   End Sub
End Module

[C#]
using System;
using System.Collections.Generic;
using System.Globalization;
using System.Text.RegularExpressions;

public class Example
{
   public static void Main()
   {
      // Create dictionary of time zones.
      Dictionary<string, string> timeZones = new Dictionary<string, string>();
      timeZones.Add("PST", "Pacific Standard Time");
      timeZones.Add("PDT", "Pacific Standard Time");
      timeZones.Add("MST", "Mountain Standard Time");
      timeZones.Add("MDT", "Mountain Standard Time");
      timeZones.Add("CST", "Central Standard Time");
      timeZones.Add("CDT", "Central Standard Time");
      timeZones.Add("EST", "Eastern Standard Time");
      timeZones.Add("EDT", "Eastern Standard Time");

      string[] values = { "12/15/2009 6:32 PM PST",
                          "1/2/2010 4:18 AM PDT",
                          "01/16/2010 11:17 PM CST" };

      string[] fmts = { "M/d/yyyy h:mm tt PST",
                        "M/d/yyyy h:mm tt PDT",
                        "M/d/yyyy h:mm tt MST",
                        "M/d/yyyy h:mm tt MDT",
                        "M/d/yyyy h:mm tt CST",
                        "M/d/yyyy h:mm tt CDT",
                        "M/d/yyyy h:mm tt EST"
                        "M/d/yyyy h:mm tt EDT" };

      Regex rgx = new Regex(@"\w(D|S)T", RegexOptions.IgnoreCase);
      DateTime dt;
      string tz = null;

      foreach (string value in values)
      {
         try {
            tz = rgx.Match(value).Value;
            dt = TimeZoneInfo.ConvertTimeBySystemTimeZoneId(
                                 DateTime.ParseExact(value, fmts, null, DateTimeStyles.None),
                                 timeZones[tz], TimeZoneInfo.Local.Id);
            Console.WriteLine("'{0}'--> {1}", value, dt);
         }
         catch (FormatException) {
            Console.WriteLine("Cannot parse '{0}'", value);
         }
         catch (TimeZoneNotFoundException) {
            Console.WriteLine("Cannot identify time zone {0}", tz);
         }
      }
   }
}

The example displays the following output:

'12/15/2009 6:32 PM PST' --> 12/15/2009 6:32:00 PM
'1/2/2010 4:18 AM PDT' --> 1/2/2010 4:18:00 AM
'01/16/2010 11:17 PM CST' --> 1/16/2010 9:17:00 PM

Hardening Server Applications [Immo]

BCL Team — Fri, 01 Apr 2011 07:04:00 GMT

From time to time a company ships a product that has a huge impact on their ecosystem. A good example for us is certainly .NET. The biggest value proposition that managed code has is that it is, well, managed code. The CLR provides runtime management components such as a garbage collector or reflection that are aimed at reducing the likelihood of bugs and at increasing the developer's productivity. These features allow developers to focus on building their applications instead of tweaking and massaging mechanics. After all, the hope is that by improving the software that is used for developing other software we improve software on a broader scale (how meta!).

Today we are proud to make an announcement that potentially marks a milestone similar to the PDC 2000 announcement of .NET. Several teams at Microsoft worked during the last years on an upcoming product – code named "Source Code".

The Problem

We all have seen it: a customer reports a bug and after debugging it for a while we realize that the bug fix involves changing a single line. Sometimes, the fix only involves fixing a single character, such as the famous off-by-one error where one only needs to replace "<=" by "<".

Quite frequently, small errors can have devastating effects. For example, an Ariane 5 launch vehicle had to self-destruct due to a single casting error where a 64 bit floating point value was converted to a 16 bit integer. This caused the flight computer to make wrong adjustments due an arithmetic overflow.

How "Source Code" works

Research has investigated several strategies to enable computers to learn. One fruitful approach is genetic programming.

In artificial intelligence, Genetic Programming is an evolutionary algorithm-based methodology inspired by biological evolution to find computer programs that perform a user-defined task.

---Wikipedia

The basic idea is that by mutating existing code and applying a selection-function one can automatically find a computer program that solves a given problem.

During the last years the BCL team, the wider CLR team and Visual Studio worked together with Microsoft Research to build a product around this idea. The result, code named "Source Code", is a combination of new technologies with evolved versions of existing technologies such as IntelliTrace, software transactional memory (STM) and Pex. The basic idea is simple. Whenever the CLR discovers an unhandled exception it rolls back the state of the applicationto a point in time prior to the crash (the default is 8 minutes but can be configured). To do this, the CLR uses a full IntelliTrace recording so that the whole runtime state, including the heap, can be properly restored. Then, the stack trace of the unhandled exception is analyzed to detect which method is the most likely culprit. This information is passed to Pex in order to create a permutated version of the method body ("mutation"). This enables new code paths not previously explored in the application. After that, the CLR resumes execution. If the application crashes again, it repeats the above process until a code path is found that avoids the error ("selection"). Over time, successful code changes persist improving the overall fitness of the application ("survival of the fittest").

Picture of the M5 computer in the Star Trek episode "The Ultimate Computer".

As a result of the above process as application will automatically correct itself over time! The brilliant Dr. Richard Daystrom, the designer of the "M5 Multitronic System" (pictured above) would be very pleased to see this advancement.

Currently the technology is in an early prototype state and we are working on removing some restrictions:

Because of the involved downtimes during rollback and mutation, this scenario will only be available to Windows Azure based ASP.NET applications. For the future, we plan to extend the support to client applications as well.
"Source Code" will only save a limited number of applications. We have seen cases where partially working applications are displaced from the cache by more buggy applications. Our next goal is to improve the cache policy to avoid these situations.
The early CTP does not include support for management and monitoring but we are actively working to get "Source Code" integrated with the Microsoft System Center products.

You can download a CTP here.

Optimizing Regex Performance, Part 3 [Ron Petrusha]

BCL Team — Tue, 29 Mar 2011 04:55:00 GMT

Regular expressions in the .NET Framework support a number of grouping constructs, which allow a regular expression pattern to be grouped into one or more subexpressions. Grouping constructs are essential for creating backreferences, as well as for defining a subexpression to which a quantifier is applied.

The Performance Impact of Capturing Groups

The most commonly used grouping constructs in the .NET Framework regular expression language are (subexpression), which defines a numbered capturing group, and (?<name> subexpression), which defines a named capturing group. The use of these language elements, however, has a definite cost; they cause the GroupCollection object returned by the Match.Groups property to be populated with the most recent unnamed or named captures, and if a single grouping construct has captured multiple substrings in the input string, they also populate the CaptureCollection object returned by the Group.Captures property of a particular capturing group with multiple Capture objects.

For example, the regular expression (\b(\w+)[;,]?\s?)+([.?!]) is designed to capture an entire sentence, and the subexpression (\w+) is designed to capture the individual words in that sentence. (See Regular Expression Language Elements for reference documentation for the regular expression language supported by the .NET Framework.) The regular expression defines three capturing groups, which are listed in the following table. In addition, the first group in the GroupCollection object (at index 0) contains the complete pattern match; the values of its Group.Index, Group.Length, Group.Success, and Group.Value properties are identical to the values of its parent Match object.

Index	Pattern	Description
1	`(\b(\w+)[;,]?\s?)`	Captures a word in the sentence, along with any following punctuation mark and trailing space. The + quantifier is applied to the group so that the regular expression engine will advance beyond the first word in the input string to capture all words in the sentence.
2	`(\w+)`	Captures a word in the sentence
3	`([.?!])`	Captures the punctuation mark used to end the sentence.

The following example uses this regular expression to extract a sentence, its individual words, and its ending punctuation mark from an input string.

[Visual Basic]
Imports System.Text.RegularExpressions

Module Example
   Public Sub Main()
      Dim pattern As String = "(\b(\w+)[;,]?\s?)+([.?!])"

      Dim input As String = "This is a short, silly sentence."

      Dim match As Match = Regex.Match(input, pattern)
      If match.Success Then

         Console.WriteLine("'{0}' found at position {1}",
                           match.Value, match.Index)

         If match.Groups.Count > 1 Then
            For ctr As Integer = 1 To match.Groups.Count - 1
               Dim grp As Group = match.Groups(ctr)
               Console.WriteLine("   Group {0}: '{1}'",
                                 ctr, grp) 

               For cctr As Integer = 0 To grp.Captures.Count - 1
                  Console.WriteLine("Capture {0}: '{1}'",
                                    cctr, grp.Captures(cctr).Value)
               Next
            Next
         End If
      End If
   End Sub
End Module

[C#]
using System;
using System.Text.RegularExpressions;

public class Example
{
   public static void Main()
   {
      string pattern = @"(\b(\w+)[;,]?\s?)+([.?!])";
      string input = "This is a short, silly sentence.";

      Match match = Regex.Match(input, pattern);
      if (match.Success) {
         Console.WriteLine("'{0}' found at position {1}",
                           match.Value, match.Index);
         if (match.Groups.Count > 1)
            for (int ctr = 1; ctr <= match.Groups.Count - 1; ctr++) {
               Group grp = match.Groups[ctr];
               Console.WriteLine("   Group {0}: '{1}'", ctr, grp);  
               for (int cctr = 0; cctr <= grp.Captures.Count - 1; cctr++)
                  Console.WriteLine("     Capture {0}: '{1}'", 
                                    cctr, grp.Captures[cctr].Value);
            }
      }
   }
}

The example displays the following output. Note that the CaptureCollection captured by the first group captures each word in the sentence along with a following punctuation character if it does not conclude the sentence, the CaptureCollection captured by the second group contains each word in the sentence, and the final group contains only the sentence’s final punctuation mark.

'This is a short, silly sentence.' found at position 0
   Group 1: 'sentence'
      Capture 0: 'This '
      Capture 1: 'is '
      Capture 2: 'a '
      Capture 3: 'short, '
      Capture 4: 'silly '
      Capture 5: 'sentence'
   Group 2: 'sentence'
      Capture 0: 'This'
      Capture 1: 'is'
      Capture 2: 'a'
      Capture 3: 'short'
      Capture 4: 'silly'
      Capture 5: 'sentence'
   Group 3: '.'
      Capture 0: '.'

In this regular expression, our first capturing group is present only so that a quantifier can be applied to it. This allows the regular expression engine to advance from one word to the next, and so to capture an entire sentence. The second group is useful if we want to programmatically capture the individual words from the sentence, which are available from the CaptureCollection object. The third is useful if we want to determine which punctuation symbol ended the sentence; it is available from the GroupCollection object.

We can disable captures (so that the GroupCollection and CaptureCollection objects are not populated as matches are found) by using the (?:subexpression) language element. We can assess the impact that capturing groups have on performance by using a StopWatch object to measure the execution time of the following three similar regular expressions when we use each to extract all of the sentences in the text of Theodore Dreiser's The Financier:

The original regular expression, (\b(\w+)[;,]?\s?)+([.?!]).
A regular expression that disables captures by the first group, (?:\b(\w+)[;,]?\s?)+([.?!]). This group exists only so that a quantifier could be applied to it.
A regular expression that has no capturing groups, (?:\b(?:\w+)[;,]?\s?)+[.?!]. It captures the entire sentence only, and disables all capturing groups. (It actually returns a Match object whose Groups property returns a GroupCollection object with a single Group object. This group at index 0 represents the entire match; it is always present and cannot be suppressed.

The source code for the example is:

[Visual Basic]
Imports System.Diagnostics
Imports System.IO

Imports System.Text.RegularExpressions

Module Example
   Public Sub Main()
      Dim patterns() As String = { "(\b(\w+)[;,]?\s?)+([.?!])", 
                                   "(?:\b(\w+)[;,]?\s?)+([.?!])",
                                   "(?:\b(?:\w+)[;,]?\s?)+[.?!]" }
      Dim input As String = (New StreamReader(".\Dreiser_TheFinancier.txt")).ReadToEnd()
      Dim sw As Stopwatch

      For Each pattern In patterns
         ' Instantiate the regular expression object before measuring performance.
         Dim regex As New Regex(pattern)

         ' Time matches.

         sw = Stopwatch.StartNew()

         Dim ctr As Integer = 0
         Dim match As Match = Regex.Match(input, pattern)
         Do While match.Success
            ctr += 1
            match = match.NextMatch()
         Loop

         sw.Stop()

         ' Display performance information.
         Console.WriteLine("'{0}' ({1} capturing groups): {2:N0} matches in {3:ss\.fffff}",
                           regex.ToString(), regex.GetGroupNumbers().Length,
                           ctr, sw.Elapsed)
         Console.WriteLine()
      Next
   End Sub
End Module

[C#]
using System;

using System.Diagnostics;
using System.IO;
using System.Text.RegularExpressions;

public class Example
{
   public static void Main()
   {
      string[] patterns = { @"(\b(\w+)[;,]?\s?)+([.?!])",
                            @"(?:\b(\w+)[;,]?\s?)+([.?!])",
                            @"(?:\b(?:\w+)[;,]?\s?)+[.?!]" };
      string input = (new StreamReader(@".\Dreiser_TheFinancier.txt")).ReadToEnd();
      Stopwatch sw;

      foreach (var pattern in patterns) {
         // Instantiate the regular expression object before measuring performance.

         Regex regex = new Regex(pattern);

         // Time matches.
         sw = Stopwatch.StartNew();

         int ctr = 0;
         Match match = Regex.Match(input, pattern);
         while (match.Success) {
            ctr++;
            match = match.NextMatch();
         }
         sw.Stop();

         // Display performance information.

         Console.WriteLine("'{0}' ({1} capturing groups): {2:N0} matches in {3:ss\\.fffff}\n",
                           regex.ToString(), regex.GetGroupNumbers().Length,
                           ctr, sw.Elapsed);
      }  
   }   
}

As the following output from the example shows, populating the GroupCollection and CaptureCollection objects has a significant impact on the regular expression engine’s performance. The regular expression with three capturing groups executed slightly more than 15faster than the one with four capturing groups, and the regular expression with just one capturing group executed more than 25% faster than the one with four. The poorer performance of the regular expression with four capturing groups is a legitimate cost if programmatic access to these captures is required. However, if it is not, populating the complete regular expression object model constitutes unnecessary overhead.

'(\b(\w+)[;,]?\s?)+([.?!])' (4 capturing groups): 12,633 matches in 02.4953428
'(?:\b(\w+)[;,]?\s?)+([.?!])' (3 capturing groups): 12,633 matches in 02.0867863
'(?:\b(?:\w+)[;,]?\s?)+[.?!]' (1 capturing groups): 12,633 matches in 01.7976809

Capturing and Non-Capturing Grouping Constructs

The regular expression language in the .NET Framework supports a number of grouping constructs, which are documented in detail in the Grouping Constructs topic in the MSDN Library. The following table lists each grouping construct and indicates whether it captures the text matched by the group.

Language Element	Description	Capturing or Non-capturing
`(subexpression)`	Captures a group, which can be referenced by number or retrieved from the GroupCollection object by its ordinal position.	Capturing
`(?<name>subexpression)`	Captures a named group, which can be referenced by name or by number or retrieved from the GroupCollection object by its name or ordinal position.	Capturing
`(?<name1-name2>subexpression)`	Balancing group definition. Tracks nested constructs such as opening and closing parentheses or opening and closing brackets. Capturing groups is integral to the operation of this language element.	Capturing
`(?:subexpression)`	Non-capturing group. Typically, it is used to define a subexpression to which a quantifier can be applied.	Non-capturing
`(?imnsx-imnsx:subexpression)`	Defines options (such as case-sensitivity or the significance of white space) that apply to `subexpression`.	Non-capturing
`(?=subexpression)`	Positive lookahead assertion. Looks ahead of the current position in the input string to determine if `subexpression` is matched.	Non-capturing
`(?!subexpression)`	Negative lookahead assertion. Looks ahead of the current position in the input string to determine if `subexpression` is not matched.	Non-capturing
`(?<=subexpression)`	Positive lookbehind assertion. Looks behind the current position in the input string to determine if `subexpression is matched.`	Non-capturing
`(?<!subexpression)`	Negative lookbehind assertion. Looks behind the current position in the input string to determine if `subexpression` is not matched.	Non-capturing
`(?>subexpression)`	Non-backtracking subexpression. Does not surrender any characters from a matched subexpression to ensure that the regular expression pattern as a whole is matched.	Non-capturing

Disabling Captures

You can disable all captures in a group, or you can disable all but named captures. Unnamed captures can be disabled for a regular expression as a whole, for a part of a regular expression, or for a particular subexpression.

As the previous example showed, the (?:subexpression) language construct disables captures by a group. It disables captures only for the group to which it applies; it does not disable captures by nested or inner groups. In the previous example, for instance, the regular expression pattern (?:\b(\w+)[;,]?\s?)+([.?!]) disables captures by the outer group (group 1). The inner group (\w+) (group 2) is not affected and remains a capturing group.

You can also control implicit captures by using regular expression language options. These options are specified either as members of the RegexOptions enumeration or as inline characters in an options language construct. For more information, see Regular Expression Options.

The RegexOptions.ExplicitCapture option or its corresponding n language option can be used to disable all unnamed or implicit captures; only matches from named groups defined by the (?<name>subexpression) language element are captured.

The RegexOptions.ExplicitCapture option disables implicit captures by the regular expression engine for a particular regular expression or for a particular matching operation. It disables implicit captures for the regular expression if it is passed to the Regex class constructor, and it disables implicit captures for a particular matching operation if it is passed to a static Regex matching method, such as Regex.Match or Regex.Matches. The following example illustrates this. It changes the previous regular expression pattern to (\b(?<words>\w+)[;,]?\s?)+([.?!]) so that it has two implicit capturing groups and one named capturing group. When the Regex object is instantiated with its options parameter set to RegexOptions.ExplicitCapture, only matches made by the named group are captured.

[Visual Basic]
Imports System.Text.RegularExpressions

Module Example

   Public Sub Main()
      Dim pattern As String = "(\b(?<words>\w+)[;,]?\s?)+([.?!])"
      Dim input As String = "This is a short, silly sentence."

      Dim rgx As New Regex(pattern, RegexOptions.ExplicitCapture)
      Dim match As Match = rgx.Match(input)

      If match.Success Then

         Console.WriteLine("'{0}' found at position {1}.", 
                           match.Value, match.Index)
         Console.WriteLine("There are {0} captured groups:", 
                           match.Groups.Count)
         Dim ctr As Integer = 0
         For Each grp As Group In match.Groups
            Console.WriteLine("   Group <{0}>: '{1}' at position {2}.", 
                              rgx.GroupNameFromNumber(ctr), grp.Value,
                              grp.Index)
            Dim capCtr As Integer = 1
            For Each capture In grp.Captures
               Console.WriteLine("      Capture {0}: '{1}' at position {2}.", 
                                 capCtr, capture.Value, capture.Index)
               capCtr += 1
            Next

            ctr += 1
         Next
      End If
   End Sub
End Module

[C#]
using System;
using System.Text.RegularExpressions;

public class Example
{
   public static void Main()
   {
      string pattern = @"(\b(?<words>\w+)[;,]?\s?)+([.?!])";
      string input = "This is a short, silly sentence.";
      Regex rgx = new Regex(pattern, RegexOptions.ExplicitCapture);
      Match match = rgx.Match(input);

      if (match.Success) {
         Console.WriteLine("'{0}' found at position {1}.",
                           match.Value, match.Index);
         Console.WriteLine("There are {0} captured groups:",
                           match.Groups.Count);
         int ctr = 0;
         foreach (Group grp in match.Groups) {
            Console.WriteLine("   Group <{0}>: '{1}' at position {2}.",
                              rgx.GroupNameFromNumber(ctr++), grp.Value, 
                              grp.Index);
            int capCtr = 1;
            foreach (Capture capture in grp.Captures)  
               Console.WriteLine("      Capture {0}: '{1}' at position {2}.",  
                                 capCtr++, capture.Value, capture.Index);
         }
      }
   }
}

The example displays the following output:

'This is a short, silly sentence.' found at position 0.
There are 2 captured groups:
   Group <0>: 'This is a short, silly sentence.' at position 0.
      Capture 1: 'This is a short, silly sentence.' at position 0.
   Group <words>: 'sentence' at position 23.
      Capture 1: 'This' at position 0.
      Capture 2: 'is' at position 5.
      Capture 3: 'a' at position 8.
      Capture 4: 'short' at position 10.
      Capture 5: 'silly' at position 17.
      Capture 6: 'sentence' at position 23.

You can also disable implicit captures for part of a regular expression pattern by using the (?n:subexpression) grouping construct. It disables implicit captures for all implicit groups defined in subexpression. If the construct is applied to the entire regular expression pattern, it is equivalent to instantiating a Regex object with the RegexOptions.ExplicitCapture option. For example, the output would remain unchanged if the regular expression pattern in the previous example were replaced by (?n:(\b(?<words>\w+)[;,]?\s?)+([.?!])). On the other hand, the regular expression pattern (?n:((a+)(b+))+)(c.) disables captures in the subexpression ((a+)(b+))+), including in the two nested expressions in the subexpression.

The (?n) inline element can also be used to disable implicit captures from the point that it is encountered in a regular expression to the end of the expression, or until the implicit captures only option is disabled by the (?-n) inline option. (Note, though, that the (?n:subexpression) syntax can be used for the same result and is significantly clearer.) For example, the regular expression pattern (?n)(\b(?<words>\w+)[;,]?\s?)+(?-n)([.?!]) disables implicit captures in the subexpression (\b(?<words>\w+)[;,]?\s?)+ but enables them in the subexpression ([.?!]). As a result, a match has two capturing groups. The first captures the punctuation mark used to end a sentence. The second captures each word in the sentence.

Summary

The regular expression engine incurs a substantial performance hit when capturing groups appear in a regular expression. In cases where captures are not needed and grouping constructs are used primarily so that quantifiers can be applied to them, captures should be disabled. The regular expression language in the .NET Framework includes a non-capturing grouping construct, (?:subexpression). In addition, implicit or unnamed captures can be disabled by supplying the RegexOptions.ExplicitCapture option to a Regex class constructor or static matching method, or by using the by using the (?n:subexpression) grouping construct or the (?n) inline element.

Announcing Portable Library Tools CTP [Justin Van Patten]

BCL Team — Wed, 19 Jan 2011 20:42:29 GMT

Today we're announcing the CTP release of a new add-in for Visual Studio 2010 that makes it easy to create C# and Visual Basic libraries that run on a variety of .NET platforms without recompilation.

Download the Portable Library Tools CTP today (install VS 2010 SP1 Beta first).

Creating Portable Libraries

The Portable Library Tools CTP adds a new "Portable Class Library" project template to Visual Studio that can be used to create class libraries in C# and VB that run on the various .NET platforms without recompiling.

Portable Class Library projects and assemblies can be referenced and used by .NET 4, Silverlight 4, Windows Phone 7, and XNA for Xbox 360 projects.

Target Platforms Determine Available APIs

Within a Portable Class Library project, you can specify which .NET platforms the library is intended to run on. Although there is a 'core' set APIs that are available on all platforms, there are also some APIs that are only available on certain platforms. An example of an API that exists on some platforms but not others is MEF. Windows Phone and Xbox 360 do not currently have built-in support for MEF, but .NET and Silverlight do. If you want to use MEF within a portable library, the library will currently only run on .NET and Silverlight.

Thus, the project's selected target platforms determine which APIs can be used within the project. The APIs that are shown in intellisense and available to the compiler are automatically filtered based on the selected target platforms, so you don't need to have any special knowledge about each platform or worry about inadvertently using an API that doesn't exist on a platform that you're targeting. The project system takes care of this for you.

In the future, as the underlying platforms evolve to support more APIs that could be portable, we'll update the API surface available to portable libraries as appropriate.

The following table summarizes the high-level functionality currently available on each platform:

Feature	.NET	Silverlight	Windows Phone	Xbox 360
Core BCL	Yes	Yes	Yes	Yes
Core XML	Yes	Yes	Yes	Yes
Core HTTP	Yes	Yes	Yes	-
WCF Client	Yes	Yes	Yes	-
MEF	Yes	Yes	-	-

Note: Once you've installed the Portable Library Tools CTP, you can use the Object Browser in Visual Studio for a more in-depth look at the individual APIs available for use within portable libraries.

You can write a surprising amount of portable code with the subset of APIs currently available to portable libraries. Everything from business logic to web service client proxies can be portable. This enables you to implement most, if not all, application logic in portable assemblies (and then use the portable assemblies in apps that have form-factor-specific UIs for a given platform).

We're still in the process of defining the APIs that will be available to portable libraries, so expect some minor tweaks and additions before the final release of the tools. We'd love to get your feedback on what's currently available in the CTP. Is there anything you think is missing that should be available to portable libraries? How easy was it to make your existing libraries portable?

Known Limitations in the CTP

Deploying Portable Libraries

The CTP of the Portable Library Tools is not a "go live" release. There are still some rough edges that need smoothing out before the final release. One area that does not currently have a complete experience is around deploying portable libraries.

Making portable libraries work on platforms such as .NET Framework 4 and Silverlight 4 required making some minor changes to those platforms. In order for portable libraries to work correctly on these platforms, you'll need to ensure the necessary updates have been applied. Silverlight for Windows Phone 7 and XNA Framework 4.0 for Xbox 360 do not require any updates to run portable assemblies.

.NET Framework 4 Update

An update to .NET Framework 4 is required to run portable libraries. For developers, this update is automatically installed as part of Visual Studio SP1 Beta, so you shouldn't need to install it on a machine that already has VS 2010 SP1 installed. A standalone beta of this update is currently available for download here. The final release of the Portable Library Tools will provide a way for developers to add a prerequisite to their installer to ensure the update is applied on end-user machines when your app is deployed.

Silverlight 4 Update

An update to Silverlight 4 is required to run portable libraries. This update hasn't been released yet, but the impact in the meantime should be limited. As long as you avoid using the following APIs in your portable libraries, they will run just fine on Silverlight 4 without the update:

System.Xml.XmlReader.Dispose
System.Xml.XmlWriter.Dispose
System.Net.WebResponse.Dispose

In the meantime, if you need to dispose these objects, you can wrap them in a using statement or cast the object to IDisposable before calling Dispose.

Once the Silverlight 4 update is available, we will provide instructions on how to specify it as the minimum version required for your application.

Visual Basic Support

The Add Service Reference is currently disabled for VB projects. This will be resolved in a future update.

Summary

We're excited to make a CTP of the Portable Library Tools available for you to try out. Download the CTP today and let us know what you think!

Also, you may be interested in watching Shawn Burke's PDC 2010 session on 3-Screen Coding: Sharing code between Windows Phone, Silverlight, and .NET, which provides some more background on portable libraries.

Time-travel with .NET or DateTime, DateTimeOffset and the lost DST hour [Greg]

BCL Team — Sun, 28 Nov 2010 20:05:00 GMT

Every year again comes the DST change...

And every year again do we need to work with customers on helping them understand some legacy design decisions, and how to work around these ghosts of the past.

Recently, around the end of October and the start of November the daylight saving time (DST) period finished in most areas of the northern hemisphere. As if it was an attempt to make up for the oncoming cold weather, we set our clocks back and got an extra hour of sleep. (In the southern hemisphere, the DST begins at the end of the year, and they turn their clocks forwards.) And one of the basic .NET types – DateTime – does not play well with the annual end-of-DST transition. This issue is well-known and so is the workaround, but nevertheless it gets reported to us regularly by customers who rediscover the issue, especially around the DST transition season. There are a number of resources that treat this topic in great detail. Some of these resources are listed below. Here, I provide some of the background for the reasons behind the issues and a brief refresher on how to solve the problems.

First the gist: The DateTimeOffset type helps avoiding most of the problems associated with DateTime. If you have trouble with DateTime in the context of a DST transition, consider using DateTimeOffset instead.

Issue: Incorrect time zone display.

One issue with the DateTime structure is that during transition from summer time to winter time (i.e. from DST to no-DST), the winter-time time-zone-offset it displayed an hour too early (when shown using a default built-in functionality such as DateTime.Now.ToString("o") ). However, the actual clock time is always displayed correctly. This symptom is caused by the interplay of the fact that DateTime stores the local time only, and the nature of DST transitions.

Similar to the DateTime type, the Windows DST database, on which Windows and .NET applications base their time calculations, specifies the transition times in local time (not UTC). During summer-to-winter time transition in the northern hemisphere the clocks are typically set one hour back. Assume, w.l.o.g., that the transition occurs at 3:00. Then, the times between 2:00 and 2:59 will occur twice: once during summer time, and once again during winter time after the clock has been set back. When the system is queried whether a time, say, 2:28, is in summer (say UTC -7) or in winter time (UTC -8), there is no way for the system to know which one is correct. In such cases, .NET assumes winter time, as that is typically the default time for the time zone (i.e. no DST). Note that if we changed this behaviour, the problem would still occur, only the other way around. For instance, the old DST state would show up one hour too long.

In this particular case, the actual time is always shown correctly, it is just the DST state that is incorrect.

Solution: Track the DST mode changes.

At one second past the hour (or any other pre-specified time), store the local system time. Then compare the current time against the timestamp stored when the one-second-part-the-hour event occurred previously. If they match, we know that we just transitioned to winter time: the offset has changed (e.g. using the previous example, to UTC-8). Otherwise, we are still in summer time: it is still UTC-7.

This workaround is complex and applies only to applications that run for at least one hour. This is not the best approach for a general fix.

Better solution: Use a type that stores time as UTC instead of local time.

UTC (coordinated universal time) describes the time at the Greenwich meridian and does not undergo any DST transitions. In a way (simplified but appropriate for this context), it can be understood as the “universal cosmological time”. The DateTimeOffset type stores the local time as a pair of values: one value describes the UTC time, and the other value describes the offset between UTC and the local time. For instance, 02:28 in the UTC-7 time zone is stored as {UtcTime - Offset} = {10:28 - 420minutes}. 02:28 in the UTC-8 time zone is stored as {11:28 - 480minutes}. In contrast to DateTime, these two values, although describing the same local time, describe two different points in time, and can be easily differentiated from each other. As a result, issues related to DST transitions can be handled correctly when using DateTimeOffset.

Putting it all together: Recommendations.

The reading list below contains a wealth of background information on the DateTime and DateTimeOffset types and how to use them. Here is a short and practical summary of how these types should be used.

Use `DateTime` to:	Use `DateTimeOffset` to:
Represent dates only (no time information). Represent times only (not on a specific date). (Alternatively, consider using the `TimeSpan` type to represent the time of day without a specific date context). Work with abstract times and dates, such as historical dates. Work with legacy APIs that do not support `DateTimeOffset` (e.g. some database and file system APIs).	Uniquely and unambiguously identify a single point in time. Log events occurring at specific times. Perform general date and time arithmetic. Work with time zones. Work with daylight saving times.

More guidance on choosing the best date and time type for your circumstances is provided through the links below.

The Caveats of Time Zone Names [Greg]

BCL Team — Tue, 12 Oct 2010 01:18:00 GMT

Hi Everyone!

My name is Greg, I joined the NetFx Base Class Libraries (BCL) team a few months ago, and I am really excited about my work on some very cool new framework features. Before joining the BCL team I was working in academic research in complex systems, network theory and artificial life at Monash University in Australia, and previously I was running a small software development and consulting company with clients in Germany and the UK.

Besides adding some awesome new stuff to the next release for the framework (can’t talk about that yet!) I am also maintaining and improving some of the existing namespaces. This involves fixing some bugs and helping our customers by explaining how to use the framework on the Connect portal. There are a few issues that come up regularly and are particularly tricky to deal with, so I would like to talk about them briefly to a wider audience and in a more generic context (i.e. here). Today I will be talking about time zones and, in particular, potential pitfalls associated with their naming.

In .NET, the class that provides most of the functionality for dealing with time zones, converting times from one zone to another, and working with daylight saving time (DST), is System.TimeZoneInfo. In turn, the data that TimeZoneInfo uses for its calculations is provided by Windows in the registry under this key:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Time Zones

(Note: be very careful if you touch the registry, never change anything if you are not absolutely sure what you are doing, and always create a back-up before your changes.)

In order to access information about a specific time zone in a .NET application, you need to instantiate a TimeZoneInfo object that represents that time zone using the factory method TimeZoneInfo.FindSystemTimeZoneById(String id) (see also here for more information). That factory method takes a string parameter specifying a time zone name and returns a TimeZoneInfo instance that represents the specified time zone. It provides information including offset from the universal standard time (UTC), information about DST adjustments, time conversion routines and more.

The difficulty is in providing the correct ID (name) of a time zone in order to get hold of the corresponding object: The naming for the time zones is not always intuitive. As a result, it is easy to get confused and to create program bugs and unexpected behaviour. Consider for instance the zone named "US Eastern Standard Time". Unexpectedly this is not the EST/EDT observed on most of the US east coast. Instead, it is a US-Indiana-Only time zone that is not used any more since 2006 and does not change to DST. The name of the time zone that people are probably looking for instead is "Eastern Standard Time". That is the ID you need to get the TimeZoneInfo object that represents the time zone used in most of east North America. That time zone does use DST.

Another time zone with a misleading name is "Greenwich Standard Time". This zone applies to places like Monrovia (Liberia) and Reykjavik (Island). These places do not observe DST in 2010. This zone is easily confused with "GMT Standard Time" that contains Portugal and the UK and does observe DST.

To exemplify the issue, consider the following code sample. Its output is given below.

using System;
 
namespace Microsoft.Misc.MSDNBlog {
public static class ConfusingTimeZoneNames_Demo {
 
private static String[] zoneIDs = new String[] {
            "China Standard Time",
            "AUS Eastern Standard Time",
            "Pacific Standard Time",
            "US Eastern Standard Time",
            "Eastern Standard Time",
            "Greenwich Standard Time",
            "GMT Standard Time"
        };
 
 
public static void Main(String[] unusedArgs) {
 
  const String format = "{0,-26} | {1} {2}| {3}";
  DateTime utcTime = new DateTime(2010, 06, 01, 12, 30, 0);
 
  Console.WriteLine(format, "Time Zone Name (ID)",
                            "Local Time      ",
                            "DST?",
                            "Time Zone Information");
  Console.WriteLine("---------------------------+------------"
          + "----------+---------------------------------------");
 
  foreach (String tzID in zoneIDs) {
 
    TimeZoneInfo tzInfo = TimeZoneInfo.FindSystemTimeZoneById(tzID);
    DateTime tzTime = TimeZoneInfo.ConvertTime(utcTime, tzInfo);
 
    Console.WriteLine(format, tzID,
              tzTime.ToString("yyyy-MM-dd HH:mm"),
              tzInfo.IsDaylightSavingTime(tzTime) ? "DST " : "    ",
              tzInfo.DisplayName);
  }        
}
 
}  // public static class ConfusingTimeZoneNames_Demo
}  // namespace Microsoft.Misc.MSDNBlog

To summarise: Be careful when instantiating TimeZoneInfo objects. Always double-check that the zone name you are using is the correct one, even if it appears obvious at first. And if your application behaves weirdly, double-check if you have possibly created the wrong TimeZoneInfo object due to a similarity in name IDs. For more information about time zones in various places around the world, check out http://www.timeanddate.com/worldclock/full.html

Enjoy!

Optimizing Regular Expression Performance, Part II: Taking Charge of Backtracking [Ron Petrusha]

BCL Team — Tue, 03 Aug 2010 20:50:00 GMT

One of the most powerful features of regular expressions in the .NET Framework -- and of Nondeterministic Finite Automaton (NFA) regular expression engines generally -- is their ability to execute in a non-linear manner. That is, instead of advancing one character at a time, they can return to a previous saved state in order to match a regular expression pattern. For example, if you try to match the string “A match.” with the regular expression pattern .+\. the regular expression engine first attempts to match the entire string. It then retreats one character so that the .+ construct matches “A match” and \. matches the period at the end of the string. This ability to return to a previous saved state in order to continue the match is known as backtracking.

Note: This is the second part of a three-part series on optimizing regular expression performance. The first part, Optimizing Regular Expression Performance, Part I: Working with the Regex Class and Regex Objects, discusses the efficient use of Regex static and instance methods.

Although backtracking contributes to the power and flexibility of regular expressions, it can have an enormous negative impact on performance. In a worst-case scenario, when backtracking is used excessively, it might take one or more days for the regular expression engine to process even a relatively short input string that does not match the regular expression pattern.

The NFA Engine places control over backtracking (and therefore responsibility for the consequences of backtracking) in the hands of regular expression developers who may be unaware of how backtracking affects performance. This fact often goes overlooked both when creating regular expressions and when assessing regular expression performance. As a result, the inappropriate use of backtracking is one of the fundamental performance bottlenecks that applications relying on regular expressions encounter.

Backtracking occurs when a regular expression contains the following language constructs and under the following conditions:

An alternation construct, if the first alternative does not provide a successful pattern match.
An indeterminate quantifier (such as * or +), when the greedy match resulting from the expression to which the quantifier is applied prevents a successful match.
A conditional expression, if the first condition does not provide a successful pattern match.

When the regular expression engine encounters one of these constructs or expressions, it saves its current state so that it can return to it if necessary.

We’ll first examine backtracking with alternation, and then examine backtracking with nested quantifiers. We’ll finish by looking at a worst case scenario in which a poor use of backtracking causes execution time to increase exponentially as the number of characters in the string to be matched increases. In each case, we’ll attempt to show that an alternative pattern that better controls backtracking offers superior performance.

Note: In the code examples in this article, we make extensive use of the System.Diagnostics.Stopwatch class to measure regular expression performance. These measurements are used only for comparative purposes, and reflect performance on a particular system. They are not intended to be construed as benchmarks.

Alternation and Backtracking

The alternation construct (|) allows a regular expression pattern to include the equivalent of an Or operator; the regular expression engine will match text that conforms to one of two or more patterns. The regular expression engine performs no optimization on alternation constructs.

Because alternation is a backtracking construct, it can entail a significant performance penalty if the alternation pattern is not well crafted. In this section, we’ll examine some of the factors to consider when constructing an alternation pattern that will perform efficiently.

Order Matters

Alternation constructs are evaluated sequentially from left to right. The second alternative is evaluated only if the first alternative fails. If there are three or more alternatives, the third alternative is evaluated only if the first and second alternatives fail, and so on. Because of this, the ordering of items in an alternation construct is significant. Subpatterns that are more likely to be encountered in an input string should precede subpatterns that are less likely to be encountered.

The following example uses an alternation construct to find words followed by one of a number of punctuation symbols in the text of Theodore Dreiser’s The Financier. In the first regular expression pattern, punctuation symbols that are less likely to be encountered precede symbols that are more frequently encountered. In the second, punctuation symbols that are more likely to be encountered precede symbols that are less frequently encountered.

[Visual Basic]
Option Strict On

Imports System.Diagnostics
Imports System.IO
Imports System.Text.RegularExpressions

Module Example
   Public Sub Main()
      ' Organize the punctuation marks in a poor order.
      Dim pattern1 As String = "\b\w+(;|:|!|\?|\.|,)\s"
      Dim rgx1 As New Regex(pattern1, RegexOptions.SingleLine)
      Dim sw As Stopwatch
      Dim sr As New StreamReader(".\Dreiser_TheFinancier.txt")
      Dim input As String = sr.ReadToEnd()
      sr.Close()

      sw = Stopwatch.StartNew()
      Dim match1 As Match = rgx1.Match(input)
      Dim ctr1 As Integer
      Do While match1.Success
         ctr1 += 1
         match1 = match1.NextMatch
      Loop
      sw.Stop()
      Console.WriteLine("Found {0} matches in {1}", ctr1, sw.Elapsed)

      ' Organize the punctuation marks in rough order of occurrence.
      Dim pattern2 As String = "\b\w+(\.|,|\?|;|:|!)\s"
      Dim rgx2 As New Regex(pattern2, RegexOptions.SingleLine)

      sw = Stopwatch.StartNew()
      Dim match2 As Match = rgx2.Match(input)
      Dim ctr2 As Integer
      Do While match2.Success
         ctr2 += 1
         match2 = match2.NextMatch
      Loop
      sw.Stop()
      Console.WriteLine("Found {0} matches in {1}", ctr2, sw.Elapsed)
   End Sub
End Module

[C#]
using System;
using System.Diagnostics;
using System.IO;
using System.Text.RegularExpressions;

public class Example
{
   public static void Main()
   {
      // Organize the punctuation marks in a poor order.
      string pattern1 = @"\b\w+(;|:|!|\?|\.|,)\s";
      Regex rgx1 = new Regex(pattern1, RegexOptions.Singleline);
      Stopwatch sw;
      StreamReader sr = new StreamReader(@".\Dreiser_TheFinancier.txt");
      string input = sr.ReadToEnd();
      sr.Close();

      sw = Stopwatch.StartNew();
      Match match1 = rgx1.Match(input);
      int ctr1 = 0;
      while (match1.Success)
      {
         ctr1++;
         match1 = match1.NextMatch();
      }
      sw.Stop();
      Console.WriteLine("Found {0} matches in {1}", ctr1, sw.Elapsed);

      // Organize the punctuation marks in rough order of occurrence.
      string pattern2 = @"\b\w+(\.|,|\?|;|:|!)\s";
      Regex rgx2 = new Regex(pattern2, RegexOptions.Singleline);

      sw = Stopwatch.StartNew();
      Match match2 = rgx2.Match(input);
      int ctr2 = 0;
      while (match2.Success)
      {
         ctr2++;
         match2 = match2.NextMatch();
      }
      sw.Stop();
      Console.WriteLine("Found {0} matches in {1}", ctr2, sw.Elapsed);
   }
}

Output from the example shows that there is a small performance benefit from ordering the alternatives based on how likely they are to be encountered in the input string. The performance improvement is approximately 6% for this very simple regular expression, as the following graph shows. In a more complex regular expression pattern, particularly if he alternation pattern is followed by additional backtracking constructs, the performance benefit is likely to be much greater.

Found 26930 matches in 00:00:00.9649407
Found 26930 matches in 00:00:00.9044185

Optimization Matters

Because alternation constructs involve backtracking, optimization is critical. Alternation patterns that are poorly written and that include duplicate elements perform much more poorly than those that do not. In the following example, the regular expression \b(the|this|that|then|there|these|those)\b matches a number of words starting with “th” (the, this, that, then, there, these, those) in a string. Because of backtracking, the regular expression engine performs the following steps to match the substring “these” in the input string:

1	It determines that the current position in the input string is on a word boundary. Because a grouping and alternation construct follow, it saves this position in the input string.
2	It begins comparing the “the” alternative with the input string. It compares “t” in the regular expression pattern with “t” in the input string and finds a match.
3	It compares “h” in the regular expression pattern with “h” in the input string and finds a match.
4	It compares “e” in the regular expression pattern with “e” in the input string and finds a match.
5	It determines whether the current position in the input string is on a word boundary. Because it is in the middle of the substring “these”, the match with the first alternation pattern fails, and the regular expression engine returns to its saved position (the beginning of the substring “these”) in the input string.
6-8	It compares the “this” alternative with the input string. It matches “t” and “h”, but fails to match “i" with “e” in the input string. The regular expression engine returns to its saved position in the input string.
9-11	It compares the “that” alternative with the input string. It matches “t” and “h”, but fails to match “a" with “e” in the input string. The regular expression engine returns to its saved position in the input string.
12-15	It compares the “then” alternative with the input string. It matches “t”, “h”, and “e”, but fails to match “n" with “s” in the input string. The regular expression engine returns to its saved position in the input string.
16-19	It compares the “there” alternative with the input string. It matches “t”, “h”, and “e”, but fails to match “r" with “s” in the input string. The regular expression engine returns to its saved position in the input string.
20-25	It compares the “these” alternative with the input string. It matches “t”, “h”, “e”, “s”, and “e”. It then determines that the current position in the input string is on a word boundary. The match succeeds. The last alternative in the regular expression (those) is not tested.

By using alternation only for unique portions of the regular expression pattern, we can improve the efficiency of string comparisons. In our example, because the substring “th” is common to all alternation patterns, we can rewrite the regular expression pattern as follows:

\bth(e|is|at|en|ere|ese|ose)\b

Since individual alternatives still share characters in common, we can further optimize this regular expression pattern through grouping:

\bth(e(n|re|se)|at|is|ose)\b

1	It determines that the current position in the input string is on a word boundary.
2	It matches “t” in the pattern with “t” in the input string.
3	It matches “h” in the pattern with “h” in the input string. Because a grouping and alternation construct follows, it saves this position in the input string.
4	It matches “e” in the pattern with “e” in the input string. Because a grouping and alternation construct follows, it saves this position (the second saved position) in the input string.
5	It attempts to match “n” in the pattern with “s” in the input string. The match fails, and the regular expression engine returns to its second saved position in the input string.
6	It attempts to match “r” in the pattern with “s” in the input string. The match fails, and the regular expression engine returns to its second saved position in the input string. Note that it does not test the “e” because it failed to match “r”.
7-8	It attempts to match “se” in the pattern with “se” in the input string. The match succeeds.
9	It determines that the current position in the input string is on a word boundary. The match succeeds.

The following is the complete example.

[Visual Basic]
Option Strict On

Imports System.Diagnostics
Imports System.IO
Imports System.Text.RegularExpressions

Module Example
   Public Sub Main()
      Dim sw As Stopwatch
      Dim sr As New StreamReader(".\Dreiser_TheFinancier.txt")
      Dim input As String = sr.ReadToEnd()
      sr.Close()
      Dim ctr As Integer

      sw = Stopwatch.StartNew()
      ctr = 0
      Dim unoptPattern As String = "\b(the|this|that|then|there|these|those)\b"
      Dim rgxUnopt As New Regex(unoptPattern, RegexOptions.IgnoreCase)
      Dim match1 As Match = rgxUnopt.Match(input)
      Do While match1.Success
         ctr += 1
         match1 = match1.NextMatch()
      Loop
      sw.Stop()
      Console.WriteLine("{0}:{3}   Found {1:N0} matches in {2}", 
                        unoptPattern, ctr, sw.Elapsed, vbCrLf)
      Console.WriteLine()

      sw = Stopwatch.StartNew()
      ctr = 0
      Dim optPattern As String = "\bth(e(n|re|se)|at|is|ose)\b"
      Dim rgxOpt As New Regex(unoptPattern, RegexOptions.IgnoreCase)
      Dim match2 As Match = rgxOpt.Match(input)
      Do While match2.Success
         ctr += 1
         match2 = match2.NextMatch()
      Loop
      sw.Stop()
      Console.WriteLine("{0}:{3}   Found {1:N0} matches in {2}", 
                        optPattern, ctr, sw.Elapsed, vbCrLf)
   End Sub
End Module

[C#]
using System;
using System.Diagnostics;
using System.IO;
using System.Text.RegularExpressions;

public class Example
{
    public static void Main()
    {
        Stopwatch sw;
        StreamReader sr = new StreamReader(@".\Dreiser_TheFinancier.txt");
        string input = sr.ReadToEnd();
        sr.Close();
        int ctr;

        sw = Stopwatch.StartNew();
        ctr = 0;
        string unoptPattern = @"\b(the|this|that|then|there|these|those)\b";
        Regex rgxUnopt = new Regex(unoptPattern, 
                                   RegexOptions.IgnoreCase);
        Match match1 = rgxUnopt.Match(input);
        while (match1.Success)
        {
            ctr++;
            match1 = match1.NextMatch();
        }
        sw.Stop();
        Console.WriteLine("{0}:\n   Found {1:N0} matches in {2}\n", 
                          unoptPattern, ctr, sw.Elapsed);

        sw = Stopwatch.StartNew();
        ctr = 0;
        string optPattern = @"\bth(e(n|re|se)|at|is|ose)\b";
        Regex rgxOpt = new Regex(unoptPattern,
                                 RegexOptions.IgnoreCase);
        Match match2 = rgxOpt.Match(input);
        while (match2.Success)
        {
            ctr++;
            match2 = match2.NextMatch();
        }
        sw.Stop();
        Console.WriteLine("{0}:\n   Found {1:N0} matches in {2}", 
                          optPattern, ctr, sw.Elapsed);
    }
}

The optimized regular expression offers approximately a 35% performance improvement over the unoptimized regular expression, as indicated by the following output and graph.

\b(the|this|that|then|there|these|those)\b:
   Found 13,459 matches in 00:00:00.1833098

\bth(e(n|re|se)|at|is|ose)\b:
   Found 13,459 matches in 00:00:00.1184890

Optional Quantifiers and Backtracking

The regular expression engine also uses backtracking whenever a subexpression includes an optional quantifier (such as * or +). For example, the regular expression pattern \b\p{Lu}\w*\b matches all words beginning with an uppercase character. To allow for the possibility that a word might consist of a single uppercase letter, the \w* subexpression matches any word character zero or more times. Because * is an optional quantifier, the regular expression engine saves its state with each word character matched, in case it has to abandon a portion of the match and return to a previous saved state in order to match the input string with a later portion of the regular expression pattern.

Is Backtracking Necessary?

An important question to ask about any regular expression that uses backtracking is whether, in view of its associated performance penalty, backtracking is necessary for a successful match. Backtracking often is necessary because regular expressions are greedy, and a greedy match of a subexpression is likely to consume some portion of what would otherwise be a successful match. But in many cases, backtracking is used unintentionally, because optional quantifiers that backtrack are more familiar and frequently used than their alternatives. However, backtracking comes at a price. Each time an optional quantifier is applied to an input string, the regular expression engine must save the state of the match at that point so that it can return to it if a later portion of the match fails. To put it another way, needless backtracking exacts a substantial performance penalty.

Let’s return to our example regular expression pattern, \b\p{Lu}\w*\b. Because a word boundary is not the same as, or a subset of, a word character, there is no possibility that the regular expression engine will cross a word boundary when matching word characters. This means that for this regular expression, backtracking can never contribute to the overall success of any match -- it can only extract a performance penalty, because the regular expression engine is forced to save its state for each successful preliminary match of a word character.

In cases where backtracking is unnecessary, it can be disabled by using the (?>subexpression) language construct, where subexpression is the regular expression subpattern that should not backtrack. To disable backtracking, we can change our regular expression from \b\p{Lu}\w*\b to \b\p{Lu}(?>\w*)\b. The following example uses these two regular expressions to extract words from the text of Theodore Dreisier’s The Financier.

[Visual Basic]
Imports System.Diagnostics
Imports System.IO
Imports System.Text.RegularExpressions

Module Example
   Public Sub Main()
      Dim sw As Stopwatch
      Dim sr As New StreamReader(".\Dreiser_TheFinancier.txt")
      Dim input As String = sr.ReadToEnd()
      sr.Close()
      Dim ctr As Integer

      sw = Stopwatch.StartNew()
      ctr = 0
      Dim backtrackingPattern As String = "\b\p{Lu}\w*\b"
      Dim backtrackingRegex As New Regex(backtrackingPattern)
      Dim match1 As Match = backtrackingRegex.Match(input)
      Do While match1.Success
         ctr += 1
         match1 = match1.NextMatch()
      Loop
      sw.Stop()
      Console.WriteLine("{0}:\n   Found {1:N0} matches in {2}\n", 
                        backtrackingPattern, ctr, sw.Elapsed)

      sw = Stopwatch.StartNew()
      ctr = 0
      Dim noBacktrackPattern As String = "\b\p{Lu}(?>\w*)\b"
      Dim nonBacktrackingRegex As New Regex(noBacktrackPattern)
      Dim match2 As Match = nonBacktrackingRegex.Match(input)
      Do While match2.Success
         ctr += 1
         match2 = match2.NextMatch()
      Loop
      sw.Stop()
      Console.WriteLine("{0}:\n   Found {1:N0} matches in {2}", 
                        noBacktrackPattern, ctr, sw.Elapsed)
   End Sub
End Module

[C#]
using System;
using System.Diagnostics;
using System.IO;
using System.Text.RegularExpressions;

public class Example
{
    public static void Main()
    {
        Stopwatch sw;
        StreamReader sr = new StreamReader(@".\Dreiser_TheFinancier.txt");
        string input = sr.ReadToEnd();
        sr.Close();
        int ctr;

        sw = Stopwatch.StartNew();
        ctr = 0;
        string backtrackingPattern = @"\b\p{Lu}\w*\b";
        Regex backtrackingRegex = new Regex(backtrackingPattern);
        Match match1 = backtrackingRegex.Match(input);
        while (match1.Success)
        {
            ctr++;
            match1 = match1.NextMatch();
        }
        sw.Stop();
        Console.WriteLine("{0}:\n   Found {1:N0} matches in {2}\n",
                          backtrackingPattern, ctr, sw.Elapsed);
        sw = Stopwatch.StartNew();
        ctr = 0;
        string noBacktrackPattern = @"\b\p{Lu}(?>\w*)\b";
        Regex nonBacktrackingRegex = new Regex(noBacktrackPattern);
        Match match2 = nonBacktrackingRegex.Match(input);
        while (match2.Success)
        {
            ctr++;
            match2 = match2.NextMatch();
        }
        sw.Stop();
        Console.WriteLine("{0}:\n   Found {1:N0} matches in {2}",
                          noBacktrackPattern, ctr, sw.Elapsed);
    }
}

Because it does not have to keep track of previous saved states, the regular expression that explicitly disables backtracking executes approximately 40% faster than the one that uses unnecessary backtracking. The following output from this example and graph illustrate the difference in performance.

\b\p{Lu}\w*\b:
   Found 24,157 matches in 00:00:00.2465120

\b\p{Lu}(?>\w)*\b:
   Found 24,157 matches in 00:00:00.1483269

Note that disabling backtracking is not always an option, since it changes the behavior of the regular expression engine. In the following example, the regular expression pattern (a+)ab matches one or more “a” characters, followed by the substring “ab”. This pattern matches the input string “aaaaaaaab”. However, if we change the regular expression pattern to (?>a+)ab to disable backtracking, the input string no longer matches. This is because when backtracking disabled, the regular expression engine is greedy and will not give up the last matched “a” in the string so that the match with “ab” succeeds.

[Visual Basic]
Imports System.Text.RegularExpressions

Module Example
   Public Sub Main()
      Dim input As String = "aaaaaaaab"
      
      Dim backtrackingPattern As String = "(a+)ab"
      Console.WriteLine(Regex.IsMatch(input, backtrackingPattern))
      ' Displays True.
      
      Dim nonBacktrackingPattern As String = "(?>a+)ab"
      Console.WriteLine(Regex.IsMatch(input, nonBacktrackingPattern))
      ' Displays False.          
   End Sub
End Module

[C#]
using System;
using System.Text.RegularExpressions;
 
public class Example
{
   public static void Main()
   {
      string input = "aaaaaaaab";
      
      string backtrackingPattern = "(a+)ab";
      Console.WriteLine(Regex.IsMatch(input, backtrackingPattern));
      // Displays True.
      
      string nonBacktrackingPattern = "(?>a+)ab";
      Console.WriteLine(Regex.IsMatch(input, nonBacktrackingPattern));
      // Displays False.          
   }
}

Nested Optional Quantifiers and Backtracking

When developing regular expressions, developers typically focus on creating patterns that will match particular input. They rarely focus on creating patterns that will efficiently handle input that does not match the pattern. Frequently, this is justified. For example, if a regular expression is designed to extract substrings from text that follows a known format, there is no need to focus on non-matches. However, the regular expression engine can exhibit some of its poorest performance when handling text that does not conform to a specific regular expression pattern. In particular, this occurs when a regular expression that uses nested optional quantifiers (and therefore relies on extensive backtracking) is used to process text that does not conform to the pattern.

For example, in an order processing system, let’s say a regular expression is used to ensure that an item identifier conforms to a standard pattern. All item identifiers must contain at least one alphanumeric character; if they have multiple characters, the second to last character must also be alphanumeric; and the last character must be a dollar sign (“$”). Any intermediate characters must be word characters (that is, alphanumeric characters or underscores), periods, or hyphens. The regular expression pattern that was developed to reflect the format of the item identifier is ^[0-9A-Z]([-.\w]*[0-9A-Z])*\$$. A new employee who is being trained to do data entry enters one item identifier correctly, but then enters # instead of $ as the ending symbol for the next item. The following code illustrates this regular expression pattern and the data entry input. As the output shows, the regular expression performs efficiently when it is able to match the input identifier, but performance degrades drastically if it encounters an identifier that does not match the pattern.

[Visual Basic]
Imports System.Diagnostics
Imports System.Text.RegularExpressions

Module Example
   Public Sub Main()
      Dim pattern As String = "^[0-9A-Z]([-.\w]*[0-9A-Z])*\$$"
      Dim rgx As New Regex(pattern, RegexOptions.IgnoreCase)
      Dim sw As Stopwatch

      Dim items() As String = { "A163.1523C$", 
                                "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa#" }
      Dim toValidate, result As String
      
      For Each item In items
         For ctr As Integer = 1 To item.Length
            toValidate = item.Substring(0, ctr)     
            sw = Stopwatch.StartNew() 
   
            If rgx.IsMatch(toValidate) Then
               result = "Match"
            Else
               result = "No match"
            End If
            sw.Stop()
            Console.WriteLine("{0} with {1} characters in {2}", result, ctr, 
                              sw.Elapsed)   
         Next
         Console.WriteLine()               
      Next
   End Sub
End Module

[C#]
using System;
using System.Diagnostics;
using System.Text.RegularExpressions;

public class Example
{
   public static void Main()
   {
      string pattern = @"^[0-9A-Z]([-.\w]*[0-9A-Z])*\$$";
      Regex rgx = new Regex(pattern, RegexOptions.IgnoreCase);
      Stopwatch sw;

      string[] items= { "A163.1523C$", 
                        "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa#" };
      string toValidate, result;
      
      foreach (var item in items) {
         for (int ctr = 1; ctr <= item.Length; ctr++) {
            toValidate = item.Substring(0, ctr);     
            sw = Stopwatch.StartNew(); 
   
            if (rgx.IsMatch(toValidate))
               result = "Match";
            else
               result = "No match";

            sw.Stop();
            Console.WriteLine("{0} with {1} characters in {2}", result, ctr, 
                              sw.Elapsed);   
         }
         Console.WriteLine();               
      }
   }
}

The example displays the following output:

No match with 1 characters in 00:00:00.0000252
No match with 2 characters in 00:00:00.0000132
No match with 3 characters in 00:00:00.0000048
No match with 4 characters in 00:00:00.0000061
No match with 5 characters in 00:00:00.0000071
No match with 6 characters in 00:00:00.0000259
No match with 7 characters in 00:00:00.0000175
No match with 8 characters in 00:00:00.0000307
No match with 9 characters in 00:00:00.0000573
No match with 10 characters in 00:00:00.0001128
Match with 11 characters in 00:00:00.0000035

No match with 1 characters in 00:00:00.0000016
No match with 2 characters in 00:00:00.0000022
No match with 3 characters in 00:00:00.0000029
No match with 4 characters in 00:00:00.0000045
No match with 5 characters in 00:00:00.0000081
No match with 6 characters in 00:00:00.0000145
No match with 7 characters in 00:00:00.0000265
No match with 8 characters in 00:00:00.0000521
No match with 9 characters in 00:00:00.0001017
No match with 10 characters in 00:00:00.0002032
No match with 11 characters in 00:00:00.0004016
No match with 12 characters in 00:00:00.0007994
No match with 13 characters in 00:00:00.0015952
No match with 14 characters in 00:00:00.0032086
No match with 15 characters in 00:00:00.0065094
No match with 16 characters in 00:00:00.0129808
No match with 17 characters in 00:00:00.0270169
No match with 18 characters in 00:00:00.0553021
No match with 19 characters in 00:00:00.1070363
No match with 20 characters in 00:00:00.2208385
No match with 21 characters in 00:00:00.4289808
No match with 22 characters in 00:00:00.8553915
No match with 23 characters in 00:00:01.6993887
No match with 24 characters in 00:00:03.3938261
No match with 25 characters in 00:00:06.8359716
No match with 26 characters in 00:00:13.5125933
No match with 27 characters in 00:00:27.2257724
No match with 28 characters in 00:00:53.8205646
No match with 29 characters in 00:01:49.2090587
No match with 30 characters in 00:03:35.8028235
No match with 31 characters in 00:07:11.6145706
No match with 32 characters in 00:10:46.0044969

This example evaluates the performance of the regular expression engine based on the number of characters passed to it in the input string. As the following chart shows, for an input string that does not match the regular expression pattern, execution time approximately doubles each time another character is added to the string. It takes slightly over 10 minutes for the regular expression engine to recognize that an input string consisting of 32 characters does not match the regular expression pattern. Expanding the input to 35 characters would take approximately 1 hour and 20 minutes. Processing a string with 38 characters would take over 10 hours. Processing a string with 40 characters would take nearly two days. Visually scanning the string would be significantly faster.

This atrociously poor performance is tied to the behavior of the NFA regular expression engine in the .NET Framework. When the regular expression engine cannot find a match and the pattern requires backtracking, the engine must exhaust all possible paths through the data before it can conclude that a match has failed; so an unsuccessful match always represents a worst-case scenario. (For more information on the NFA engine and backtracking, see Details of Regular Expression Behavior and Backtracking in the MSDN Library.) Because of the nested quantifiers in our example, the number of possible matches that the regular expression engine has to examine before concluding that a match has failed increases exponentially with each character after the first character in the input string.

Not all nested quantifiers impact performance in this way. In our example, the real problem in the subexpression ([-.\w]*[0-9A-Z])* is that the last character to be matched (any numeric character from 0 to 9, or any alphanumeric character from A to Z) is a subset of the previous subexpression (any word character – that is, any numeric or alphanumeric character or an underscore -- along with a hyphen and a period). A partially successful match makes it difficult for the regular expression engine to determine whether a particular character belongs to the first subexpression, the ending character of the expression, or to a repetition of the first subexpression or the ending character. The inability to successfully match the entire regular expression results in excessive backtracking, so that the number of possible paths through the input string increases exponentially with each character that can be matched. (For a description of the additional comparisons performed by the regular expression engine when backtracking, see the “Backtracking with Nested Optional Quantifiers” section in the Backtracking topic in the MSDN Library.)

In cases where excessive backtracking causes performance problems, other language constructs can often be used in place of the nested quantifiers. In many cases, nested quantifiers can be replaced by a single quantifier and zero-width lookbehind or lookahead assertion. Zero-width assertions do not advance the position counter in the input string. They either look ahead of or immediately behind the current position to determine whether the pattern defined by the assertion is true. If it is not, the match fails. (For information on positive lookahead and positive lookbehind, see Grouping Constructs in the MSDN documentation.) For example, we can replace our previous regular expression to validate an item identifier with ^[0-9A-Z][-.\w]*(?<=[0-9A-Z])\$, which uses positive lookbehind when the $ symbol is matched to ensure that the character before $ is either numeric or alphanumeric. This regular expression is interpreted as follows:

Pattern	Description
^	Begin the match at the beginning of the string.
[0-9A-Z]	Match any numeric (0-9) or alphanumeric (A-Z) character.The comparison is not case-sensitive.
[-.\w]*	Match zero or more word, hyphen, or period characters.
\$	Match the $ character.
(?<=[0-9A-Z])	Determine whether the character before the current character is numeric (0-9) or alphanumeric (A-Z).The comparison is not case sensitive. This is a zero-width assertion; it does not cause the position counter in the input string to advance.

This produces the following output:

No match with 1 characters in 00:00:00.0498953
No match with 2 characters in 00:00:00.0002162
No match with 3 characters in 00:00:00.0000237
No match with 4 characters in 00:00:00.0000333
No match with 5 characters in 00:00:00.0000596
Match with 6 characters in 00:00:00.0000500
Match with 7 characters in 00:00:00.0000256
Match with 8 characters in 00:00:00.0000237
Match with 9 characters in 00:00:00.0000198
Match with 10 characters in 00:00:00.0000198
Match with 11 characters in 00:00:00.0000186
Match with 12 characters in 00:00:00.0000192
Match with 13 characters in 00:00:00.0000494
Match with 14 characters in 00:00:00.0000449

No match with 1 characters in 00:00:00.0000288
No match with 2 characters in 00:00:00.0000205
No match with 3 characters in 00:00:00.0000205
No match with 4 characters in 00:00:00.0000301
No match with 5 characters in 00:00:00.0000481
No match with 6 characters in 00:00:00.0000269
No match with 7 characters in 00:00:00.0000218
No match with 8 characters in 00:00:00.0000346
No match with 9 characters in 00:00:00.0000468
No match with 10 characters in 00:00:00.0000442
No match with 11 characters in 00:00:00.0000346
No match with 12 characters in 00:00:00.0000891
No match with 13 characters in 00:00:00.0000192
No match with 14 characters in 00:00:00.0000372
No match with 15 characters in 00:00:00.0000218
No match with 16 characters in 00:00:00.0000192
No match with 17 characters in 00:00:00.0000243
No match with 18 characters in 00:00:00.0000224
No match with 19 characters in 00:00:00.0000205
No match with 20 characters in 00:00:00.0000224
No match with 21 characters in 00:00:00.0000218
No match with 22 characters in 00:00:00.0000224
No match with 23 characters in 00:00:00.0000263
No match with 24 characters in 00:00:00.0000218
No match with 25 characters in 00:00:00.0000243
No match with 26 characters in 00:00:00.0000218
No match with 27 characters in 00:00:00.0000224
No match with 28 characters in 00:00:00.0000224
No match with 29 characters in 00:00:00.0000230
No match with 30 characters in 00:00:00.0000237
No match with 31 characters in 00:00:00.0000243
No match with 32 characters in 00:00:00.0000243

In other cases, some combination of alternation (the construct “|”) along with lookahead and lookbehind can be used to replace a regular expression that is subject to excessive backtracking. For example, regular expressions that are designed to match numbers are frequently troublesome if the numeric value can be either integral or floating point. A common pattern used to match such numeric values is ^\d+(|.\d+)*$, as shown in the following example.

[Visual Basic]
Imports System.Diagnostics
Imports System.Text.RegularExpressions

Module Example
   Public Sub Main()   
      Dim number As String = "111111111111111111111111111111!"
      Dim pattern As String = "^\d+(|.\d+)*$"
      
      Dim rgx As New Regex(pattern, RegexOptions.IgnoreCase)
      Dim sw As Stopwatch
      Dim str As String 
  
      For ctr As Integer = 1 To number.Length
         str = number.Substring(0, ctr)
         sw = Stopwatch.StartNew()
         If rgx.IsMatch(str) Then 
            Console.Write("Match with {0} characters ", str.Length)
         Else
            Console.Write("No match with {0} characters ", str.Length)
         End If  
         sw.Stop()
         Console.WriteLine("in {0}.", sw.Elapsed)
      Next
   End Sub
End Module

[C#] 
using System;
using System.Diagnostics;
using System.Text.RegularExpressions;

public class Example
{
   public static void Main()
   {
      string number = "111111111111111111111111111111!";
      string pattern = @"^\d+(|.\d+)*$";
      
      Regex rgx = new Regex(pattern, 
                  RegexOptions.IgnoreCase);
      Stopwatch sw;
      string str; 
  
      for (int ctr = 1; ctr <= number.Length; ctr++) {
         str = number.Substring(0, ctr);
         sw = Stopwatch.StartNew();
         if (rgx.IsMatch(str)) 
            Console.Write("Match with {0} characters ", str.Length);
         else
            Console.Write("No match with {0} characters ", str.Length);

         sw.Stop();
         Console.WriteLine("in {0}.", sw.Elapsed);
      }
   }
}

However, as the output from the example shows, this pattern is subject to excessive backtracking.

Match with 1 characters in 00:00:00.0011311.
Match with 2 characters in 00:00:00.0010926.
Match with 3 characters in 00:00:00.0011003.
Match with 4 characters in 00:00:00.0014570.
Match with 5 characters in 00:00:00.0002797.
Match with 6 characters in 00:00:00.0001565.
Match with 7 characters in 00:00:00.0012619.
Match with 8 characters in 00:00:00.0001456.
Match with 9 characters in 00:00:00.0011439.
Match with 10 characters in 00:00:00.0010682.
Match with 11 characters in 00:00:00.0011125.
Match with 12 characters in 00:00:00.0002053.
Match with 13 characters in 00:00:00.0001854.
Match with 14 characters in 00:00:00.0012485.
Match with 15 characters in 00:00:00.0010675.
Match with 16 characters in 00:00:00.0011137.
Match with 17 characters in 00:00:00.0012299.
Match with 18 characters in 00:00:00.0001937.
Match with 19 characters in 00:00:00.0012709.
Match with 20 characters in 00:00:00.0011323.
Match with 21 characters in 00:00:00.0010996.
Match with 22 characters in 00:00:00.0010188.
Match with 23 characters in 00:00:00.0011125.
Match with 24 characters in 00:00:00.0001770.
Match with 25 characters in 00:00:00.0001469.
Match with 26 characters in 00:00:00.0001687.
Match with 27 characters in 00:00:00.0004356.
Match with 28 characters in 00:00:00.0013370.
Match with 29 characters in 00:00:00.0010958.
Match with 30 characters in 00:00:00.0010983.
No match with 31 characters in 00:00:03.0872118.

The root of the problem is that the subexpression (|.\d+)* is intended to match either a group separator or a decimal separator, but it also matches any other character. The subexpression also includes a nested quantifier. In the event of a near-match with numeric digits, the regular expression engine will have difficulty apportioning numeric digits between the first occurrence of the \d* pattern or its subsequent possible repetitions.

A regular expression that eliminates this unnecessary backtracking and also addresses a number of limitations of the original regular expression is ^\d*(,\d{3})*(?((?=\.))\.\d+|(?<=\d))$. This regular expression uses the conditional construct with an expression (for more information, see Alternation Constructs in the MSDN Library. Note that this regular expression works with numeric values formatted using the conventions of the English (United States) culture. A regular expression that uses the group and decimal separators of the current culture could be built dynamically by retrieving these values from the current culture’s DateTimeFormatInfo object. The regular expression is interpreted as shown in the following table.

Pattern	Description
^	Begin the match at the beginning of the string.
\d*	Match zero or more decimal digits. This matches the integral portion of the numeric value.
(,\d{3})*	Match zero or more occurrences of a group separator followed by three decimal digits.
(?((?=\.))	Use positive lookahead to determine whether the next character is a decimal separator. (This is the condition clause of an alternation pattern.)
\.\d+	If the next character is a decimal separator, match the decimal separator followed by one or more decimal digits.
\|(?<=\d))	If the next character is not a decimal separator, look behind to determine whether the previous character is a decimal digit. This ensures that the number contains at least one integral digit.
$	Match the end of the input string.

If this regular expression pattern replaces the pattern in the previous example, the example produces the following output:

Match with 1 characters in 00:00:00.0012331.
Match with 2 characters in 00:00:00.0010586.
Match with 3 characters in 00:00:00.0010509.
Match with 4 characters in 00:00:00.0011676.
Match with 5 characters in 00:00:00.0002303.
Match with 6 characters in 00:00:00.0001892.
Match with 7 characters in 00:00:00.0005203.
Match with 8 characters in 00:00:00.0014442.
Match with 9 characters in 00:00:00.0010740.
Match with 10 characters in 00:00:00.0010573.
Match with 11 characters in 00:00:00.0002046.
Match with 12 characters in 00:00:00.0002309.
Match with 13 characters in 00:00:00.0012318.
Match with 14 characters in 00:00:00.0001411.
Match with 15 characters in 00:00:00.0010900.
Match with 16 characters in 00:00:00.0010785.
Match with 17 characters in 00:00:00.0002008.
Match with 18 characters in 00:00:00.0002149.
Match with 19 characters in 00:00:00.0012850.
Match with 20 characters in 00:00:00.0011137.
No match with 21 characters in 00:00:00.0010887.

Best Practices for Controlling Backtracking

To maximize the performance of regular expressions in the .NET Framework, we recommend that you do the following to limit or eliminate unnecessary backtracking:

In alternation constructs, order alternatives based on their approximate frequency of occurrence in the input.
Take the time to optimize alternation patterns.
Determine whether backtracking is required for the regular expression to successfully match a subexpression with a portion of the input string. If it is not necessary, disable backtracking with the (?>subexpression) language element.
If subexpressions are intended to validate input, replace nested quantifiers such as (\w*[A-Z])* with other language elements. The most useful language elements for this purpose are the positive lookahead assertion and the positive lookbehind assertion, the alternation construct, and the conditional construct.

More on Zip in .NET [Richard Lee]

BCL Team — Thu, 29 Jul 2010 23:21:29 GMT

First, I’d like to thank everybody for their comments on the Zip APIs. It’s great to know that I’m working on something that a lot of people will hopefully find useful. I’ll try to address the themes that came up in the comments.

Streams

A lot of the comments mentioned support for streams. The API does support creating a ZipArchive with a stream as the backing store, and input from streams. The following code sample takes the contents of instream and writes a Zip archive to outstream containing just that one file:

using (ZipArchive archive = new ZipArchive(outstream, ZipArchiveMode.Create))
{
	ZipArchiveEntry entry = archive.CreateEntry("data.dat");
	using (Stream entryStream = entry.Open())
	{
		instream.CopyTo(entryStream);
	}
}

This will write out the Zip archive directly to outstream, without buffering the entire contents of the archive in memory or writing to a temporary file. We really think of the methods used above as the core APIs. Methods like CreateEntryFromFile and ExtractToDirectory are purely convenience methods – their main purpose is to make some of the more common scenarios with files easier.

Compression and Encryption

Another common theme was custom encryption and compression algorithms. The vast majority of Zip archives that are meant to be interoperable with the widest range of libraries, tools, and applications use the Deflate compression algorithm without encryption. Our main goal for this API is to be able to read and write such archives. As such, we’re currently planning to support writing Zip archives with Deflate, reading Zip archives that use Deflate or no compression, and to not support encryption. Not only will this enable reading/writing interoperable Zip archives, it also scopes the work to something reasonable that can be delivered during my internship.

If we’re not providing built-in support for additional compression or encryption algorithms, an obvious question, then, is why not provide extensibility hooks so that custom compression or encryption algorithms can be plugged-in? We explored doing this and it turns out it’d be more complex than you might initially think.

A lot of people mentioned the CryptoStream and ICryptoTransform model as a powerful way to allow for extensibility. This works well because the ICryptoTransform only needs to do one thing – transform a series of bytes into another series of bytes. Unfortunately, encrypting Zip files is a much more complex operation. Fields and flags need to be set to appropriate values and headers need to be encrypted depending on which algorithm is used. Implementing either of the two secure encryption methods mentioned in the Zip specification would require access to essentially all of the metadata in the Zip file. The resulting interface for such an extension would be enormously complex.

Compression is a bit simpler, as only one entry is compressed at a time. However, there are still fields and flags that need to be set in the headers, depending on the algorithm used. Furthermore, for both encryption and compression, only a few compression/encryption methods are specified in the Zip specification. We don’t want to give the impression that any compression method can be used to produce a Zip file, when that isn’t the case. If you want to use a compression or encryption algorithm that isn’t specified in the Zip spec, you might as well just compress or encrypt the stream yourself.

So an extensibility interface would be substantially more complex than something like ICryptoTransform, and add significant complexity to the ZipArchive/ZipArchiveEntry class. We don’t think providing extensibility hooks in this way adds enough value to the API to justify the added complexity that they bring. However, we are open to adding built-in support for encryption or other compression algorithms in the future, based on customer demand. We’ve specifically designed the API in a way that would allow us to do this. If this is something you’re interested in, we’d love to better understand your needs.

Abstract Base Class

Another common question was around providing a base class for Archives. We explored this and have decided not to add such an abstraction at this time. The main reason we are holding off is because right now we’re only planning to provide support for Zip archives and have no plans to support other archive formats (such as CAB, RAR, etc.). As a design principle, we try to avoid adding abstractions when there is only one implementation, otherwise we risk getting the abstraction wrong. Then we’re either stuck with the bad abstraction or are forced to add another when adding another implementation. All of this adds additional complexity that we’d like to avoid.

The concerns about test-driven development are certainly valid, but there are ways around this. For example, because archives can be made with streams as the backing store, a FileStream could be mocked and put behind an archive. That may not be totally ideal, but these concerns don’t seem compelling enough to justify adding an abstract base class at this time.

Miscellaneous

There were a couple comments about treating the Zip archive like the filesystem, and supporting searching for certain kinds of files. We made the decision to treat the archive as a flat container of files because that is how they are actually stored. As a library rather than an application, we thought it made more sense to represent the archive as it exists on disk. Also, using LINQ on the entries means getting all of the files in a certain subdirectory that end in .txt is a relatively simple operation.

Another interesting comment was that MoveTo was confusing. MoveTo was intended to act like Rename, for renaming entries but keeping everything else about them the same. The method was named MoveTo to mimic the naming for the method on FileInfo. However, because it is confusing and probably has very few compelling usage scenarios, we’re thinking of cutting it from the API.

I hope I addressed some of your comments. We’d love to keep hearing from you, especially if you have extremely compelling use cases for some of the features that we’re not including in this version of the API. Being the designer of the API, I think it would be really cool, too, to support custom compression and encryption, or some of these other features. But at the risk of sounding like a broken record, I’m trying to build a simple, usable API, and these decisions are made with that in mind.

Working with Zip Files in .NET [Richard Lee]

BCL Team — Mon, 28 Jun 2010 21:37:00 GMT

Before getting started, I’ll introduce myself. My name is Richard Lee, and I’m a developer intern on the BCL for the summer. I’ve only been here for a few weeks, but it’s been great working here. The people, the environment, and my project are all great. Speaking of which, my project is to add general purpose .NET APIs for reading and writing Zip files, which we’re considering adding to the next version of the .NET Framework.

The most common Zip tasks are extracting to a directory and archiving a directory. For these mainline scenarios, we have static convenience methods. The following code takes all of the files in the Zip file, photos.zip, and extracts them to a folder on the file system:

ZipArchive.ExtractToDirectory("photos.zip", @"photos\summer2010");

This code does the reverse, putting all of the files in the folder into the Zip file:

ZipArchive.CreateFromDirectory(@"docs\attach", "attachment.zip");

For more sophisticated manipulations of Zip archives, there are two main classes. ZipArchive represents a zip archive, which is a collection of entries, and ZipArchiveEntry represents an archived file entry. The following code extracts only text files from the given archive.

using (var archive = new ZipArchive("data.zip"))
{
    foreach (var entry in archive.Entries)
    {
        if (entry.FullName.EndsWith(".txt", StringComparison.OrdinalIgnoreCase))
        {
            entry.ExtractToFile(Path.Combine(directory, entry.FullName));
        }
    }
}

Zip archives can also be created on-the-fly. This example creates a new archive with a readme file that is created without the need for a corresponding file on disk, and a file from the file system.

using (var archive = new ZipArchive("new.zip", ZipArchiveMode.Create))
{
    var readmeEntry = archive.CreateEntry("Readme.txt");
    using (var writer = new StreamWriter(readmeEntry.Open()))
    {
        writer.WriteLine("Included files: ");
        writer.WriteLine("data.dat");
    }

    archive.CreateEntryFromFile("data.dat", "data.dat");
}

The ZipArchive class supports three modes:

In Read mode, data is read from the file on demand, using only a small buffer.
In Create mode, data is written directly to the file using only a small buffer. Only one entry may be held open for writing at a time.
In Update mode it is possible to read and write from existing archives, as well as rename or delete entries. This mode requires loading the entire archive into memory, and as such we recommend that it be used only with small archives when this functionality is needed.

Below is our current thinking on what the public API listing will look like (note that this hasn’t been finalized yet).

namespace System.IO.Compression
{
    public enum ZipArchiveMode { Read, Create, Update }

    public class ZipArchive : IDisposable {
        // Constructors
        public ZipArchive(String path);
        public ZipArchive(String path, ZipArchiveMode mode); 
        public ZipArchive(Stream stream);
        public ZipArchive(Stream stream, ZipArchiveMode mode);
        public ZipArchive(Stream stream, ZipArchiveMode mode, Boolean leaveOpen);

        // Properties
        public ReadOnlyCollection<ZipArchiveEntry> Entries { get; }
        public ZipArchiveMode Mode { get; }
        
        // Instance methods
        public ZipArchiveEntry GetEntry(String entryName);
        public ZipArchiveEntry CreateEntry(String entryName);

        public void Dispose();
        protected virtual void Dispose(Boolean disposing);

        public override String ToString();

        // Instance convenience methods
        public ZipArchiveEntry CreateEntryFromFile(String sourceFileName, String entryName);

        public void ExtractToDirectory(String destinationDirectoryName);

        // Static convenience methods
        public static void CreateFromDirectory(String sourceDirectoryName, String destinationArchive);
        public static void CreateFromDirectory(String sourceDirectoryName, String destinationArchive, Boolean includeBaseDirectory);

        public static void ExtractToDirectory(String sourceArchive, String destinationDirectoryName);
    }

    public class ZipArchiveEntry {
        // Properties
        public DateTimeOffset LastWriteTime { get; set; }
        public String FullName { get; }
        public String Name { get; }
        public Int64 Length { get; }
        public Int64 CompressedLength { get; }
        public ZipArchive Archive { get; }

        // Methods
        public Stream Open();
        public void Delete();
        public void MoveTo(String destinationEntryName);

        // Convenience methods
        public void ExtractToFile(String destinationFileName);
        public void ExtractToFile(String destinationFileName, Boolean overwrite);

        public override String ToString();
    }
}

We would love to hear what you think of the APIs so far, and how you plan on using them.