BCL Team Blog

What’s new in MEF 2 Preview 5? [Alok, Nick]

BCL Team — Mon, 19 Dec 2011 19:18:00 GMT

Nearly two months since we published MEF 2 Preview 4, we have just posted our fifth preview on the MEF CodePlex site.

The new preview has been shaped by your feedback; everyone who took the time to download, evaluate or comment on the last version, thank you!

So, what has changed since Preview 4?

CompositionException.RootCauses, previously only visible in the debugger, can now be accessed directly in code
Using RegistrationBuilder, all interfaces on a part can now be exported using the ExportInterfaces method
IDisposable and IPartImportsSatisfiedNotification will now be filtered out by RegistrationBuilder automatically (these should never be exported)
CompositionProvider in the ASP.NET MVC integration now exposes ApplicationSharedPartMetadataKey for those wishing to create their own sharing conventions compatible with the provider
Some additional performance tuning has been done

As we get closer to making our next supported release as part of the .NET Framework, the bar is now high for changes to be included in MEF 2. Even so, we’d love to hear about your experiences with the new preview, and planning for MEF 3 is just around the corner. You can get in touch via this blog or through the discussion forum and issue tracker on CodePlex.

Sharing with CompositionScopeDefinition in MEF 2 [Alok]

BCL Team — Mon, 19 Dec 2011 17:23:00 GMT

This post discusses features in the preview version of MEF, and some details may change between now and the full release.

In the previous post, we introduced ExportFactory<T> and demonstrated how it can be used to control the lifetimes of MEF parts. In this article we will talk about a new type MEF called CompositionScopeDefinition that can be used in conjunction with ExportFactory<T> to control sharing. Let’s take a scenario that we could not achieve using what we know currently and how CompositionScopeDefinition provides the needed control.

Important note: CompositionScopeDefinition is geared towards more advanced, desktop application scenarios. If you're using MEF in an ASP.NET MVC web application and want to control sharing and lifetime on a per-HTTP-request level, you should use the pre-defined ASP.NET MVC integration.

Lets go back to our application block diagram that we built in Part 1 of this article.

Now let’s add a few details to this architecture. As our application has grown more complex, we have decided to factor out the data access part into a separate component called DataAccessLayer, which handles manages the connection to the database and reading and writing to it. In addition to this we also want to add a logger, which also logs some tracking information to the same database. So our slightly expanded functional diagram looks like the following.

In the context of our scenario, all data access within a single request should be made through the same database connection. In the default configuration, MEF will share a single DatabaseConnection across the application. How do we instruct MEF to share the database connection per request? Lets start by laying out the class definitions for the different parts.

    [Export]
    public class RequestListener
    {
        ExportFactory<RequestHandler> _handlerFactory;

        public RequestListener(ExportFactory<RequestHandler> handlerFactory)
        {
            _handlerFactory = handlerFactory;
        }

        public void HandleRequest()
        {
            using (var request = _handlerFactory.CreateExport())
            {
                request.Value.Process();
            }
        }
    }

    [Export]
    public class RequestHandler
    {
        [ImportingConstructor]
        public RequestHandler(DataAccessLayer dataAccess, Logger logger)
        {
        }

        public void Process()
        {
            //Do stuff using DataAccess and Logger
        }
    }

    [Export]
    public class DataAccessLayer
    {
        [ImportingConstructor]
        public DataAccessLayer(DatabaseConnection conn)
        {
        }
    }

    [Export]
    public class Logger
    {
        [ImportingConstructor]
        public Logger(DatabaseConnection conn)
        {
        }
    }

    [Export]
    public class DatabaseConnection
    {
        public DatabaseConnection()
        {
        }
    }

Introducing CompositionScopeDefinition

Scenario 5: Sharing within the scope of an ExportFactory<T> instantiation

As we can observe as we had in the previous article the RequestListener class has an ExportFactory to create instances of the RequestHandler class. The RequestHandler now depends on two classes the DataAccess and the Logger class, both of which depend on the DatabaseConnection class. What we want to achieve looks like the following figure.

If we look from the DatabaseConnection down to the root of the composition at RequestListener, we can see that from a composition beginning at the RequestHandler (the highlighted portion), we can achieve the sharing using standard MEF we have discussed in part 1 of this article. This is one way to figure out how to divide your catalog into scopes, when you hit a sharing pattern that cannot be achieved by using the simple MEF options. We would then need to write the following code to obtain the desired sharing.

var listenerLevel = new TypeCatalog(typeof(RequestListener));
var requestLevel = new TypeCatalog(
    typeof(RequestHandler), 
    typeof(Logger), 
    typeof(DataAccessLayer), 
    typeof(DatabaseConnection));

var container = new CompositionContainer(
     new CompositionScopeDefinition(listenerLevel, 
         new [] { new CompositionScopeDefinition(requestLevel, null) }));

var requestListener = container.GetExportedValue<RequestListener>();
requestListener.HandleRequest();
requestListener.HandleRequest();

The types making up the scenario are distributed between two catalogs — those that should be created and shared at the request listener level, and those that need to be created and shared at the per-request level. When an ExportFactory<RequestHandler> is imported into the request listener, the container sees that the RequestHandler type is within a child scope, so the child scope definition is used to determine the components (including the DatabaseConnection) that should be shared in the child scope.

In order for a parent scope to be able to discover a part in a child scope, it has to be imported as an ExportFactory<T>. If the RequestListener imported the RequestHandler type and not an ExportFactory<RequestHandler>, a CompositionException would be thrown.

How have you used CompositionScopeDefintion? Did you find it easy to use? Are there still sharing scenarios that you cannot achieve? Leave us a message in the comments or in the dicussion forum on the MEF CodePlex site.

ExportFactory in MEF 2 [Alok]

BCL Team — Fri, 18 Nov 2011 01:12:00 GMT

This post discusses features in the preview version of MEF, and some details may change between now and the full release.

In the first version of MEF there are only two notions of lifetime: A shared global lifetime or a per instance lifetime. In MEF 2 Preview 4 new support has been added to enable a finer grained control over the lifetime and sharing of parts. Before we can cover those changes in detail, we need to introduce ExportFactory<T>.

Let us consider a simple application scenario of a RequestListener, which spawns a RequestHandler, which uses a DatabaseConnection to connect to a data source (Fig 1):

Let us also consider the following parts that implement the RequestHandler and DatabaseConnection.

    [Export]
    public class RequestHandler
    {
        [ImportingConstructor]
        public RequestHandler(DatabaseConnection connection)
        {
        }
    }

    [Export]
    public class DatabaseConnection
    {
        public DatabaseConnection()
        {
        }
    }

In MEF version 1...

To better illustrate the new capabilities we’ve added in MEF 2, some of the scenarios that could be achieved in MEF version 1 are discussed below.

Scenario 1: RequestHandler and DatabaseConnection instances are shared

When we first start building our app, we are being very conservative about resources and not knowing our load we spin up a single instance of RequestHandler and DatabaseConnection

We could instantiate an instance of the RequestHandler using the following code.

TypeCatalog global = new TypeCatalog(
    typeof(RequestHandler), typeof(DatabaseConnection));
var container = new CompositionContainer(global);
var requestHandler = container.GetExportedValue<RequestHandler>();

Regardless of the number of times GetExportedValue() is called, the same instance of the RequestHandler is always returned (Fig 2).

Scenario 2: Separate instances of RequestHandler and DatabaseConnection

Now we realize that the throughput for our system is really poor, since we can only process one request at a time. So in order to increase our throughput, we spin up new instance of RequestHandler and DatabaseConnection for every request that comes in. In order to achieve this with MEF we can put a PartCreationPolicyAttribute on RequestHandler and DatabaseConnection:

    [PartCreationPolicy(CreationPolicy.NonShared)]
    public class RequestHandler { … }

    [PartCreationPolicy(CreationPolicy.NonShared)]
    public class DatabaseConnection { … }

Now, with the same invoking code to GetExportedValue(), we now get new instances for RequestHandler and DatabaseConnection every time.

Scenario 3: RequestHandler instances created and DatabaseConnection instance is shared

Our throughput is now up, but creating a new database connection per request is really taking a toll on our database server which can handle only a few open connections at a time. To ease the load on our database server we hence decide, that we will share a single instance of a DatabaseConnection, among multiple RequestHandlers. This will give us our throughput improvements without overloading our database server. We can achieve the following by getting rid of the PartCreationPolicy we have on the DatabaseConnection class, so it would now look like the following:

    public class DatabaseConnection { … }

With the removal of this attribute, we now get only a new instance of the RequestHandler every time GetExportedValue() is called, which all share a single instance of the DatabaseConnection.

In MEF 2... introducing ExportFactory<T>

The above scenarios that we have seen are the sharing scenarios that could be achieved using MEF in the past releases. However there is one glaring omission in the matrix that we presented above. What if I wanted a multiple instance of a dependency from a single instance of what it was depending on? To put it in the context of our example, how does the RequestListener class which was the first block in our diagram create multiple instances of the RequestHandler.

This brings us to:

Scenario 4: Single instance of RequestListener and multiple (dynamic) instances of RequestHandler

In order to enable this scenario, in the latest version of MEF, we introduce a new class called the ExportFactory<T>. ExportFactory<T> may be familiar to Silverlight 4 developers, since it was shipped as a part of the Silverlight 4 SDK. The export factory allows us to create new instances of dependencies and control the lifetime of the created parts.

Let us go back to our initial block diagram and write out the code for the RequestListener class.

    [Export]
    public class RequestListener
    {
        ExportFactory<RequestHandler> _factory;

        [ImportingConstructor]
        public RequestListener(ExportFactory<RequestHandler> factory)
        {
            _factory = factory;
        }

        public void HandleRequest()
        {
            using (var instance = _factory.CreateExport())
            {
                instance.Value.Process();
            }
        }
    }

Now we see in the code that instead of importing a RequestHandler as a depedency, we import an ExportFactory<RequestHandler>. MEF treats ExportFactory<T> as a special class, and will automatically hook it up to the part providing T. We then added a HandleRequest() member, which does the work of instantiating the RequestHandler. The CreateExport() method creates an ExportLifetimeContext<T>, which implements IDisposable, which is used to control the lifetime of the objects created by the factory. Since the listener is creating the parts, we can get also get rid of the PartCreationPolicy attribute on the RequestHandler. Putting all the pieces together a call to:

TypeCatalog global = new TypeCatalog(typeof(RequestHandler), typeof(DatabaseConnection) , typeof(RequestListener));
var container = new CompositionContainer(global);
var requestListener = container.GetExportedValue<RequestListener>();
requestListener.HandleRequest();

will result in the following composition graph:

Stay tuned for part two of this post where we talk about some more sharing scenarios and how we can use the new scoping enhancements in MEF in conjunction with ExportFactory<T> to accomplish these.

Using MEF 2 with ASP.NET MVC 3 [Nick]

BCL Team — Fri, 11 Nov 2011 23:47:00 GMT

This post discusses features in the preview version of MEF, and some details may change between now and the full release.

The latest MEF preview includes a new assembly, System.ComponentModel.Composition.Web.Mvc.CodePlex.dll, that simplifies the task of building flexible, testable and maitainable ASP.NET MVC web applications.

To achieve this it:

provides dependency injection for Controller classes,
defines a simple convention for identifying and configuring MEF parts,
maps the lifetime of part instances to the ASP.NET request processing lifecycle, and
simplifies the use of dependency injection in action filter attributes and model binders.

The .NET 4.0 version of the 'composition provider' assembly that can be downloaded from CodePlex only supports the preview version of MEF 2 with which it is bundled, and requires ASP.NET MVC 3. We expect to release the final version as a NuGet package targeting the MEF 2 RTM.

In this article, we'll walk through the process of setting up basic composition for controllers and parts. We'll then examine some of the deeper support provided for other ASP.NET MVC features. You'll need some familiarity with both managed composition (MEF/IoC) and ASP.NET MVC.

This post is accompanied by a sample application called 'Fabrikam Books.' You may wish to download the sample solution now so that you can explore and experiment with runnable code.

Enabling the composition provider in an ASP.NET MVC application

Setting up the composition provider is super simple:

If you haven't already, create your ASP.NET MVC 3 web application.
Add assembly references to:
- System.ComponentModel.Composition.CodePlex.dll
- System.ComponentModel.Composition.Web.Mvc.CodePlex.dll
Use Visual Studio to create a folder in the root of your web application project called Parts

The last step — creating a Parts folder — is a good way to get started. In the long run, you may reorganise and have several parts folders in your project, but the important thing is that parts for composition live in a namespace that includes a 'Parts' element.

The composition provider from this point on is configuration-free. You can press F5 and watch your site run — so far nothing obvious should have changed, but under the covers, ASP.NET will be using MEF to construct the controllers used to handle requests! Before you can take advantage of this, you'll need to create some parts.

Creating a part

ASP.NET MVC already defines some important roles for application classes – there are controllers, models, views, filters and so-on. There are many other roles that make up a complete application; classes for:

Accessing data
Performing calculations
Communicating with other sites and web services
The secret sauce that makes your application special!

When using the composition provider for ASP.NET MVC, these classes can be implemented as parts.

The Parts folder

By convention, classes created in the Parts namespace are assumed to be parts for managed composition. Create a regular class by right-clicking on the Parts folder and selecting “Add New” → “Class”. Call the class “TraceLogger”.

    public class TraceLogger
    {
        public void Write(string text)
        {
            System.Diagnostics.Trace.WriteLine(text);
        }
    }

Exporting contracts

The goal of managed composition is to decouple the parts in an application, usually through the use of interfaces as ‘contracts’ that a part provides to other parts.

In the Parts folder, create a new interface called “ILogger” and declare a “Write” method on it:

    public interface ILogger
    {
        void Write(string text);
    }

Now go back to the TraceLogger part and implement the ILogger interface:

    public class TraceLogger : ILogger { … }

By convention, the composition provider assumes that any interfaces implemented by a part are the exports that it provides to other parts in the system.

In the included sample application, you can find a number of parts under its Parts folder. The simplest example is TraceLogger, under Parts/Tracing. The FabrikamBooksDbContext part, under Parts/Data, exports the IDbContext contract. The constructor of this part imports an ILogger, showing how parts can consume the contracts provided by other parts.

Using a part from a controller

If your ASP.NET MVC application was created from the default template, you will have a controller class called “HomeController” in the application’s Controllers folder.

Controllers are the primary place to consume parts – by adding a constructor to HomeController that accepts an ILogger, the HomeController can import the ILogger contract supplied by the TraceLogger part:

    public class HomeController : Controller
    {
        ILogger _logger;

        public HomeController(ILogger logger)
        {
            _logger = logger;
        }
        
        public ActionResult Index()
        {
            _logger.Write("Executing the Index() action method.");
            return View();
        }
    }

Whenever the HomeController.Index() action is executed, the message from the _logger.Write() call will be written to the application’s debug output window (try it!).

In the same way that HomeController can import the ILogger contract by declaring a constructor parameter, other parts can import instances of each other. This is how, from small pieces, the managed composition system can build up complex applications to do interesting things.

In the sample application, HomeController can be found under the Controllers folder.

Property imports

Although the composition provider will automatically supply constructor parameters to controllers and other parts, it will not automatically set properties, even when those properties are of a type supplied by another part. To mark a property as being an import, apply the MEF Import attribute to the property declaration.

The composition scopes

Scope or 'lifetime' determines how long a part created by the composition provider will live, and how it will be shared between other parts.

Request scope

When instances of parts are created during the processing of a web request, those parts will be shared within the scope of that web request.

So, if CatalogController requires both a ProductRepository part and a CategoryRepository part, and each repository requires a DataContext, then the same DataContext part instance will be shared by both repositories within a single web request. Within different web requests, different part instances will be created and shared.

Once processing of a request completes, the parts that were created during the web request are no longer required. If the parts implement IDisposable, their Dispose() methods will be called automatically. This is a great way to make sure that resources like database connections, files and service proxies are always released correctly.

In the Fabrikam Books sample, FabrikamDbContext is an example of a request-scoped part. It can be found under Parts/Data.

The application scope

Occasionally, creating a part instance per-request is not the desired behavior. For example, a part implementing an in-memory cache of category names might be created once and then shared by all parts in all requests served by a web application.

To mark a part as being shared at the application level, apply the ApplicationShared attribute:

    [ApplicationShared]
    public class CategoryCache : ICache<Category> { … }

Application-shared parts can only depend on contracts supplied by other application-shared parts.

Be careful when creating such parts, they will need to be thread-safe.

In the sample, TraceLogger is implemented as an application-shared part. It can be found under Parts/Tracing.

Non-shared parts, contract names and wiring

MEF comes with many additional options for wiring parts together. By applying attributes to part types, the defaults for sharing and contract matching can be overridden.

Adding parts from additional assemblies

Sometimes, perhaps because code needs to be shared between multiple applications, a web application will need to be implemented with more than one assembly.

To find parts in assemblies other than the main web application, first, the parts must be in a namespace that includes a segment ‘Parts’ such as “MyApp.OtherAssembly.Parts” or even “MyApp.Parts.OtherAssembly”.

Then, to ensure it is found by the composition provider, the assembly must be explicitly added during application startup in Global.asax.

    protected void Application_Start()
    {
        Assembly otherAssembly = ...;
        CompositionProvider.AddPartsAssembly(otherAssembly);
    }

Having to use a Parts namespace for all parts may seem a little excessive, but it is a good reminder that not every class should be a part. Parts are the ‘coarse-grained’ chunks that make up an application. Assemblies that add non-trivial functionality will almost always have a large number of additional supporting classes that the parts in the assembly use, but that are not parts themselves.

Action filter attributes

Action filter attributes can be applied to ASP.NET MVC controllers or action methods in order to change the way the action is executed. Action filter attributes are not created by the composition provider, but can be provided with contracts via properties marked with the Import attribute.

    public class SaveChangesAttribute : ActionFilterAttribute
    {
        [Import]
        public IDbContext DbContext { get; set; }

        public override void OnActionExecuted(ActionExecutedContext filterContext)
        {
            if (filterContext.Exception == null)
                DbContext.SaveChanges();
        }
    }

The sample includes Parts/Data/SaveChangesAttribute.cs. This is applied to methods on Controllers/HomeController.cs.

Model binders

The final place in ASP.NET MVC where the composition provider can be used is to create model binders. These parts transform HTTP request information into strongly-typed objects that can be used in action methods and elsewhere.

Model binders are regular parts and appear under the Parts folder, but must also be marked with the ModelBinderExport attribute so that the binder is associated with the type of model that it can bind.

    [ModelBinderExport(typeof(Book))]
    public class BookModelBinder : IModelBinder { … }

A full model binder example is given in the Fabrikam Books sample, under Parts/Binding/BookModelBinder.cs.

Summary

The composition provider for ASP.NET MVC is a great step towards making your ASP.NET MVC applications more flexible, testable and maintainable.

Using managed composition to construct parts at runtime is only a small part of the picture. The real value in this technique is that it promotes good design by separating the design of contracts from the parts that implement them.

Some good sources for discussion of these concepts are:

Mark Seemann's book on dependency injection in .NET
The SOLID principles described by Bob Martin

It is our hope that the combination of MEF and ASP.NET MVC 3 makes for an enjoyable and rewarding development experience. We'd love to hear your feedback and suggestions!

Download MEF 2 Preview 4 from the CodePlex site.

Overriding part registration conventions with the MEF attributes [Nick]

BCL Team — Thu, 03 Nov 2011 23:49:00 GMT

This post discusses features in the preview version of MEF, and some details may change between now and the full release.

One of the big advances made in the new MEF preview is a convention-driven programming model aimed at eliminating repetitive attribute usage.

In this new model, the MEF attributes like Export and Import have a new role as the clean and powerful mechanism for overriding the registration conventions.

Configuration by exception

The power in convention-driven rules is that they require 'constant effort' to handle an arbitrary number of parts.

    var registrationBuilder = new RegistrationBuilder();

    registrationBuilder.ForTypesDerivedFrom<IMessageHandler>()
        .Export<IMessageHandler>()
        .SetCreationPolicy(CreationPolicy.NonShared);

The rule above remains unchanged whether there are 10 or 10,000 message handlers in an application, while using explicit configuration for each hander would result in extra effort expended for every handler added.

Inevitably, there are edge cases and parts for which a rule doesn’t quite fit. How these cases are handled has an impact on the effectiveness and maintainability of a convention-based system.

Handling non-conforming parts (“exceptions”) via rules

It is possible to embed exceptions to part discovery/configuration rules into the rules themselves. This approach does not scale well to many parts or parts distributed between loosely-coupled assemblies.

In an application with 50 message handler parts, where 10% have some need of special handling, conventions in MEF's syntax would look like:

    registrationBuilder.ForTypesMatching(t =>
           t != typeof(OrderPlacedMessageHandler) &&
           t != typeof(NewCustomerMessageHandler) &&
           t != typeof(CheckoutMessageHandler) &&
           t != typeof(CheckBouncedMessageHandler) &&
           t != typeof(OutOfStockMessageHandler) &&
           typeof(IMessageHandler).IsAssignableFrom(t))
        .Export<IMessageHandler>()
        .SetCreationPolicy(CreationPolicy.NonShared);

    // Special contract name
    registrationBuilder.ForType<OrderPlacedMessageHandler>()
        .Export<IMessageHandler>(x => x.Named("orders"))
        .SetCreationPolicy(CreationPolicy.NonShared);

    // Default creation policy
    registrationBuilder.ForType<NewCustomerMessageHandler>()
        .Export<IMessageHandler>();

    // Similar single-component rules for remaining three parts above.
    // ...

This has the following issues:

Applying an exception requires tinkering with the host, which is more complicated than changing an individual part
As soon as there is even one exception, the declarative 'syntactic sugar' of ForTypesDerivedFrom() becomes inapplicable
All exceptional types must be visible from the code defining the rule - this does not work well with dynamic discovery e.g. DirectoryCatalog
The rule becomes unreadable and verbose
Keeping the 'shared' aspects of the conventions in sync between the single-component registrations and the general case may introduce errors
The rule text may be a point of source control contention on large product teams

Handling exceptions via attributes

Fortunately, MEF has deep support for attributed configuration, which provides a way to specify exceptions to the rules without the issues discussed above.

In a MEF application, the rule can remain unchanged.

    registrationBuilder.ForTypesDerivedFrom<IMessageHandler>()
        .Export<IMessageHandler>()
        .SetCreationPolicy(CreationPolicy.NonShared);

Then, for example OrderPlacedMessageHandler can have its particular contract name expressed by the addition of an export attribute.

    [Export("orders", typeof(IMessageHandler))]
    class OrderPlacedMessageHandler : IMessageHandler {
    }

The NewCustomerMessageHandler can modify its creation policy.

    [PartCreationPolicy(CreationPolicy.NonShared)]
    class NewCustomerMessageHandler : IMessageHandler {
    }

In contrast with the rule-based approach preceding it, the attributed approach has the following benefits:

A developer applying an exception only needs to deal with the aspect of the part they want to configure, rather than unrelated hosting code
The rule does not change in the presence of exceptions
There is no duplication of the parts of the rule that are not affected by the exception
The rule is not coupled to the implementation types
Developers viewing the 'exceptional' part source code can see that it is treated differently, for example, avoiding threading bugs caused by variations in the sharing model

How attributes and rules interact

Where a rule and an applied attribute overlap, the attribute will always take precendence over the rule.

MEF attributes in a part’s source file always override any rules that affect the same aspect of the same member.

The groups of attributes that are considered ‘aspects’ of a member are:

Export - the Export attribute, ExportMetadata, InheritedExport, custom export and metadata attributes
Import - the Import and ImportMany attributes
Constructor selection - the ImportingConstructor attribute
Part creation policy - the PartCreationPolicy attribute
Part metadata - the PartMetadata attribute

The source code items considered as a single member for the purpose of convention overrides are:

The class declaration
All constructors, considered as one group
A single constructor parameter within a constructor
A single property
A single method

Examples

Let’s take a look at some common situations where attributes may be used to override conventions.

Implementing decorators by changing the exported contract name

In the Decorator Design Pattern, one part 'wraps' another part, providing the same contract but adding funtionality to what the wrapped part provides.

For this scenario, an example convention specifies that public interfaces on a certain set of types are exported:

    registrationBuilder.ForTypesMatching(t => t.Namespace.EndsWith(".Parts"))
        .ExportInterfaces(i => i.IsPublic);

The RemoteProductRepository part provides the IProductRepository contract, while a second part, ProductCache, is introduced in order to improve product retrieval performance.

These are hooked up in a chain, so that other parts importing IProductRepository will be given the cache, while the cache itself imports the underlying repository.

The convention that applies to these types is that they export their implemented interfaces.

    public interface IProductRepository {
        Product GetById(int id);
    }

    [Export("impl", typeof(IProductRepository))]
    class RemoteProductRepository : IProductRepository { ... }

    class ProductCache : IProductRepository {
        public ProductCache([Import("impl")] IProductRepository impl) { ... }
    }

The named export and import ensure that only the ProductCache sees the decorated repository.

Changing export metadata to customise behavior

The TaskRunner component imports task components that it runs periodically. Metadata is used to determine the frequency at which the task runs.

    public interface ITask {
        void Run();
    }

    class TaskRunner : ITaskRunner {
        public TaskRunner(IEnumerable<Lazy<ITask,ITaskFrequency>> tasks) { ... }
    }

Most tasks run hourly, so this is the default by convention.

    registrationBuilder.ForTypesDerivedFrom<ITask>()
        .Export<ITask>(x => x.AddMetadata(“Frequency”, 60));

The SendMailTask runs every 5 minutes and uses an override to specify this.

     [Export(typeof(ITask)), ExportMetadata("Frequency", 5)]
    class SendMailTask : ITask { ... }

Selecting a shorter constructor

By default a part configured with RegistrationBuilder will use the constructor with the most parameters. In this example, the longer constructor is for unit testing purposes only, so the shorter constructor is marked with an attribute.

    class MyPart {
        public MyPart(IFoo foo, IBarTestStub bar) { }

        [ImportingConstructor]
        public MyPart(IFoo foo) { }
    }

Changing the contract name of an import for explicit wiring

If a property is selected as an import by convention, an Import attribute can be applied to the property to override the contract name or other facet of the ImportDefinition. This is useful to implement explicit wiring.

    class MyPart {
        [Import("impl")]
        public IFoo Foo { get; set; }
    }

    class Foo1 : IFoo { }

    [Export("impl", typeof(IFoo))]
    class Foo2 : IFoo { }

Here the export and import attributes specify names so that the Foo import on MyPart will be provided with a Foo2 instance.

Opting out

If a type that would otherwise be matched by a rule should not be used as a part, it can be marked with the PartNotDiscoverable attribute:

    [PartNotDiscoverable]
    class MyPart {… }

Diagnostics

Whenever a rule is overridden by a source-level attribute, you can find a diagnostic message in the debugger trace output. This helps keep track of where the exceptions to the rules are. The message looks like:

    A convention that would apply to member A of type B has been overridden
    by attributes applied in the source file.

Conclusions

When using conventions to register parts in MEF, rules can be kept clean and simple by using the standard MEF attributes to override rules where necessary.

We hope you enjoy working with the new preview; we deeply value your feedback and would love to hear about your experiences via the Discussion Forum or Issue Tracker on the MEF developer preview site.

Coming up on this blog we’ll explore the other major addition to MEF in version 2 – composition scoping.

Download MEF 2 Preview 4 from the CodePlex site.

Getting started with convention-based part registration in MEF 2 [Nick]

BCL Team — Tue, 01 Nov 2011 18:14:00 GMT

This post discusses features in the preview version of MEF, and some details may change between now and the full release.

MEF ‘version 1’ uses attributes to describe how a type should be handled for composition purposes.

    [Export, PartCreationPolicy(CreationPolicy.NonShared)]
    public class HomeController :  Controller
    {
        [ImportingConstructor]
        public HomeController(/* Dependencies here */) { … }
        public ActionResult Index() { … }
    }

Attributes are a good fit for plug-in development when building extensible applications, and we haven't changed any of this experience in the new version. Attributes are still a great way to unambiguously mark-up types for composition.

Planning MEF ‘version 2’ we observed that in large MEF-based applications, only a small proportion of imports and exports are actually extensibility related. A large amount of composition happens to support the internal architecture of the application itself. In many cases, applications that use MEF don’t provide a third-party plug-in model at all, and use MEF purely as a way to bring the benefits of loosely-coupled architecture to monolithic apps.

In the context of a single application, it is much easier to set and communicate rules about how parts should be written, loosening up the need to explicitly specify every detail of every part.

In the example above, if all controllers are exported, all have the NonShared creation policy, and all use a single constructor to receive dependencies, then it is in some ways harmful to repeat that information using attributes on every single controller class. Given these ‘norms’ for a controller, a programmer working in a structured application should be able to write:

    public class HomeController :  Controller
    {
        public HomeController(/* Dependencies here */) { … }
        public ActionResult Index() { … }
    }

…and have the composition aspects inferred automatically. Less repetition leads to more readable code, less overall lines of code and fewer opportunities to make mistakes.

RegistrationBuilder is a new MEF feature coming in .NET 4.5 that provides exactly this kind of experience. Using RegistrationBuilder, the rules defining controller parts look like:

    var conventions = new RegistrationBuilder();
    conventions.ForTypesDerivedFrom<Controller>()
        .Export()
        .SetCreationPolicy(CreationPolicy.NonShared);

Notice that the rule doesn’t say anything about how to find types that derive from Controller. Catalogs still play the role of locating types, while RegistrationBuilder rules replace the attribute syntax. So, to apply these rules to all types in an assembly, an AssemblyCatalog is used:

    Assembly controllersAssembly = // …
    var catalog = new AssemblyCatalog(controllersAssembly, conventions);
    var container = new CompositionContainer(catalog);

Notice the additional constructor parameter accepted by the catalog. All of the familiar MEF catalog types have been updated in the new version to support RegistrationBuilder conventions.

Building Conventions

Using RegistrationBuilder has two aspects:

Identifying types that should be considered parts; and,
Configuring types for use as parts.

Identifying Parts

Each rule is defined by calling one of the three For*() methods on RegistrationBuilder to specify which types the rule will apply to.

ForType<T>() — selects the single type T. For example ForType<HomeController>() would set up a rule for the concrete HomeController type.
ForTypesDerivedFrom<T>() — selects types assignable, but not equal to, a contract type T. This may be a base class or an interface. The example rule for identifying controllers earlier in this article uses this mechanism.
ForTypesMatching(Predicate<Type> predicate) — selects types that match a Boolean selector. For example, ForTypesMatching(t => t.Name.EndsWith("Repository")) will select all types whose name ends with the word “Repository”. This is the most flexible option.

Multiple rules may match the same type. In such cases, as long as the rules don’t overlap in an incompatible way, all matching rules are applied.

The selector methods above return a PartBuilder that can then be used to configure the matching types for use as parts.

Configuring Parts

Rules can be used to configure all aspects of a MEF part:

Exported interfaces and properties, with metadata
Importing constructors, constructor parameters and properties
Part creation policy
Part metadata

Exports

Exports can be configured at the type declaration/interface level, or via properties in the case of property exports.

At the level of the part type itself, the concrete type of the part can be exported (in this case MainWindow):

    conventions.ForType<MainWindow>()
        .Export()

Or, more commonly, one or more of its interfaces will be exported:

    conventions.ForType<MainWindow>()
        .Export<IView>();

Multiple interfaces can be exported using a filter:

    conventions.ForType<MainWindow>()
        .ExportInterfaces(i => i.IsPublic);

Overloads exist for configuring the export with a contract name or with metadata:

    conventions.ForType<MainWindow>()
        .Export<IView>(x => x.AddMetadata("Name", "Main"))

Properties can be exported using a similar syntax. If the concrete type is specified, properties can be selected using a Linq expression:

    conventions.ForType<MainWindow>()
        .ExportProperty(mw => mw.Notifier);

When the concrete type is not known, for example with ForTypesMatching(), properties can be exported by selecting them:

    conventions.ForTypesMatching(t => t.Namespace == "MyApp.Windows")
        .ExportProperties(pi => pi.IsPublic);

Multiple export-related configuration rules on the same part are additive, so a type can export several contracts:

    conventions.ForType<MainWindow>()
        .Export<IView>()
        .Export<ICloseable>();

Imports

Imports are specified either at the constructor or at the property level.

By default, the RegistrationBuilder will select the public constructor with the most parameters as the importing constructor.

To select a different constructor, the SelectConstructor() method is used.

    conventions.ForTypesDerivedFrom<IView>()
        .SelectConstructor(ctors =>
            ctors.Min(ctor => ctor.GetParameters().Length));

When the concrete type of the part is known at configuration time, the constructor can be selected using strongly-typed syntax:

    conventions.ForType<MainWindow>()
        .SelectConstructor(pb => new MainWindow(pb.Import<ILogger>()));

By default, any constructor parameters or imported properties of an enumerable or collection type will be considered to “Import Many”.

The syntax for performing property imports mirrors the syntax described for property exports previously.

Part Creation Policy

By default, parts configured by RegistrationBuilder will use the standard MEF CreationPolicy.Any setting. In normal use, this will boil down to ‘single instance per container’ sharing, unless the part is created using an ExportFactory<T>.

To select the creation policy, use the SetCreationPolicy() method:

    conventions.ForTypesDerivedFrom<Controller>()
        .SetCreationPolicy(CreationPolicy.NonShared);

(Note, the ASP.NET MVC integration also included in the preview interprets CreationPolicy.Any and CreationPolicy.Shared to mean ‘single instance per HTTP request.’ To make a part use ‘single instance per application’ sharing under ASP.NET MVC, you need to apply the ApplicationShared attribute to the part.)

Part Metadata

Part metadata is useful when filtering parts in customized catalogs, or when setting up a CompositionScopeDefinition. Part metadata can be added explicitly, as in:

    conventions.ForType<OutputWindow>()
        .AddMetadata("Mode", "Debug");

Or, it can be calculated from the type:

    conventions.ForType<OutputWindow>()
        .AddMetadata("Mode", t => GetMode(t));

Overriding Rules

Inevitably, there are exceptions to every rule. In these situations, the MEF attributes can often be used to override the rules on a case-by-case basis. This is the topic of an upcoming post – stay tuned!

Download MEF 2 Preview 4 from the CodePlex site.

What’s new in MEF 2 Preview 4? [Alok, Nick]

BCL Team — Thu, 27 Oct 2011 23:15:00 GMT

The MEF team has been quietly at work for some time getting the finishing touches on another developer preview. You can download the new source and .NET 4.0-compatible binaries from the Codeplex site.

This preview is another step towards the next supported release, which will be made as part of .NET 4.5.

Highlights of the new version are:

Support for generic parts
A new convention-based registration API (attributes optional!)
New scoping features that use ExportFactory and CompositionScopeDefinition
Integrated support for MEF composition in ASP.NET MVC
Diagnostics improvements

Open generic parts

Open generic types can now be used as parts. For example, if you are using the Generic Repository pattern, you may use IRepository<T> as a contract:

public interface IRepository<T> where T : class
{
    T FindById(int id);
}

Other parts then import this contract using closed types like IRepository<Customer>:

[Export]
public class CustomersWindow :  Window
{
    [ImportingConstructor]
    public CustomerWindow(IRepository<Customer> customers)
    {
        // … use customers

Rather than create a specific CustomerRepository class, and a similar implementation for other kinds of repository, the repository interface can be exported on a generic type:

[Export(typeof(IRepository<>))]
public class EntityFrameworkRepository<T> : IRepository<T> where T : class
{
    // … generic repository implementation

Open generic parts reduce the effort required to implement many common scenarios.

Convention-based part registration

The previous MEF release was optimized for third-party extensibility scenarios, where explicit annotation of parts with attributes makes a lot of sense. When the entire application is under the control of one group of developers, applying attributes to describe parts can be redundant and repetitive.

For example, every class that derives from Window in an application might have an Export attribute applied:

[Export]
class CustomersWindow : Window { … }

RegistrationBuilder is a class added in the new release that allows a set of rules (‘conventions’) to be defined that map types to parts, without the need for attributes.

var rb = new RegistrationBuilder();
rb.ForTypesDerivedFrom<Window>().Export();

The standard MEF catalogs accept a RegistrationBuilder as a constructor parameter, and will apply the rules defined in the registration builder to the types included in the catalog:

var catalog = new AssemblyCatalog(Assembly.GetExecutingAssembly(), rb);

RegistrationBuilder is available in System.ComponentModel.Composition.Registration.

Composition scoping enhancements

In the first version of MEF, part instances could be either shared or per-importer, but it was much harder to control scope and lifetime in a more fine-grained way. Often it is desirable to configure and manage parts as a group to handle some kind of unit of work – a web request, processing a message or managing a single visual element like a WPF view and ViewModel.

The new MEF release makes it easier to control the lifetime and sharing of parts. This example shows how the new scoping support can be used to isolate the parts used for processing 'messages' — the basic pattern applies to many scenarios.

[Export]
public class DataConnection
{
}
 
[Export]
public class MessageHandler
{
    [ImportingConstructor]
    public MessageHandler(DataConnection dataConnection) { … }

    // HandleMessage(Message m) omitted
}

The two 'skeleton' parts above work together to process a network message of some kind. Each time a message handler is created, it will be initialized with a DataConnection that it will use to make changes in response to the message.

More importantly, other dependencies of MessageHandler (not shown) should share the same DataConnection. MessageHandlers processing different messages should get different DataConnections. Any of these parts may also have dependencies that are global, i.e. common to all scopes.

[Export]
public class MessageListener
{
    [ImportingConstructor]
    public MessageListener(ExportFactory<MessageHandler> handlerFactory) { … }
 
    public void ProcessRequest()
    {
        var message = GetNextMessage();
        using (var handler = _handlerFactory.CreateExport())
            handler.HandleMessage(message);
    }
}

The implementation of MessageListener may be familiar to users of MEF in Silverlight 4. The ExportFactory<T> type allows the MessageListener to create a new message handler for each message.

Now, by default, this won’t enable any kind of scoping around the individual message listeners — they’ll all share the same scope (and DataConnection) as the MessageListener.

To describe the scoping we want, the container needs to be created with a CompositionScopeDefinition in place of a ComposablePartCatalog:

var globalParts = new TypeCatalog(typeof(MessageListener));
var perMessageParts = new TypeCatalog(typeof(MessageHandler), typeof(DbConnection));
var scopeDefinition = new CompositionScopeDefinition(globalParts,
            new[] { new CompositionScopeDefinition(perMessageParts, null)});

var container = new CompositionContainer(scopeDefinition);
var listener = container.GetExportedValue<GlobalRequestManager>();
listener.HandleMessage();
listener.HandleMessage();

Every time HandleMessage() is called, new scoped instances of MessageHandler and DataConnection are created and released.

CompositionScopeDefinition is available in the System.ComponentModel.Composition.Hosting namespace.

Composition provider for ASP.NET MVC

Simplifying the use of MEF in web applications is a core goal of the new release. To further this aim, we are supplying functionality that integrates MEF into ASP.NET MVC, to allow controllers and other MVC types to participate in composition.

To use this feature, reference the System.ComponentModel.Composition.Web.Mvc assembly in your ASP.NET MVC project. All configuration is handled automatically — to create parts, add them to a Parts namespace (folder) under the MVC site:

namespace FabrikamBooks.Parts
{
    class TraceLogger : ILogger { … }

Just by being under the Parts folder, the class will automatically ‘export’ all of its implemented interfaces. The longest constructor on the class will be treated as its importing constructor. No attributes are required, unless you want to override the inbuilt defaults.

Controllers simply take dependencies on parts through their constructors:

class HomeController : Controller
{
    // MEF will automatically supply TraceLogger
    public HomeController(ILogger logger) { … }

The composition provider implements a set of defaults that we feel are sensible for most web applications. These defaults can be overridden by applying attributes to parts. Advanced users can still set up MEF ‘by hand’ to meet more specific goals.

The downloadable binaries for this feature are built against ASP.NET MVC version 3; if you want to rebuild the assemblies from source, you'll need to get ASP.NET MVC separately. To maintain a flexible release schedule as this component evolves, we anticipate delivering binaries via NuGet rather than the .NET Framework SDK.

Diagnostics improvements

Providing good diagnostic support in a framework as dynamic and flexible as MEF is a challenge, and is something that we expect to go on improving over time.

In this release, two key improvements have been made to diagnostics in MEF.

CompositionOptions.DisableSilentRejection

MEF’s roots in extensibility make it more tolerant of composition problems than other similar frameworks. Parts with missing dependencies are identified and excluded from composition, which makes diagnostics hard in some circumstances.

When constructing a CompositionContainer in the new MEF version, flags can be passed controlling this and other behaviors. Specifying CompositionOptions.DisableSilentRejection will cause exceptions to be thrown when any composition problem occurs.

var container = new CompositionContainer(
    catalog, CompositionOptions.DisableSilentRejection);

It is recommended that most users building non-third-party-extensible applications specify this flag. It is set by default when using the ASP.NET MVC composition provider described previously.

Better exception messages

Finally, the exception message text in the first version of MEF can be complex and verbose in some situations. The new release improves the way exception messages are formatted, presenting the most important information up-front.

Moving forward

Subscribe to this blog for examples and more detailed documentation on the new features.

We’re looking forward to your feedback on this release. As always, you can get in touch with the team through this blog or at the MEF Codeplex site’s issue list and discussion forum.

Porting existing .NET code to Metro style apps [Wes]

BCL Team — Tue, 04 Oct 2011 04:25:00 GMT

As many of you know at //build we announced that there will be a new .NET target framework for Metro style apps. To learn more about the goals of this new target framework I encourage you to watch Krzystzof Cwalina’s build talk "A .NET developer's view of Windows 8 app development".

One of the things we know people want to do is reuse existing .NET code but due to the constraints on Metro style apps it is not possible to directly reuse existing .NET class libraries nor is it always possible to simply recompile existing code. We realize this is going to be a pain point for many current .NET developers so we are working on a guide at the .NET for Metro style apps overview page which will assist developers in translating their existing .NET skill set into the Metro style apps world. Sometimes that means explaining why something was removed and other times it means giving another .NET or WinRT API to call instead. Today the page is limited but we are actively working on it internally to update it with more data for Beta and RTM.

For a demo of the kind of things you are likely to run into while porting code I suggest watching Daniel Plaisted’s build talk "Bringing existing managed code into Metro style apps".

While we are looking at many forms of feedback we would greatly appreciate any feedback on issues you run into while porting existing .NET code to Metro style apps. Please leave feedback in the comments or on the forums.

Memory Mapped File Quirks [Greg]

BCL Team — Mon, 06 Jun 2011 20:56:00 GMT

Memory mapped files are segments of virtual memory that are directly mapped to a physical file on disk, byte-by-byte. This technology has a number of benefits over traditional stream based I/O, such as performance during random access to large files, or the ability to share the mapped memory between different threads and processes. The .NET framework provides built-in support for memory mapped files starting from version 4. In most cases using memory mapped files in .NET is as easy as using traditional files streams: just map a file into the memory, open a seekable memory stream and use it as if it was a normal file while enjoying all the performance benefits. However, there are some subtle, but important differences. One of them is that memory mapped file views must always be aligned at the boundaries of the system’s virtual memory pages. An important consequence is that:

Compared to the size of the underlying file on disk, the size of a memory mapped view stream is always rounded up towards a multiple of the system’s virtual memory page size. You must take care to differentiate between useful data and junk bytes used to fill up the space.

A little background

A memory mapped file is a continuous chunk of memory in a process address space that has a direct byte-by-byte mapping to an external resource. Typically, such resource is a data file on the physical hard drive. However, it is also possible to represent device memory or any other OS object that can be accessed via a file handle as a memory mapped file structure. In Windows, memory mapped files can also be backed by the system page file rather than by a separate file system object.

In Windows, managing memory mapped files has been supported for a while (see the function reference). The .NET framework natively supports this technology starting from version 4.0 through the types in the System.IO.MemoryMappedFiles namespace.

In .NET you can create a memory mapping either for an existing file on the physical hard drive by specifying its path, or for an existing mapping by referring to the map name. Alternatively, you can create a new memory mapped file and have the system automatically manage a temporary file on disk to back it up. The temp file will be automatically deleted when you dispose of the memory mapped file; this is particularly useful when using memory mapped files for inter-process communication rather than for data persistence. In any case you need to use one of the static methods in the MemoryMappedFile class to create a memory map instance.

On the OS level, in order to actually read data from or write data to a file mapping you need to first create a view (projection) of that map into your process. Luckily, .NET handles this for you and all you need to do is to create either a MemoryMappedViewAccessor or a MemoryMappedViewStream using the CreateViewAccessor or CreateViewStream methods respectively. You can then use the accessor or the stream for I/O on the memory mapped file. Particularly when using the stream, the approach is not much different from using a traditional FileStream.

Subtle differences (A): The map is not quite the same as the source

Every technology must be fully understood in order to fully realise the improvements it brings over older technologies, and this is no different for memory mapped files. Although a memory map can be usually understood as a byte-by-byte mapping of the underlying resource (file) there are some subtle, but important cases where this does not apply. It is important to be aware of these cases when working with memory mapped files.

For instance, recall that NTFS optionally supports both, compression and encryption of files. Even if compression has not been activated, NTFS can also support sparse files. The idea behind sparse files is, in a nutshell, that if a file contains a very large number of consecutive zeros, the file system performs a very simple form of compression that avoids writing out all those zeros to the disk. Instead, just the number of the zeros is written, thus saving disk space without a measurable performance cost. When the zeros are overwritten with other data, the file system replaces those file chunks with normal data segments. All this happens transparently to the layers above the file system. So, when you create a memory mapping of an encrypted, compressed or sparse file, the memory map holds the respectively decrypted, decompressed or inflated content of the file. As a result, the memory page does not correspond to the physical disk file content on a byte-by-byte level. In most cases you do not need to worry about it. However, in some cases this difference may become important, for instance when you want to copy the map contents to a file manually rather than flushing the view, or when you want to use the memory map contents to reason about the size of the file on disk.

Subtle differences (B): Where are all those zeros coming from?

Another important difference relates to the size or length of the file on disk vs. the size of the memory map. To emphasize the problem, let us consider an example program that reads and displays the contents of a text file that has been mapped to memory:

 
using System;
using System.IO;
using System.IO.MemoryMappedFiles;
using System.Text;
 
 
namespace Microsoft.NetFx.BCL.MSDNBlog {
class MemoryMappedViewStreamLengthDemo {
 
  const String dataFilePath = @"data01.txt";
 
 
  public void Exec() {
 
    using (MemoryMappedFile mmFile = MemoryMappedFile.CreateFromFile(dataFilePath)) {
 
      Stream mmvStream = mmFile.CreateViewStream();
      Console.WriteLine("Stream length: {0}", mmvStream.Length);
            
      using (StreamReader sr = new StreamReader(mmvStream, ASCIIEncoding.ASCII)) {
 
        while (!sr.EndOfStream) {
 
          String line = sr.ReadLine();
          Console.WriteLine("[{0:D4}] \"{1}\"", line.Length, line);
        }
      }            
    }
  }
 
 
  public static void Main(String[] unusedArgs) {
 
    (new MemoryMappedViewStreamLengthDemo()).Exec();
  }
 
}
}

The input file data01.txt looks like this:

This file is 13 bytes long: 9 bytes are taken up by the digit characters, and 4 more by the invisible carriage return (\r) and new line (\n) characters in lines 1 and 2 (line 3 does not have them because there is no new line 4). Here is a hex-binary view of the file:

 
31  31  31  0D  0A  32  32  32  0D  0A  33  33  33
 1   1   1  \r  \n   2   2   2  \r  \n   3   3   3

So naturally, we expect the following output: (note that ReadLine() discards the training \r and \n characters)

Stream length: 13
[0003] "111"
[0003] "222"
[0003] "333"

(Note: the prefix "[0003]" describes the length of each line. It results from the "[{0:D4}]"-part of the WriteLine command in the inner loop of the above example program.)

However, this is not what we get. Instead, we get this (on my machine):

Stream length: 4096
[0003] "111"
[0003] "222"
[4086] "333

































              "

If we step through our demo we can quickly discover that during the last iteration the string variable line consists of three '3'-characters followed by 4083 null characters:

 
    line = "333\0\0\0\0\0 ... \0\0\0";

So what is the reason for this unexpected behaviour and how can we prevent it from happening

Recall that when you create a view of a memory mapped file, Windows aligns the view to virtual memory page boundaries. E.g., if your page size is, say, 4096 bytes, a memory-map view can only be 4096, 8192, 12288, ..., N × 4096 bytes large.

As our data file is smaller than a memory page, we get a view that is as large as a memory page (if the size of virtual memory pages on your machine is not 4096, then the output will also differ accordingly). The memory-map view contains the data supplemented with zeros. When we use a stream reader to read this data, we only reach the EOF mark when we get to the end of the entire view, not to the end of the original data.

One may wish that the .NET APIs were smart enough to notice the size of the underlying file on disk and to use this information to make sure that the stream reader hits an EOF at the end of the original data. However, this is not always possible, even in principle, for three types of reasons:

First, it is not always possible to even determine the path of the backing file on disk. The MemoryMappedFile may have been created not by specifying a physical file, but by referring to a previously memory-mapped file [OS ref]. In such case, it is not possible to determine the backing file path. Recall that it is also possible to create a new MemoryMappedFile without specifying a backing file on disk at all (by either using the system page file of a temp file) [OS ref].

The second reason is that even when the path of the backing disk file is known, it is not in general possible to reliably determine the subset of the memory page that represents actual data versus the subset that is just filling zeros. The memory contents may not directly correspond to the file on disk due to any of the reasons pointed out in the previous section, or simply because the memory has been modified but is not yet flushed.

Finally, the framework APIs do not have the application-specific domain knowledge about whether consecutive zeros could at all be an intended content of the mapped file or must represent filler bytes to the end of the memory page.

Solution: Know your data

On the other hand, application logic may very well be able to easily differentiate between useful and junk data.

For instance, in our particular case we know that the actual data does not contain any null characters, so we could peek ahead and stop reading when we see null characters coming up. Another approach is to define a constant such as

 
    static readonly char[] FillerChars = new char[] { '\0' };

and then to trim the strings as we read them in:

 
    String line = rdr.ReadLine().Trim(FillerChars);

Summary

Memory mapped files provide many performance benefits and can be used for inter-process communication as well as for random file access. When using memory mapped files, developers need to be aware of some quirks. One of these quirks is that the size of a memory mapped file view is always blown up to the next virtual memory page boundary. When accessing data from such memory-map views, it is the responsibility of the application developer to use the domain knowledge in order to determine what memory contents represent original data, and what are junk bytes used to fill the space up to the next virtual memory page boundary.

Aligning Numbers on the Decimal Separator [Ron Petrusha]

BCL Team — Mon, 16 May 2011 01:34:00 GMT

The composite formatting feature in the .NET Framework makes it easy to left-align or right-align a value in a fixed-width field. If the alignment component of a format item in a composite format string is negative, its corresponding argument is left-aligned. If the alignment component is positive, its corresponding argument is right-aligned. For instance, the following example assigns random numbers to an eleven-element array and displays their values. In the output, the array index is left-aligned in a five-character field, and the array value is right-aligned in a twelve-character field.

[Visual Basic]
Dim rnd As New Random()
Dim values(10) As Double
Console.WriteLine("{0,-5}{1,12}", "Index", "Value")
For ctr As Integer = values.GetLowerBound(0) To values.GetUpperBound(0)
    values(ctr) = rnd.NextDouble()
    Console.WriteLine(" {0,-5} {1,12:F6}", ctr, values(ctr))
Next

[C#]
Random rnd = new Random();
double[] values = new double[11];
Console.WriteLine("{0,-5}{1,12}", "Index", "Value");
for (int ctr = values.GetLowerBound(0); ctr <= values.GetUpperBound(0); ctr++)
{
   values[ctr] = rnd.NextDouble();
   Console.WriteLine(" {0,-5} {1,12:F6}", ctr, values[ctr]);
}

The example produces output similar to the following:

Index      Value

 0         0.682236
 1         0.827866
 2         0.927542
 3         0.670535
 4         0.023612
 5         0.353050
 6         0.685052
 7         0.938842
 8         0.494007
 9         0.532631
 10        0.978260

The .NET Framework composite formatting feature doesn’t support additional forms of alignment such as centering a value or aligning numeric values in a field on their decimal points. However, such forms of alignment can be easily implemented by dynamically building a composite format string or by modifying the string representation of the value to be formatted. This blog post will illustrate both approaches by showing how to align a series of numeric values on their decimal point.

The first step is to determine the maximum size of the field that is to display the aligned values. In some cases, the field size is static and is known at compile time. In other cases, the precise range of the values to be formatted is based on the format of data that is known only at runtime. However, the alignment component of a format string must consist of a literal numeric value; you cannot supply a variable that represents that value. For example, the composite format string "The value is {0,12}." indicates that the first argument is to be right-aligned in a twelve-character field. But the composite format string "The value is {0,width}." throws a FormatException.

This means that if we cannot supply a numeric value to the alignment component of a composite format string at design time, we must use string concatenation to dynamically create the format string at runtime. This, in turn, requires an initial pass through the data to be formatted. To align numeric data on a decimal point, we need two items of information:

The number of characters in the data item that has the largest number of integral digits.
The number of characters in the data item that has the largest number of fractional digits.

The sum of these two values, along with the number of characters reserved for the decimal separator, any other formatting styles, and white space, defines the total width of the field. The number of characters in the data item with the largest number of integral digits, along with any formatting styles and leading white space, defines the maximum number of characters that you must accommodate for the value before the decimal separator in the formatted string. For example, to align the values 16.509 and 13052.4 on their decimal point, requires a field width of nine (assuming that we do not want to include any additional styles such as white space or group separators) with a maximum of five integral digits.

The following example illustrates this first pass through an array of numeric values to determine the total field size and the maximum number of integral characters. Note that the example determines the number of characters in a particular numeric value by calling the ToString method with a format specifier.

[Visual Basic]
Dim numbers() As Decimal = { 3D, 15.4D, 19.008D, 18.62D, 1093.425D }
Dim intDigits, decDigits, decPosition As Integer
Dim decimalSeparator As String = NumberFormatInfo.CurrentInfo.NumberDecimalSeparator
For Each number As Decimal In numbers
   Dim value As String = number.ToString("G")
   decPosition = value.IndexOf(decimalSeparator)
   If intDigits < decPosition Then intDigits = decPosition
   If decDigits < value.Length - (decPosition + 1) Then decDigits = value.Length - (decPosition + 1)
Next

Dim fieldWidth As Integer = intDigits + decimalSeparator.Length + decDigits

Console.WriteLine("Maximum number of integral digits: {0}", intDigits)
Console.WriteLine("Maximum number of decimal digits: {0}", decDigits)
Console.WriteLine("Total number of characters: {0}", fieldWidth)
Console.WriteLine()

[C#]
decimal[] numbers = { 3m, 15.4m, 19.008m, 18.62m, 1093.425m };
int intDigits = 0, decDigits = 0, decPosition = 0;
string decimalSeparator = NumberFormatInfo.CurrentInfo.NumberDecimalSeparator;
foreach (decimal number in numbers)
{
   string value = number.ToString("G");
   decPosition = value.IndexOf(decimalSeparator);
   if (intDigits < decPosition) intDigits = decPosition;
   if (decDigits < value.Length - (decPosition + 1)) decDigits = value.Length - (decPosition + 1);
}

int fieldWidth = intDigits + decimalSeparator.Length + decDigits;

Console.WriteLine("Maximum number of integral digits: {0}", intDigits);
Console.WriteLine("Maximum number of decimal digits: {0}", decDigits);
Console.WriteLine("Total number of characters: {0}", fieldWidth);
Console.WriteLine();

As the output shows, aligning this data on its decimal separator requires that each data item be displayed in an eight-character field, and that each data item be allocated space for four integral characters.

Maximum number of integral digits: 4
Maximum number of decimal digits: 3
Total number of characters: 8

Once the first pass through the data determines the total field width and the number of integral characters in each field, a second pass can display the data. To align numeric values on their decimal separator, this second pass must dynamically add the field width to the format string. Since we are aligning the numeric values on their decimal points based on the number of integral characters, the string must define a left-aligned field. When iterating the numeric values in the array, each number must be padded with a sufficient number of leading spaces so that the total length of its integral portion equals the maximum number of integral digits. This allows the numbers to be aligned on their decimal separators.

[Visual Basic]
Dim fmt As String = "Value: {0,-" + fieldWidth.ToString() + "}"
For Each number As Decimal In numbers
   decPosition = number.ToString().IndexOf(decimalSeparator)
   If decPosition = -1 Then decPosition = number.ToString().Length

   Dim value As String = String.Format("{0}{1}", _
                         New String(" "c, intDigits - decPosition), number)

   Console.WriteLine(fmt, value)
Next

[C#]
string fmt = "Value: {0,-" + fieldWidth.ToString() + "}";
foreach (decimal number in numbers)
{
   decPosition = number.ToString().IndexOf(decimalSeparator);
   if (decPosition == -1) decPosition = number.ToString().Length;

   string value = String.Format("{0}{1}", 
                  new String(' ', intDigits - decPosition), number);

   Console.WriteLine(fmt, value);
}

The example displays the following output:

Value:    3
Value:   15.4
Value:   19.008
Value:   18.62
Value: 1093.425

Although this example shows how to align numeric values on their decimal point, you can adopt a similar approach to other forms of alignment. To center a numeric value in a field, for example, requires that you know the total field width. Centering the number is then just a matter of determining the length of the string that represents a particular number and padding it with a number of spaces equal to half of the difference between the total field width and the number’s length.

Unexpected effects of the RegEx SET operator [Greg]

BCL Team — Mon, 02 May 2011 23:24:00 GMT

Regular Expressions (RegEx) are a powerful tool for searching for text that matches specific patterns, but it is also a complex tool that requires care and attention to detail. There are many caveats to using RegEx.
Here, I would like to briefly talk about a common caveat when using RegEx: the set operator [ ] and its effect on the contained tokens...

...(read more)

Parsing Non-Standard Date and Time Formats [Ron Petrusha]

BCL Team — Thu, 14 Apr 2011 00:19:00 GMT

Frequently, particularly when dealing with remote data collection devices, an application receives string data containing date and time information that must be converted to either DateTime or DateTimeOffset values. In these cases, the most commonly used parsing methods, the overloads of the DateTime.Parse or DateTimeOffset.Parse methods, all throw a FormatException when they attempt to perform the conversion, and their corresponding TryParse overloads all return false and assign a data and time of MinValue to the method’s date and time argument. However, the .NET Framework provides two alternative overloaded methods, ParseExact and TryParseExact, that can be used to parse the string representations of date and time values.

In parsing date and time strings, both Parse and TryParse use a NumberFormatInfo object that corresponds either to the current culture or, if one of the overloads with an IFormatProvider parameter is called, to the culture that corresponds to the IFormatProvider argument. The properties of this NumberFormatInfo object determine the acceptable date and time formats that a string can have in order to parse successfully. Even though the number of supported formats is typically sizable, the Parse and TryParse methods cannot parse strings in non-standard formats successfully.

On the other hand, the ParseExact or TryParseExact methods are ideal, since they require that a string exactly conform to a particular definable pattern if the parse operation is to succeed. They are useful when either of the following conditions holds true:

The precise format of a string containing date and time information is known in advance, as it is when the data is gathered by some data collection object or device.
User input must have a particular format.

The overloads of the ParseExact and TryParseExact methods of both the DateTime and DateTimeOffset objects allow you to specify either a single pattern or an array of patterns. You to define a pattern by specifying a format string that consists of either a standard format specifier or one or more custom format specifiers. The input string must then conform to that single pattern or to any one of the patterns in the array. Two of the overloads also include a DateTimeStyles parameter that allows you to define style elements (such as white-space characters) that may be included in the input, or to determine how the input string is to be interpreted (for example, as local or universal time).

To see how the ParseExact and TryParseExact methods can help with parsing date and time strings, let’s take three examples. First, in some cases, date and time data is provided to an application as a set of numbers without any separators (for example, “20100501061245500”). The following code uses a custom date and time format string to do this.

[Visual Basic]
Module Example
   Public Sub Main()
      Dim fmt As String = "yyyyMMddhhmmssFFFFFFF"
      Dim value As String = "20100501061245500"
      Dim dt As DateTime = DateTime.ParseExact(value, fmt, Nothing)
      Console.WriteLine("{0}--> {1}", value, dt)
   End Sub
End Module

[C#]
using System;

public class  Example
{
   public static void Main()
   {
      string fmt = "yyyyMMddhhmmssFFFFFFF";
      string value = "20100501061245500";
      DateTime dt = DateTime.ParseExact(value, fmt, null);
      Console.WriteLine("{0} --> {1}", value, dt);
   }
}

The example displays the following output:

20100501061245500 --> 5/1/2010 6:12:45 AM

Second, in some cases, date and time strings include time zone information expressed as the time zone’s offset from Coordinated Universal Time (UTC). The z, zz, or zzz custom format specifiers can be used in a custom format string to handle such time zone offsets. For example, the following code uses both the ParseExact and TryParseExact methods to parse the string representation of a date and time that contains a time zone offset.

[Visual Basic]
Imports System.Globalization

Module Example 
   Sub Main() 
      Dim value As String = "12/15/2009 6:32 PM -05:00"
      Dim fmt As String = "MM/dd/yyyyh:mm tt zzz"
      Dim date1 As DateTime 
      If DateTime.TryParseExact(value, fmt, Nothing, _
                                DateTimeStyles.AllowWhiteSpaces, date1) Then
         Console.WriteLine("Converted '{0}' to {1}, Kind {2}", _
                           value, date1, date1.Kind) 
      Else 
         Console.WriteLine("Unable to convert '{0} to a date.", value)
      End If

      Try 
         Dim date2 As DateTime = DateTime.ParseExact(value, fmt, _
                                 Nothing, DateTimeStyles.AdjustToUniversal) 
         Console.WriteLine("Converted '{0}' to {1}, Kind {2}", _ 
                           value, date2, date2.Kind) 
      Catch e As FormatException 
         Console.WriteLine("Unable to convert '{0} to a date.",
                           value) 
      End Try
   End Sub
End Module

[C#]
using System;
using System.Globalization;

class Example
{
   static void Main()
   {
      string value = @"12/15/2009 6:32 PM -05:00";
      string fmt = @"MM/dd/yyyy h:mm tt zzz";
      DateTime date1;

      if (DateTime.TryParseExact(value, fmt, null,
                   DateTimeStyles.AllowWhiteSpaces, out date1))
         Console.WriteLine("Converted '{0}' to {1}, Kind {2}",
                           value, date1, date1.Kind);
      else
         Console.WriteLine("Unable to convert '{0} to a date.", value);

      try {
         DateTime date2 = DateTime.ParseExact(value, fmt, null,
                          DateTimeStyles.AdjustToUniversal);
         Console.WriteLine("Converted '{0}' to {1}, Kind {2}",
                           value, date2, date2.Kind); 
      }
      catch (FormatException)
      {
         Console.WriteLine("Unable to convert '{0} to a date.", value); 
      } 
   }
}

The example displays the following output:

Converted '12/15/2009 6:32 PM -05:00' to 12/15/2009 3:32:00 PM, Kind Local
Converted '12/15/2009 6:32 PM -05:00' to 12/15/2009 11:32:00 PM, Kind Utc

Note that the parsing method has not only converted a string to a DateTime value, but that in the process it has also performed a time zone conversion. In the first case, the time was adjusted to the time zone of the computer system on which the application was run (which was U.S. Pacific Standard Time in the case of our example). In the second case, because the ParseExact method was called with the styles parameter set to DateTimeStyles.AdjustToUniversal, the string was converted to Coordinated Universal Time.

It is possible to avoid the loss of time zone information in the input string by calling the DateTimeOffset.ParseExact or DateTimeOffset.TryParseExact method rather than the DateTime.ParseExact or DateTime.TryParseExact method, as the following example code shows.

[Visual Basic]
Imports System.Globalization

Module Example
   Public Sub Main()
      Dim value As String = "12/15/2009 6:32 PM -05:00"
      Dim fmt As String = "MM/dd/yyyy h:mm tt zzz"
      Dim date1 As DateTimeOffset

      If DateTimeOffset.TryParseExact(value, fmt, Nothing, _ 
                               DateTimeStyles.AllowWhiteSpaces, date1) Then
         Console.WriteLine("Converted '{0}' to {1}", value, date1)
      Else
         Console.WriteLine("Unable to convert '{0} to a date.", value)
      End If
   End Sub
End Module

[C#]
using System;
using System.Globalization;

public class Example
{
   public static void Main()
   {
      string value = @"12/15/2009 6:32 PM -05:00";
      string fmt = @"MM/dd/yyyy h:mm tt zzz";
      DateTimeOffset date1;

      if (DateTimeOffset.TryParseExact(value, fmt, null,
                         DateTimeStyles.AllowWhiteSpaces, out date1))
         Console.WriteLine("Converted '{0}' to {1}", value, date1);
      else
         Console.WriteLine("Unable to convert '{0} to a date.", value);
   }
}

The example displays the following output:

Converted '12/15/2009 6:32 PM -05:00' to 12/15/2009 6:32:00 PM -05:00

As the third and final example, assume that our application must parse a string in the form “mm/dd/yy hh:mm:ss PST” submitted by a control device. In this case, the time zone is represented by a predefined time zone abbreviation. In this case, the following code successfully parses the string.

[Visual Basic]
Module Example
   Public Sub Main()
      Dim values() As String = {"12/15/2009 6:32 PM PST", _
                                "1/2/2010 4:18 AM PST", _
                                "01/16/2010 11:17 PM PST"}
      Dim fmt As String = "M/d/yyyy h:mm tt PST";
      Dim dt As DateTime

      For Each value As String In values
         Try
            dt = DateTime.ParseExact(value, fmt, Nothing)
            Console.WriteLine("'{0}' --> {1}", value, dt)
         Catch e As FormatException
            Console.WriteLine("'{0}': Bad Format", value)
         End Try
      Next
   End Sub
End Module

[C#]
using System;

public class Example
{
    public static void Main()
    {
        string[] values = { "12/15/2009 6:32 PM PST",
                            "1/2/2010 4:18 AM PST",
                            "01/16/2010 11:17 PM PST" };
        string fmt = "M/d/yyyy h:mm tt PST";
        DateTime dt;

        foreach (string value in values)
        {
            try
            {
                dt = DateTime.ParseExact(value, fmt, null);
                Console.WriteLine("'{0}' --> {1}", value, dt);
            }

            catch (FormatException)
            {
                Console.WriteLine("'{0}': Bad Format", value);
            }
        }
    }
}

The example displays the following output:

'12/15/2009 6:32 PM PST' --> 12/15/2009 6:32:00 PM
'1/2/2010 4:18 AM PST' --> 1/2/2010 4:18:00 AM
'01/16/2010 11:17 PM PST' --> 1/16/2010 11:17:00 PM

In this case, the ParseExact method recognizes only a single time zone abbreviation at the end of the string. By calling another overload that accepts an array of custom format strings, we can extend the example so that it recognizes multiple time zone abbreviations. The following code uses the DateTimeOffset.TryParseExact method for this purpose.

[Visual Basic]
Imports System.Globalization

Module Example
   Public Sub Main()
      Dim values() As String = {"12/15/2009 6:32 PM PST", _
                                 "1/2/2010 4:18 AM PDT", _
                                 "01/16/2010 11:17 PM CST"}
      Dim fmts() As String = {"M/d/yyyy h:mm tt PST", _
                               "M/d/yyyy h:mm tt PDT", _
                               "M/d/yyyy h:mm tt MST", _
                               "M/d/yyyy h:mm tt MDT", _
                               "M/d/yyyy h:mm tt CST", _
                               "M/d/yyyy h:mm tt CDT", _
                               "M/d/yyyy h:mm tt EST", _
                               "M/d/yyyy h:mm tt EDT"}
      Dim dt As DateTimeOffset

      For Each value As String In values
         If DateTimeOffset.TryParseExact(value, fmts, Nothing,
                           DateTimeStyles.None, dt) Then
            Console.WriteLine("'{0}' --> {1}", value, dt)
         Else
            Console.WriteLine("Cannot parse '{0}'", value)
         End If
      Next
   End Sub
End Module

[C#]
using System;
using System.Globalization;

public class Example
{
   public static void  Main()
   {
      string[] values = { "12/15/2009 6:32 PM PST",
                          "1/2/2010 4:18 AM PDT",
                          "01/16/2010 11:17 PM CST" };
      string[] fmts = { "M/d/yyyy h:mm tt PST",
                        "M/d/yyyy h:mm tt PDT",
                        "M/d/yyyy h:mm tt MST",
                        "M/d/yyyy h:mm tt MDT",
                        "M/d/yyyy h:mm tt CST",
                        "M/d/yyyy h:mm tt CDT",
                        "M/d/yyyy h:mm tt EST",
                        "M/d/yyyy h:mm tt EDT" };
      DateTimeOffset dt;

      foreach (string value in values)
      {
         if (DateTimeOffset.TryParseExact(value, fmts, null,
                               DateTimeStyles.None, out dt))
            Console.WriteLine("'{0}' --> {1}", value, dt);
         else
            Console.WriteLine("Cannot parse '{0}'", value);
      }
   }
}

The example displays the following output:

'12/15/2009 6:32 PM PST' --> 12/15/2009 6:32:00 PM -08:00
'1/2/2010 4:18 AM PDT' --> 1/2/2010 4:18:00 AM -08:00
'01/16/2010 11:17 PM CST' --> 1/16/2010 11:17:00 PM -08:00

But since we are now parsing times from different time zones, the output reveals a significant limitation of this example: it doesn’t recognize the time zone abbreviation in our input strings, and so it assumes that the input string represents the data and time in the local system’s time zone (which in this case is in the U.S. Pacific Standard Time zone). We can make our example time zone-aware by using a generic Dictionary object to look up the time zone identifier from its abbreviation, and supplying the identifier as an argument to the TimeZoneInfo.ConvertTimeBySystemTimeZoneId method. The following is the revised, time zone-aware version of our example.

[Visual Basic]
Imports System.Collections.Generic
Imports System.Globalization
Imports System.Text.RegularExpressions

Module Example
   Public Sub Main()
      ' Create dictionary of time zones.
      Dim timeZones As New Dictionary(Of String, String)
      timezones.Add("PST", "Pacific Standard Time")
      timezones.Add("PDT", "Pacific Standard Time")
      timezones.Add("MST", "Mountain Standard Time")
      timezones.Add("MDT", "Mountain Standard Time")
      timezones.Add("CST", "Central Standard Time")
      timezones.Add("CDT", "Central Standard Time")
      timezones.Add("EST", "Eastern Standard Time")
      timezones.Add("EDT", "Eastern Standard Time")

      Dim values() As String = {"12/15/2009 6:32 PM PST", _
                                "1/2/2010 4:18 AM PDT", _
                                "01/16/2010 11:17 PM CST"}

      Dim fmts() As String = {"M/d/yyyy h:mm tt PST", _
                              "M/d/yyyy h:mm tt PDT", _
                              "M/d/yyyy h:mm tt MST", _
                              "M/d/yyyy h:mm tt MDT", _
                              "M/d/yyyy h:mm tt CST", _
                              "M/d/yyyy h:mm tt CDT", _
                              "M/d/yyyy h:mm tt EST", _
                              "M/d/yyyy h:mm tt EDT"}

      Dim rgx As New Regex("\w(D|S)T", RegexOptions.IgnoreCase)
      Dim dt As DateTime
      Dim tz As String = Nothing

      For Each value As String In values
         Try
            tz = rgx.Match(value).Value
            dt = TimeZoneInfo.ConvertTimeBySystemTimeZoneId( _
                             DateTime.ParseExact(value, fmts, Nothing, DateTimeStyles.None), _
                             timeZones.Item(tz), _
                             TimeZoneInfo.Local.Id)
            Console.WriteLine("'{0}' --> {1}", value, dt)
         Catch e As FormatException
            Console.WriteLine("Cannot parse '{0}'", value)
         Catch e As TimeZoneNotFoundException
            Console.WriteLine("Cannot identify time zone {0}", tz)
         End Try
      Next
   End Sub
End Module

[C#]
using System;
using System.Collections.Generic;
using System.Globalization;
using System.Text.RegularExpressions;

public class Example
{
   public static void Main()
   {
      // Create dictionary of time zones.
      Dictionary<string, string> timeZones = new Dictionary<string, string>();
      timeZones.Add("PST", "Pacific Standard Time");
      timeZones.Add("PDT", "Pacific Standard Time");
      timeZones.Add("MST", "Mountain Standard Time");
      timeZones.Add("MDT", "Mountain Standard Time");
      timeZones.Add("CST", "Central Standard Time");
      timeZones.Add("CDT", "Central Standard Time");
      timeZones.Add("EST", "Eastern Standard Time");
      timeZones.Add("EDT", "Eastern Standard Time");

      string[] values = { "12/15/2009 6:32 PM PST",
                          "1/2/2010 4:18 AM PDT",
                          "01/16/2010 11:17 PM CST" };

      string[] fmts = { "M/d/yyyy h:mm tt PST",
                        "M/d/yyyy h:mm tt PDT",
                        "M/d/yyyy h:mm tt MST",
                        "M/d/yyyy h:mm tt MDT",
                        "M/d/yyyy h:mm tt CST",
                        "M/d/yyyy h:mm tt CDT",
                        "M/d/yyyy h:mm tt EST"
                        "M/d/yyyy h:mm tt EDT" };

      Regex rgx = new Regex(@"\w(D|S)T", RegexOptions.IgnoreCase);
      DateTime dt;
      string tz = null;

      foreach (string value in values)
      {
         try {
            tz = rgx.Match(value).Value;
            dt = TimeZoneInfo.ConvertTimeBySystemTimeZoneId(
                                 DateTime.ParseExact(value, fmts, null, DateTimeStyles.None),
                                 timeZones[tz], TimeZoneInfo.Local.Id);
            Console.WriteLine("'{0}'--> {1}", value, dt);
         }
         catch (FormatException) {
            Console.WriteLine("Cannot parse '{0}'", value);
         }
         catch (TimeZoneNotFoundException) {
            Console.WriteLine("Cannot identify time zone {0}", tz);
         }
      }
   }
}

The example displays the following output:

'12/15/2009 6:32 PM PST' --> 12/15/2009 6:32:00 PM
'1/2/2010 4:18 AM PDT' --> 1/2/2010 4:18:00 AM
'01/16/2010 11:17 PM CST' --> 1/16/2010 9:17:00 PM

Hardening Server Applications [Immo]

BCL Team — Fri, 01 Apr 2011 07:04:00 GMT

From time to time a company ships a product that has a huge impact on their ecosystem. A good example for us is certainly .NET. The biggest value proposition that managed code has is that it is, well, managed code. The CLR provides runtime management components such as a garbage collector or reflection that are aimed at reducing the likelihood of bugs and at increasing the developer's productivity. These features allow developers to focus on building their applications instead of tweaking and massaging mechanics. After all, the hope is that by improving the software that is used for developing other software we improve software on a broader scale (how meta!).

Today we are proud to make an announcement that potentially marks a milestone similar to the PDC 2000 announcement of .NET. Several teams at Microsoft worked during the last years on an upcoming product – code named "Source Code".

The Problem

We all have seen it: a customer reports a bug and after debugging it for a while we realize that the bug fix involves changing a single line. Sometimes, the fix only involves fixing a single character, such as the famous off-by-one error where one only needs to replace "<=" by "<".

Quite frequently, small errors can have devastating effects. For example, an Ariane 5 launch vehicle had to self-destruct due to a single casting error where a 64 bit floating point value was converted to a 16 bit integer. This caused the flight computer to make wrong adjustments due an arithmetic overflow.

How "Source Code" works

Research has investigated several strategies to enable computers to learn. One fruitful approach is genetic programming.

In artificial intelligence, Genetic Programming is an evolutionary algorithm-based methodology inspired by biological evolution to find computer programs that perform a user-defined task.

---Wikipedia

The basic idea is that by mutating existing code and applying a selection-function one can automatically find a computer program that solves a given problem.

During the last years the BCL team, the wider CLR team and Visual Studio worked together with Microsoft Research to build a product around this idea. The result, code named "Source Code", is a combination of new technologies with evolved versions of existing technologies such as IntelliTrace, software transactional memory (STM) and Pex. The basic idea is simple. Whenever the CLR discovers an unhandled exception it rolls back the state of the applicationto a point in time prior to the crash (the default is 8 minutes but can be configured). To do this, the CLR uses a full IntelliTrace recording so that the whole runtime state, including the heap, can be properly restored. Then, the stack trace of the unhandled exception is analyzed to detect which method is the most likely culprit. This information is passed to Pex in order to create a permutated version of the method body ("mutation"). This enables new code paths not previously explored in the application. After that, the CLR resumes execution. If the application crashes again, it repeats the above process until a code path is found that avoids the error ("selection"). Over time, successful code changes persist improving the overall fitness of the application ("survival of the fittest").

Picture of the M5 computer in the Star Trek episode "The Ultimate Computer".

As a result of the above process as application will automatically correct itself over time! The brilliant Dr. Richard Daystrom, the designer of the "M5 Multitronic System" (pictured above) would be very pleased to see this advancement.

Currently the technology is in an early prototype state and we are working on removing some restrictions:

Because of the involved downtimes during rollback and mutation, this scenario will only be available to Windows Azure based ASP.NET applications. For the future, we plan to extend the support to client applications as well.
"Source Code" will only save a limited number of applications. We have seen cases where partially working applications are displaced from the cache by more buggy applications. Our next goal is to improve the cache policy to avoid these situations.
The early CTP does not include support for management and monitoring but we are actively working to get "Source Code" integrated with the Microsoft System Center products.

You can download a CTP here.

Optimizing Regex Performance, Part 3 [Ron Petrusha]

BCL Team — Tue, 29 Mar 2011 04:55:00 GMT

Regular expressions in the .NET Framework support a number of grouping constructs, which allow a regular expression pattern to be grouped into one or more subexpressions. Grouping constructs are essential for creating backreferences, as well as for defining a subexpression to which a quantifier is applied.

The Performance Impact of Capturing Groups

The most commonly used grouping constructs in the .NET Framework regular expression language are (subexpression), which defines a numbered capturing group, and (?<name> subexpression), which defines a named capturing group. The use of these language elements, however, has a definite cost; they cause the GroupCollection object returned by the Match.Groups property to be populated with the most recent unnamed or named captures, and if a single grouping construct has captured multiple substrings in the input string, they also populate the CaptureCollection object returned by the Group.Captures property of a particular capturing group with multiple Capture objects.

For example, the regular expression (\b(\w+)[;,]?\s?)+([.?!]) is designed to capture an entire sentence, and the subexpression (\w+) is designed to capture the individual words in that sentence. (See Regular Expression Language Elements for reference documentation for the regular expression language supported by the .NET Framework.) The regular expression defines three capturing groups, which are listed in the following table. In addition, the first group in the GroupCollection object (at index 0) contains the complete pattern match; the values of its Group.Index, Group.Length, Group.Success, and Group.Value properties are identical to the values of its parent Match object.

Index	Pattern	Description
1	`(\b(\w+)[;,]?\s?)`	Captures a word in the sentence, along with any following punctuation mark and trailing space. The + quantifier is applied to the group so that the regular expression engine will advance beyond the first word in the input string to capture all words in the sentence.
2	`(\w+)`	Captures a word in the sentence
3	`([.?!])`	Captures the punctuation mark used to end the sentence.

The following example uses this regular expression to extract a sentence, its individual words, and its ending punctuation mark from an input string.

[Visual Basic]
Imports System.Text.RegularExpressions

Module Example
   Public Sub Main()
      Dim pattern As String = "(\b(\w+)[;,]?\s?)+([.?!])"

      Dim input As String = "This is a short, silly sentence."

      Dim match As Match = Regex.Match(input, pattern)
      If match.Success Then

         Console.WriteLine("'{0}' found at position {1}",
                           match.Value, match.Index)

         If match.Groups.Count > 1 Then
            For ctr As Integer = 1 To match.Groups.Count - 1
               Dim grp As Group = match.Groups(ctr)
               Console.WriteLine("   Group {0}: '{1}'",
                                 ctr, grp) 

               For cctr As Integer = 0 To grp.Captures.Count - 1
                  Console.WriteLine("Capture {0}: '{1}'",
                                    cctr, grp.Captures(cctr).Value)
               Next
            Next
         End If
      End If
   End Sub
End Module

[C#]
using System;
using System.Text.RegularExpressions;

public class Example
{
   public static void Main()
   {
      string pattern = @"(\b(\w+)[;,]?\s?)+([.?!])";
      string input = "This is a short, silly sentence.";

      Match match = Regex.Match(input, pattern);
      if (match.Success) {
         Console.WriteLine("'{0}' found at position {1}",
                           match.Value, match.Index);
         if (match.Groups.Count > 1)
            for (int ctr = 1; ctr <= match.Groups.Count - 1; ctr++) {
               Group grp = match.Groups[ctr];
               Console.WriteLine("   Group {0}: '{1}'", ctr, grp);  
               for (int cctr = 0; cctr <= grp.Captures.Count - 1; cctr++)
                  Console.WriteLine("     Capture {0}: '{1}'", 
                                    cctr, grp.Captures[cctr].Value);
            }
      }
   }
}

The example displays the following output. Note that the CaptureCollection captured by the first group captures each word in the sentence along with a following punctuation character if it does not conclude the sentence, the CaptureCollection captured by the second group contains each word in the sentence, and the final group contains only the sentence’s final punctuation mark.

'This is a short, silly sentence.' found at position 0
   Group 1: 'sentence'
      Capture 0: 'This '
      Capture 1: 'is '
      Capture 2: 'a '
      Capture 3: 'short, '
      Capture 4: 'silly '
      Capture 5: 'sentence'
   Group 2: 'sentence'
      Capture 0: 'This'
      Capture 1: 'is'
      Capture 2: 'a'
      Capture 3: 'short'
      Capture 4: 'silly'
      Capture 5: 'sentence'
   Group 3: '.'
      Capture 0: '.'

In this regular expression, our first capturing group is present only so that a quantifier can be applied to it. This allows the regular expression engine to advance from one word to the next, and so to capture an entire sentence. The second group is useful if we want to programmatically capture the individual words from the sentence, which are available from the CaptureCollection object. The third is useful if we want to determine which punctuation symbol ended the sentence; it is available from the GroupCollection object.

We can disable captures (so that the GroupCollection and CaptureCollection objects are not populated as matches are found) by using the (?:subexpression) language element. We can assess the impact that capturing groups have on performance by using a StopWatch object to measure the execution time of the following three similar regular expressions when we use each to extract all of the sentences in the text of Theodore Dreiser's The Financier:

The original regular expression, (\b(\w+)[;,]?\s?)+([.?!]).
A regular expression that disables captures by the first group, (?:\b(\w+)[;,]?\s?)+([.?!]). This group exists only so that a quantifier could be applied to it.
A regular expression that has no capturing groups, (?:\b(?:\w+)[;,]?\s?)+[.?!]. It captures the entire sentence only, and disables all capturing groups. (It actually returns a Match object whose Groups property returns a GroupCollection object with a single Group object. This group at index 0 represents the entire match; it is always present and cannot be suppressed.

The source code for the example is:

[Visual Basic]
Imports System.Diagnostics
Imports System.IO

Imports System.Text.RegularExpressions

Module Example
   Public Sub Main()
      Dim patterns() As String = { "(\b(\w+)[;,]?\s?)+([.?!])", 
                                   "(?:\b(\w+)[;,]?\s?)+([.?!])",
                                   "(?:\b(?:\w+)[;,]?\s?)+[.?!]" }
      Dim input As String = (New StreamReader(".\Dreiser_TheFinancier.txt")).ReadToEnd()
      Dim sw As Stopwatch

      For Each pattern In patterns
         ' Instantiate the regular expression object before measuring performance.
         Dim regex As New Regex(pattern)

         ' Time matches.

         sw = Stopwatch.StartNew()

         Dim ctr As Integer = 0
         Dim match As Match = Regex.Match(input, pattern)
         Do While match.Success
            ctr += 1
            match = match.NextMatch()
         Loop

         sw.Stop()

         ' Display performance information.
         Console.WriteLine("'{0}' ({1} capturing groups): {2:N0} matches in {3:ss\.fffff}",
                           regex.ToString(), regex.GetGroupNumbers().Length,
                           ctr, sw.Elapsed)
         Console.WriteLine()
      Next
   End Sub
End Module

[C#]
using System;

using System.Diagnostics;
using System.IO;
using System.Text.RegularExpressions;

public class Example
{
   public static void Main()
   {
      string[] patterns = { @"(\b(\w+)[;,]?\s?)+([.?!])",
                            @"(?:\b(\w+)[;,]?\s?)+([.?!])",
                            @"(?:\b(?:\w+)[;,]?\s?)+[.?!]" };
      string input = (new StreamReader(@".\Dreiser_TheFinancier.txt")).ReadToEnd();
      Stopwatch sw;

      foreach (var pattern in patterns) {
         // Instantiate the regular expression object before measuring performance.

         Regex regex = new Regex(pattern);

         // Time matches.
         sw = Stopwatch.StartNew();

         int ctr = 0;
         Match match = Regex.Match(input, pattern);
         while (match.Success) {
            ctr++;
            match = match.NextMatch();
         }
         sw.Stop();

         // Display performance information.

         Console.WriteLine("'{0}' ({1} capturing groups): {2:N0} matches in {3:ss\\.fffff}\n",
                           regex.ToString(), regex.GetGroupNumbers().Length,
                           ctr, sw.Elapsed);
      }  
   }   
}

As the following output from the example shows, populating the GroupCollection and CaptureCollection objects has a significant impact on the regular expression engine’s performance. The regular expression with three capturing groups executed slightly more than 15faster than the one with four capturing groups, and the regular expression with just one capturing group executed more than 25% faster than the one with four. The poorer performance of the regular expression with four capturing groups is a legitimate cost if programmatic access to these captures is required. However, if it is not, populating the complete regular expression object model constitutes unnecessary overhead.

'(\b(\w+)[;,]?\s?)+([.?!])' (4 capturing groups): 12,633 matches in 02.4953428
'(?:\b(\w+)[;,]?\s?)+([.?!])' (3 capturing groups): 12,633 matches in 02.0867863
'(?:\b(?:\w+)[;,]?\s?)+[.?!]' (1 capturing groups): 12,633 matches in 01.7976809

Capturing and Non-Capturing Grouping Constructs

The regular expression language in the .NET Framework supports a number of grouping constructs, which are documented in detail in the Grouping Constructs topic in the MSDN Library. The following table lists each grouping construct and indicates whether it captures the text matched by the group.

Language Element	Description	Capturing or Non-capturing
`(subexpression)`	Captures a group, which can be referenced by number or retrieved from the GroupCollection object by its ordinal position.	Capturing
`(?<name>subexpression)`	Captures a named group, which can be referenced by name or by number or retrieved from the GroupCollection object by its name or ordinal position.	Capturing
`(?<name1-name2>subexpression)`	Balancing group definition. Tracks nested constructs such as opening and closing parentheses or opening and closing brackets. Capturing groups is integral to the operation of this language element.	Capturing
`(?:subexpression)`	Non-capturing group. Typically, it is used to define a subexpression to which a quantifier can be applied.	Non-capturing
`(?imnsx-imnsx:subexpression)`	Defines options (such as case-sensitivity or the significance of white space) that apply to `subexpression`.	Non-capturing
`(?=subexpression)`	Positive lookahead assertion. Looks ahead of the current position in the input string to determine if `subexpression` is matched.	Non-capturing
`(?!subexpression)`	Negative lookahead assertion. Looks ahead of the current position in the input string to determine if `subexpression` is not matched.	Non-capturing
`(?<=subexpression)`	Positive lookbehind assertion. Looks behind the current position in the input string to determine if `subexpression is matched.`	Non-capturing
`(?<!subexpression)`	Negative lookbehind assertion. Looks behind the current position in the input string to determine if `subexpression` is not matched.	Non-capturing
`(?>subexpression)`	Non-backtracking subexpression. Does not surrender any characters from a matched subexpression to ensure that the regular expression pattern as a whole is matched.	Non-capturing

Disabling Captures

You can disable all captures in a group, or you can disable all but named captures. Unnamed captures can be disabled for a regular expression as a whole, for a part of a regular expression, or for a particular subexpression.

As the previous example showed, the (?:subexpression) language construct disables captures by a group. It disables captures only for the group to which it applies; it does not disable captures by nested or inner groups. In the previous example, for instance, the regular expression pattern (?:\b(\w+)[;,]?\s?)+([.?!]) disables captures by the outer group (group 1). The inner group (\w+) (group 2) is not affected and remains a capturing group.

You can also control implicit captures by using regular expression language options. These options are specified either as members of the RegexOptions enumeration or as inline characters in an options language construct. For more information, see Regular Expression Options.

The RegexOptions.ExplicitCapture option or its corresponding n language option can be used to disable all unnamed or implicit captures; only matches from named groups defined by the (?<name>subexpression) language element are captured.

The RegexOptions.ExplicitCapture option disables implicit captures by the regular expression engine for a particular regular expression or for a particular matching operation. It disables implicit captures for the regular expression if it is passed to the Regex class constructor, and it disables implicit captures for a particular matching operation if it is passed to a static Regex matching method, such as Regex.Match or Regex.Matches. The following example illustrates this. It changes the previous regular expression pattern to (\b(?<words>\w+)[;,]?\s?)+([.?!]) so that it has two implicit capturing groups and one named capturing group. When the Regex object is instantiated with its options parameter set to RegexOptions.ExplicitCapture, only matches made by the named group are captured.

[Visual Basic]
Imports System.Text.RegularExpressions

Module Example

   Public Sub Main()
      Dim pattern As String = "(\b(?<words>\w+)[;,]?\s?)+([.?!])"
      Dim input As String = "This is a short, silly sentence."

      Dim rgx As New Regex(pattern, RegexOptions.ExplicitCapture)
      Dim match As Match = rgx.Match(input)

      If match.Success Then

         Console.WriteLine("'{0}' found at position {1}.", 
                           match.Value, match.Index)
         Console.WriteLine("There are {0} captured groups:", 
                           match.Groups.Count)
         Dim ctr As Integer = 0
         For Each grp As Group In match.Groups
            Console.WriteLine("   Group <{0}>: '{1}' at position {2}.", 
                              rgx.GroupNameFromNumber(ctr), grp.Value,
                              grp.Index)
            Dim capCtr As Integer = 1
            For Each capture In grp.Captures
               Console.WriteLine("      Capture {0}: '{1}' at position {2}.", 
                                 capCtr, capture.Value, capture.Index)
               capCtr += 1
            Next

            ctr += 1
         Next
      End If
   End Sub
End Module

[C#]
using System;
using System.Text.RegularExpressions;

public class Example
{
   public static void Main()
   {
      string pattern = @"(\b(?<words>\w+)[;,]?\s?)+([.?!])";
      string input = "This is a short, silly sentence.";
      Regex rgx = new Regex(pattern, RegexOptions.ExplicitCapture);
      Match match = rgx.Match(input);

      if (match.Success) {
         Console.WriteLine("'{0}' found at position {1}.",
                           match.Value, match.Index);
         Console.WriteLine("There are {0} captured groups:",
                           match.Groups.Count);
         int ctr = 0;
         foreach (Group grp in match.Groups) {
            Console.WriteLine("   Group <{0}>: '{1}' at position {2}.",
                              rgx.GroupNameFromNumber(ctr++), grp.Value, 
                              grp.Index);
            int capCtr = 1;
            foreach (Capture capture in grp.Captures)  
               Console.WriteLine("      Capture {0}: '{1}' at position {2}.",  
                                 capCtr++, capture.Value, capture.Index);
         }
      }
   }
}

The example displays the following output:

'This is a short, silly sentence.' found at position 0.
There are 2 captured groups:
   Group <0>: 'This is a short, silly sentence.' at position 0.
      Capture 1: 'This is a short, silly sentence.' at position 0.
   Group <words>: 'sentence' at position 23.
      Capture 1: 'This' at position 0.
      Capture 2: 'is' at position 5.
      Capture 3: 'a' at position 8.
      Capture 4: 'short' at position 10.
      Capture 5: 'silly' at position 17.
      Capture 6: 'sentence' at position 23.

You can also disable implicit captures for part of a regular expression pattern by using the (?n:subexpression) grouping construct. It disables implicit captures for all implicit groups defined in subexpression. If the construct is applied to the entire regular expression pattern, it is equivalent to instantiating a Regex object with the RegexOptions.ExplicitCapture option. For example, the output would remain unchanged if the regular expression pattern in the previous example were replaced by (?n:(\b(?<words>\w+)[;,]?\s?)+([.?!])). On the other hand, the regular expression pattern (?n:((a+)(b+))+)(c.) disables captures in the subexpression ((a+)(b+))+), including in the two nested expressions in the subexpression.

The (?n) inline element can also be used to disable implicit captures from the point that it is encountered in a regular expression to the end of the expression, or until the implicit captures only option is disabled by the (?-n) inline option. (Note, though, that the (?n:subexpression) syntax can be used for the same result and is significantly clearer.) For example, the regular expression pattern (?n)(\b(?<words>\w+)[;,]?\s?)+(?-n)([.?!]) disables implicit captures in the subexpression (\b(?<words>\w+)[;,]?\s?)+ but enables them in the subexpression ([.?!]). As a result, a match has two capturing groups. The first captures the punctuation mark used to end a sentence. The second captures each word in the sentence.

Summary

The regular expression engine incurs a substantial performance hit when capturing groups appear in a regular expression. In cases where captures are not needed and grouping constructs are used primarily so that quantifiers can be applied to them, captures should be disabled. The regular expression language in the .NET Framework includes a non-capturing grouping construct, (?:subexpression). In addition, implicit or unnamed captures can be disabled by supplying the RegexOptions.ExplicitCapture option to a Regex class constructor or static matching method, or by using the by using the (?n:subexpression) grouping construct or the (?n) inline element.

Announcing Portable Library Tools CTP [Justin Van Patten]

BCL Team — Wed, 19 Jan 2011 20:42:29 GMT

Today we're announcing the CTP release of a new add-in for Visual Studio 2010 that makes it easy to create C# and Visual Basic libraries that run on a variety of .NET platforms without recompilation.

Download the Portable Library Tools CTP today (install VS 2010 SP1 Beta first).

Creating Portable Libraries

The Portable Library Tools CTP adds a new "Portable Class Library" project template to Visual Studio that can be used to create class libraries in C# and VB that run on the various .NET platforms without recompiling.

Portable Class Library projects and assemblies can be referenced and used by .NET 4, Silverlight 4, Windows Phone 7, and XNA for Xbox 360 projects.

Target Platforms Determine Available APIs

Within a Portable Class Library project, you can specify which .NET platforms the library is intended to run on. Although there is a 'core' set APIs that are available on all platforms, there are also some APIs that are only available on certain platforms. An example of an API that exists on some platforms but not others is MEF. Windows Phone and Xbox 360 do not currently have built-in support for MEF, but .NET and Silverlight do. If you want to use MEF within a portable library, the library will currently only run on .NET and Silverlight.

Thus, the project's selected target platforms determine which APIs can be used within the project. The APIs that are shown in intellisense and available to the compiler are automatically filtered based on the selected target platforms, so you don't need to have any special knowledge about each platform or worry about inadvertently using an API that doesn't exist on a platform that you're targeting. The project system takes care of this for you.

In the future, as the underlying platforms evolve to support more APIs that could be portable, we'll update the API surface available to portable libraries as appropriate.

The following table summarizes the high-level functionality currently available on each platform:

Feature	.NET	Silverlight	Windows Phone	Xbox 360
Core BCL	Yes	Yes	Yes	Yes
Core XML	Yes	Yes	Yes	Yes
Core HTTP	Yes	Yes	Yes	-
WCF Client	Yes	Yes	Yes	-
MEF	Yes	Yes	-	-

Note: Once you've installed the Portable Library Tools CTP, you can use the Object Browser in Visual Studio for a more in-depth look at the individual APIs available for use within portable libraries.

You can write a surprising amount of portable code with the subset of APIs currently available to portable libraries. Everything from business logic to web service client proxies can be portable. This enables you to implement most, if not all, application logic in portable assemblies (and then use the portable assemblies in apps that have form-factor-specific UIs for a given platform).

We're still in the process of defining the APIs that will be available to portable libraries, so expect some minor tweaks and additions before the final release of the tools. We'd love to get your feedback on what's currently available in the CTP. Is there anything you think is missing that should be available to portable libraries? How easy was it to make your existing libraries portable?

Known Limitations in the CTP

Deploying Portable Libraries

The CTP of the Portable Library Tools is not a "go live" release. There are still some rough edges that need smoothing out before the final release. One area that does not currently have a complete experience is around deploying portable libraries.

Making portable libraries work on platforms such as .NET Framework 4 and Silverlight 4 required making some minor changes to those platforms. In order for portable libraries to work correctly on these platforms, you'll need to ensure the necessary updates have been applied. Silverlight for Windows Phone 7 and XNA Framework 4.0 for Xbox 360 do not require any updates to run portable assemblies.

.NET Framework 4 Update

An update to .NET Framework 4 is required to run portable libraries. For developers, this update is automatically installed as part of Visual Studio SP1 Beta, so you shouldn't need to install it on a machine that already has VS 2010 SP1 installed. A standalone beta of this update is currently available for download here. The final release of the Portable Library Tools will provide a way for developers to add a prerequisite to their installer to ensure the update is applied on end-user machines when your app is deployed.

Silverlight 4 Update

An update to Silverlight 4 is required to run portable libraries. This update hasn't been released yet, but the impact in the meantime should be limited. As long as you avoid using the following APIs in your portable libraries, they will run just fine on Silverlight 4 without the update:

System.Xml.XmlReader.Dispose
System.Xml.XmlWriter.Dispose
System.Net.WebResponse.Dispose

In the meantime, if you need to dispose these objects, you can wrap them in a using statement or cast the object to IDisposable before calling Dispose.

Once the Silverlight 4 update is available, we will provide instructions on how to specify it as the minimum version required for your application.

Visual Basic Support

The Add Service Reference is currently disabled for VB projects. This will be resolved in a future update.

Summary

We're excited to make a CTP of the Portable Library Tools available for you to try out. Download the CTP today and let us know what you think!

Also, you may be interested in watching Shawn Burke's PDC 2010 session on 3-Screen Coding: Sharing code between Windows Phone, Silverlight, and .NET, which provides some more background on portable libraries.

Time-travel with .NET or DateTime, DateTimeOffset and the lost DST hour [Greg]

BCL Team — Sun, 28 Nov 2010 20:05:00 GMT

Every year again comes the DST change...

And every year again do we need to work with customers on helping them understand some legacy design decisions, and how to work around these ghosts of the past.

Recently, around the end of October and the start of November the daylight saving time (DST) period finished in most areas of the northern hemisphere. As if it was an attempt to make up for the oncoming cold weather, we set our clocks back and got an extra hour of sleep. (In the southern hemisphere, the DST begins at the end of the year, and they turn their clocks forwards.) And one of the basic .NET types – DateTime – does not play well with the annual end-of-DST transition. This issue is well-known and so is the workaround, but nevertheless it gets reported to us regularly by customers who rediscover the issue, especially around the DST transition season. There are a number of resources that treat this topic in great detail. Some of these resources are listed below. Here, I provide some of the background for the reasons behind the issues and a brief refresher on how to solve the problems.

First the gist: The DateTimeOffset type helps avoiding most of the problems associated with DateTime. If you have trouble with DateTime in the context of a DST transition, consider using DateTimeOffset instead.

Issue: Incorrect time zone display.

One issue with the DateTime structure is that during transition from summer time to winter time (i.e. from DST to no-DST), the winter-time time-zone-offset it displayed an hour too early (when shown using a default built-in functionality such as DateTime.Now.ToString("o") ). However, the actual clock time is always displayed correctly. This symptom is caused by the interplay of the fact that DateTime stores the local time only, and the nature of DST transitions.

Similar to the DateTime type, the Windows DST database, on which Windows and .NET applications base their time calculations, specifies the transition times in local time (not UTC). During summer-to-winter time transition in the northern hemisphere the clocks are typically set one hour back. Assume, w.l.o.g., that the transition occurs at 3:00. Then, the times between 2:00 and 2:59 will occur twice: once during summer time, and once again during winter time after the clock has been set back. When the system is queried whether a time, say, 2:28, is in summer (say UTC -7) or in winter time (UTC -8), there is no way for the system to know which one is correct. In such cases, .NET assumes winter time, as that is typically the default time for the time zone (i.e. no DST). Note that if we changed this behaviour, the problem would still occur, only the other way around. For instance, the old DST state would show up one hour too long.

In this particular case, the actual time is always shown correctly, it is just the DST state that is incorrect.

Solution: Track the DST mode changes.

At one second past the hour (or any other pre-specified time), store the local system time. Then compare the current time against the timestamp stored when the one-second-part-the-hour event occurred previously. If they match, we know that we just transitioned to winter time: the offset has changed (e.g. using the previous example, to UTC-8). Otherwise, we are still in summer time: it is still UTC-7.

This workaround is complex and applies only to applications that run for at least one hour. This is not the best approach for a general fix.

Better solution: Use a type that stores time as UTC instead of local time.

UTC (coordinated universal time) describes the time at the Greenwich meridian and does not undergo any DST transitions. In a way (simplified but appropriate for this context), it can be understood as the “universal cosmological time”. The DateTimeOffset type stores the local time as a pair of values: one value describes the UTC time, and the other value describes the offset between UTC and the local time. For instance, 02:28 in the UTC-7 time zone is stored as {UtcTime - Offset} = {10:28 - 420minutes}. 02:28 in the UTC-8 time zone is stored as {11:28 - 480minutes}. In contrast to DateTime, these two values, although describing the same local time, describe two different points in time, and can be easily differentiated from each other. As a result, issues related to DST transitions can be handled correctly when using DateTimeOffset.

Putting it all together: Recommendations.

The reading list below contains a wealth of background information on the DateTime and DateTimeOffset types and how to use them. Here is a short and practical summary of how these types should be used.

Use `DateTime` to:	Use `DateTimeOffset` to:
Represent dates only (no time information). Represent times only (not on a specific date). (Alternatively, consider using the `TimeSpan` type to represent the time of day without a specific date context). Work with abstract times and dates, such as historical dates. Work with legacy APIs that do not support `DateTimeOffset` (e.g. some database and file system APIs).	Uniquely and unambiguously identify a single point in time. Log events occurring at specific times. Perform general date and time arithmetic. Work with time zones. Work with daylight saving times.

More guidance on choosing the best date and time type for your circumstances is provided through the links below.

The Caveats of Time Zone Names [Greg]

BCL Team — Tue, 12 Oct 2010 01:18:00 GMT

Hi Everyone!

My name is Greg, I joined the NetFx Base Class Libraries (BCL) team a few months ago, and I am really excited about my work on some very cool new framework features. Before joining the BCL team I was working in academic research in complex systems, network theory and artificial life at Monash University in Australia, and previously I was running a small software development and consulting company with clients in Germany and the UK.

Besides adding some awesome new stuff to the next release for the framework (can’t talk about that yet!) I am also maintaining and improving some of the existing namespaces. This involves fixing some bugs and helping our customers by explaining how to use the framework on the Connect portal. There are a few issues that come up regularly and are particularly tricky to deal with, so I would like to talk about them briefly to a wider audience and in a more generic context (i.e. here). Today I will be talking about time zones and, in particular, potential pitfalls associated with their naming.

In .NET, the class that provides most of the functionality for dealing with time zones, converting times from one zone to another, and working with daylight saving time (DST), is System.TimeZoneInfo. In turn, the data that TimeZoneInfo uses for its calculations is provided by Windows in the registry under this key:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Time Zones

(Note: be very careful if you touch the registry, never change anything if you are not absolutely sure what you are doing, and always create a back-up before your changes.)

In order to access information about a specific time zone in a .NET application, you need to instantiate a TimeZoneInfo object that represents that time zone using the factory method TimeZoneInfo.FindSystemTimeZoneById(String id) (see also here for more information). That factory method takes a string parameter specifying a time zone name and returns a TimeZoneInfo instance that represents the specified time zone. It provides information including offset from the universal standard time (UTC), information about DST adjustments, time conversion routines and more.

The difficulty is in providing the correct ID (name) of a time zone in order to get hold of the corresponding object: The naming for the time zones is not always intuitive. As a result, it is easy to get confused and to create program bugs and unexpected behaviour. Consider for instance the zone named "US Eastern Standard Time". Unexpectedly this is not the EST/EDT observed on most of the US east coast. Instead, it is a US-Indiana-Only time zone that is not used any more since 2006 and does not change to DST. The name of the time zone that people are probably looking for instead is "Eastern Standard Time". That is the ID you need to get the TimeZoneInfo object that represents the time zone used in most of east North America. That time zone does use DST.

Another time zone with a misleading name is "Greenwich Standard Time". This zone applies to places like Monrovia (Liberia) and Reykjavik (Island). These places do not observe DST in 2010. This zone is easily confused with "GMT Standard Time" that contains Portugal and the UK and does observe DST.

To exemplify the issue, consider the following code sample. Its output is given below.

using System;
 
namespace Microsoft.Misc.MSDNBlog {
public static class ConfusingTimeZoneNames_Demo {
 
private static String[] zoneIDs = new String[] {
            "China Standard Time",
            "AUS Eastern Standard Time",
            "Pacific Standard Time",
            "US Eastern Standard Time",
            "Eastern Standard Time",
            "Greenwich Standard Time",
            "GMT Standard Time"
        };
 
 
public static void Main(String[] unusedArgs) {
 
  const String format = "{0,-26} | {1} {2}| {3}";
  DateTime utcTime = new DateTime(2010, 06, 01, 12, 30, 0);
 
  Console.WriteLine(format, "Time Zone Name (ID)",
                            "Local Time      ",
                            "DST?",
                            "Time Zone Information");
  Console.WriteLine("---------------------------+------------"
          + "----------+---------------------------------------");
 
  foreach (String tzID in zoneIDs) {
 
    TimeZoneInfo tzInfo = TimeZoneInfo.FindSystemTimeZoneById(tzID);
    DateTime tzTime = TimeZoneInfo.ConvertTime(utcTime, tzInfo);
 
    Console.WriteLine(format, tzID,
              tzTime.ToString("yyyy-MM-dd HH:mm"),
              tzInfo.IsDaylightSavingTime(tzTime) ? "DST " : "    ",
              tzInfo.DisplayName);
  }        
}
 
}  // public static class ConfusingTimeZoneNames_Demo
}  // namespace Microsoft.Misc.MSDNBlog

To summarise: Be careful when instantiating TimeZoneInfo objects. Always double-check that the zone name you are using is the correct one, even if it appears obvious at first. And if your application behaves weirdly, double-check if you have possibly created the wrong TimeZoneInfo object due to a similarity in name IDs. For more information about time zones in various places around the world, check out http://www.timeanddate.com/worldclock/full.html

Enjoy!

Optimizing Regular Expression Performance, Part II: Taking Charge of Backtracking [Ron Petrusha]

BCL Team — Tue, 03 Aug 2010 20:50:00 GMT

One of the most powerful features of regular expressions in the .NET Framework -- and of Nondeterministic Finite Automaton (NFA) regular expression engines generally -- is their ability to execute in a non-linear manner. That is, instead of advancing one character at a time, they can return to a previous saved state in order to match a regular expression pattern. For example, if you try to match the string “A match.” with the regular expression pattern .+\. the regular expression engine first attempts to match the entire string. It then retreats one character so that the .+ construct matches “A match” and \. matches the period at the end of the string. This ability to return to a previous saved state in order to continue the match is known as backtracking.

Note: This is the second part of a three-part series on optimizing regular expression performance. The first part, Optimizing Regular Expression Performance, Part I: Working with the Regex Class and Regex Objects, discusses the efficient use of Regex static and instance methods.

Although backtracking contributes to the power and flexibility of regular expressions, it can have an enormous negative impact on performance. In a worst-case scenario, when backtracking is used excessively, it might take one or more days for the regular expression engine to process even a relatively short input string that does not match the regular expression pattern.

The NFA Engine places control over backtracking (and therefore responsibility for the consequences of backtracking) in the hands of regular expression developers who may be unaware of how backtracking affects performance. This fact often goes overlooked both when creating regular expressions and when assessing regular expression performance. As a result, the inappropriate use of backtracking is one of the fundamental performance bottlenecks that applications relying on regular expressions encounter.

Backtracking occurs when a regular expression contains the following language constructs and under the following conditions:

An alternation construct, if the first alternative does not provide a successful pattern match.
An indeterminate quantifier (such as * or +), when the greedy match resulting from the expression to which the quantifier is applied prevents a successful match.
A conditional expression, if the first condition does not provide a successful pattern match.

When the regular expression engine encounters one of these constructs or expressions, it saves its current state so that it can return to it if necessary.

We’ll first examine backtracking with alternation, and then examine backtracking with nested quantifiers. We’ll finish by looking at a worst case scenario in which a poor use of backtracking causes execution time to increase exponentially as the number of characters in the string to be matched increases. In each case, we’ll attempt to show that an alternative pattern that better controls backtracking offers superior performance.

Note: In the code examples in this article, we make extensive use of the System.Diagnostics.Stopwatch class to measure regular expression performance. These measurements are used only for comparative purposes, and reflect performance on a particular system. They are not intended to be construed as benchmarks.

Alternation and Backtracking

The alternation construct (|) allows a regular expression pattern to include the equivalent of an Or operator; the regular expression engine will match text that conforms to one of two or more patterns. The regular expression engine performs no optimization on alternation constructs.

Because alternation is a backtracking construct, it can entail a significant performance penalty if the alternation pattern is not well crafted. In this section, we’ll examine some of the factors to consider when constructing an alternation pattern that will perform efficiently.

Order Matters

Alternation constructs are evaluated sequentially from left to right. The second alternative is evaluated only if the first alternative fails. If there are three or more alternatives, the third alternative is evaluated only if the first and second alternatives fail, and so on. Because of this, the ordering of items in an alternation construct is significant. Subpatterns that are more likely to be encountered in an input string should precede subpatterns that are less likely to be encountered.

The following example uses an alternation construct to find words followed by one of a number of punctuation symbols in the text of Theodore Dreiser’s The Financier. In the first regular expression pattern, punctuation symbols that are less likely to be encountered precede symbols that are more frequently encountered. In the second, punctuation symbols that are more likely to be encountered precede symbols that are less frequently encountered.

[Visual Basic]
Option Strict On

Imports System.Diagnostics
Imports System.IO
Imports System.Text.RegularExpressions

Module Example
   Public Sub Main()
      ' Organize the punctuation marks in a poor order.
      Dim pattern1 As String = "\b\w+(;|:|!|\?|\.|,)\s"
      Dim rgx1 As New Regex(pattern1, RegexOptions.SingleLine)
      Dim sw As Stopwatch
      Dim sr As New StreamReader(".\Dreiser_TheFinancier.txt")
      Dim input As String = sr.ReadToEnd()
      sr.Close()

      sw = Stopwatch.StartNew()
      Dim match1 As Match = rgx1.Match(input)
      Dim ctr1 As Integer
      Do While match1.Success
         ctr1 += 1
         match1 = match1.NextMatch
      Loop
      sw.Stop()
      Console.WriteLine("Found {0} matches in {1}", ctr1, sw.Elapsed)

      ' Organize the punctuation marks in rough order of occurrence.
      Dim pattern2 As String = "\b\w+(\.|,|\?|;|:|!)\s"
      Dim rgx2 As New Regex(pattern2, RegexOptions.SingleLine)

      sw = Stopwatch.StartNew()
      Dim match2 As Match = rgx2.Match(input)
      Dim ctr2 As Integer
      Do While match2.Success
         ctr2 += 1
         match2 = match2.NextMatch
      Loop
      sw.Stop()
      Console.WriteLine("Found {0} matches in {1}", ctr2, sw.Elapsed)
   End Sub
End Module

[C#]
using System;
using System.Diagnostics;
using System.IO;
using System.Text.RegularExpressions;

public class Example
{
   public static void Main()
   {
      // Organize the punctuation marks in a poor order.
      string pattern1 = @"\b\w+(;|:|!|\?|\.|,)\s";
      Regex rgx1 = new Regex(pattern1, RegexOptions.Singleline);
      Stopwatch sw;
      StreamReader sr = new StreamReader(@".\Dreiser_TheFinancier.txt");
      string input = sr.ReadToEnd();
      sr.Close();

      sw = Stopwatch.StartNew();
      Match match1 = rgx1.Match(input);
      int ctr1 = 0;
      while (match1.Success)
      {
         ctr1++;
         match1 = match1.NextMatch();
      }
      sw.Stop();
      Console.WriteLine("Found {0} matches in {1}", ctr1, sw.Elapsed);

      // Organize the punctuation marks in rough order of occurrence.
      string pattern2 = @"\b\w+(\.|,|\?|;|:|!)\s";
      Regex rgx2 = new Regex(pattern2, RegexOptions.Singleline);

      sw = Stopwatch.StartNew();
      Match match2 = rgx2.Match(input);
      int ctr2 = 0;
      while (match2.Success)
      {
         ctr2++;
         match2 = match2.NextMatch();
      }
      sw.Stop();
      Console.WriteLine("Found {0} matches in {1}", ctr2, sw.Elapsed);
   }
}

Output from the example shows that there is a small performance benefit from ordering the alternatives based on how likely they are to be encountered in the input string. The performance improvement is approximately 6% for this very simple regular expression, as the following graph shows. In a more complex regular expression pattern, particularly if he alternation pattern is followed by additional backtracking constructs, the performance benefit is likely to be much greater.

Found 26930 matches in 00:00:00.9649407
Found 26930 matches in 00:00:00.9044185

Optimization Matters

Because alternation constructs involve backtracking, optimization is critical. Alternation patterns that are poorly written and that include duplicate elements perform much more poorly than those that do not. In the following example, the regular expression \b(the|this|that|then|there|these|those)\b matches a number of words starting with “th” (the, this, that, then, there, these, those) in a string. Because of backtracking, the regular expression engine performs the following steps to match the substring “these” in the input string:

1	It determines that the current position in the input string is on a word boundary. Because a grouping and alternation construct follow, it saves this position in the input string.
2	It begins comparing the “the” alternative with the input string. It compares “t” in the regular expression pattern with “t” in the input string and finds a match.
3	It compares “h” in the regular expression pattern with “h” in the input string and finds a match.
4	It compares “e” in the regular expression pattern with “e” in the input string and finds a match.
5	It determines whether the current position in the input string is on a word boundary. Because it is in the middle of the substring “these”, the match with the first alternation pattern fails, and the regular expression engine returns to its saved position (the beginning of the substring “these”) in the input string.
6-8	It compares the “this” alternative with the input string. It matches “t” and “h”, but fails to match “i" with “e” in the input string. The regular expression engine returns to its saved position in the input string.
9-11	It compares the “that” alternative with the input string. It matches “t” and “h”, but fails to match “a" with “e” in the input string. The regular expression engine returns to its saved position in the input string.
12-15	It compares the “then” alternative with the input string. It matches “t”, “h”, and “e”, but fails to match “n" with “s” in the input string. The regular expression engine returns to its saved position in the input string.
16-19	It compares the “there” alternative with the input string. It matches “t”, “h”, and “e”, but fails to match “r" with “s” in the input string. The regular expression engine returns to its saved position in the input string.
20-25	It compares the “these” alternative with the input string. It matches “t”, “h”, “e”, “s”, and “e”. It then determines that the current position in the input string is on a word boundary. The match succeeds. The last alternative in the regular expression (those) is not tested.

By using alternation only for unique portions of the regular expression pattern, we can improve the efficiency of string comparisons. In our example, because the substring “th” is common to all alternation patterns, we can rewrite the regular expression pattern as follows:

\bth(e|is|at|en|ere|ese|ose)\b

Since individual alternatives still share characters in common, we can further optimize this regular expression pattern through grouping:

\bth(e(n|re|se)|at|is|ose)\b

1	It determines that the current position in the input string is on a word boundary.
2	It matches “t” in the pattern with “t” in the input string.
3	It matches “h” in the pattern with “h” in the input string. Because a grouping and alternation construct follows, it saves this position in the input string.
4	It matches “e” in the pattern with “e” in the input string. Because a grouping and alternation construct follows, it saves this position (the second saved position) in the input string.
5	It attempts to match “n” in the pattern with “s” in the input string. The match fails, and the regular expression engine returns to its second saved position in the input string.
6	It attempts to match “r” in the pattern with “s” in the input string. The match fails, and the regular expression engine returns to its second saved position in the input string. Note that it does not test the “e” because it failed to match “r”.
7-8	It attempts to match “se” in the pattern with “se” in the input string. The match succeeds.
9	It determines that the current position in the input string is on a word boundary. The match succeeds.

The following is the complete example.

[Visual Basic]
Option Strict On

Imports System.Diagnostics
Imports System.IO
Imports System.Text.RegularExpressions

Module Example
   Public Sub Main()
      Dim sw As Stopwatch
      Dim sr As New StreamReader(".\Dreiser_TheFinancier.txt")
      Dim input As String = sr.ReadToEnd()
      sr.Close()
      Dim ctr As Integer

      sw = Stopwatch.StartNew()
      ctr = 0
      Dim unoptPattern As String = "\b(the|this|that|then|there|these|those)\b"
      Dim rgxUnopt As New Regex(unoptPattern, RegexOptions.IgnoreCase)
      Dim match1 As Match = rgxUnopt.Match(input)
      Do While match1.Success
         ctr += 1
         match1 = match1.NextMatch()
      Loop
      sw.Stop()
      Console.WriteLine("{0}:{3}   Found {1:N0} matches in {2}", 
                        unoptPattern, ctr, sw.Elapsed, vbCrLf)
      Console.WriteLine()

      sw = Stopwatch.StartNew()
      ctr = 0
      Dim optPattern As String = "\bth(e(n|re|se)|at|is|ose)\b"
      Dim rgxOpt As New Regex(unoptPattern, RegexOptions.IgnoreCase)
      Dim match2 As Match = rgxOpt.Match(input)
      Do While match2.Success
         ctr += 1
         match2 = match2.NextMatch()
      Loop
      sw.Stop()
      Console.WriteLine("{0}:{3}   Found {1:N0} matches in {2}", 
                        optPattern, ctr, sw.Elapsed, vbCrLf)
   End Sub
End Module

[C#]
using System;
using System.Diagnostics;
using System.IO;
using System.Text.RegularExpressions;

public class Example
{
    public static void Main()
    {
        Stopwatch sw;
        StreamReader sr = new StreamReader(@".\Dreiser_TheFinancier.txt");
        string input = sr.ReadToEnd();
        sr.Close();
        int ctr;

        sw = Stopwatch.StartNew();
        ctr = 0;
        string unoptPattern = @"\b(the|this|that|then|there|these|those)\b";
        Regex rgxUnopt = new Regex(unoptPattern, 
                                   RegexOptions.IgnoreCase);
        Match match1 = rgxUnopt.Match(input);
        while (match1.Success)
        {
            ctr++;
            match1 = match1.NextMatch();
        }
        sw.Stop();
        Console.WriteLine("{0}:\n   Found {1:N0} matches in {2}\n", 
                          unoptPattern, ctr, sw.Elapsed);

        sw = Stopwatch.StartNew();
        ctr = 0;
        string optPattern = @"\bth(e(n|re|se)|at|is|ose)\b";
        Regex rgxOpt = new Regex(unoptPattern,
                                 RegexOptions.IgnoreCase);
        Match match2 = rgxOpt.Match(input);
        while (match2.Success)
        {
            ctr++;
            match2 = match2.NextMatch();
        }
        sw.Stop();
        Console.WriteLine("{0}:\n   Found {1:N0} matches in {2}", 
                          optPattern, ctr, sw.Elapsed);
    }
}

The optimized regular expression offers approximately a 35% performance improvement over the unoptimized regular expression, as indicated by the following output and graph.

\b(the|this|that|then|there|these|those)\b:
   Found 13,459 matches in 00:00:00.1833098

\bth(e(n|re|se)|at|is|ose)\b:
   Found 13,459 matches in 00:00:00.1184890

Optional Quantifiers and Backtracking

The regular expression engine also uses backtracking whenever a subexpression includes an optional quantifier (such as * or +). For example, the regular expression pattern \b\p{Lu}\w*\b matches all words beginning with an uppercase character. To allow for the possibility that a word might consist of a single uppercase letter, the \w* subexpression matches any word character zero or more times. Because * is an optional quantifier, the regular expression engine saves its state with each word character matched, in case it has to abandon a portion of the match and return to a previous saved state in order to match the input string with a later portion of the regular expression pattern.

Is Backtracking Necessary?

An important question to ask about any regular expression that uses backtracking is whether, in view of its associated performance penalty, backtracking is necessary for a successful match. Backtracking often is necessary because regular expressions are greedy, and a greedy match of a subexpression is likely to consume some portion of what would otherwise be a successful match. But in many cases, backtracking is used unintentionally, because optional quantifiers that backtrack are more familiar and frequently used than their alternatives. However, backtracking comes at a price. Each time an optional quantifier is applied to an input string, the regular expression engine must save the state of the match at that point so that it can return to it if a later portion of the match fails. To put it another way, needless backtracking exacts a substantial performance penalty.

Let’s return to our example regular expression pattern, \b\p{Lu}\w*\b. Because a word boundary is not the same as, or a subset of, a word character, there is no possibility that the regular expression engine will cross a word boundary when matching word characters. This means that for this regular expression, backtracking can never contribute to the overall success of any match -- it can only extract a performance penalty, because the regular expression engine is forced to save its state for each successful preliminary match of a word character.

In cases where backtracking is unnecessary, it can be disabled by using the (?>subexpression) language construct, where subexpression is the regular expression subpattern that should not backtrack. To disable backtracking, we can change our regular expression from \b\p{Lu}\w*\b to \b\p{Lu}(?>\w*)\b. The following example uses these two regular expressions to extract words from the text of Theodore Dreisier’s The Financier.

[Visual Basic]
Imports System.Diagnostics
Imports System.IO
Imports System.Text.RegularExpressions

Module Example
   Public Sub Main()
      Dim sw As Stopwatch
      Dim sr As New StreamReader(".\Dreiser_TheFinancier.txt")
      Dim input As String = sr.ReadToEnd()
      sr.Close()
      Dim ctr As Integer

      sw = Stopwatch.StartNew()
      ctr = 0
      Dim backtrackingPattern As String = "\b\p{Lu}\w*\b"
      Dim backtrackingRegex As New Regex(backtrackingPattern)
      Dim match1 As Match = backtrackingRegex.Match(input)
      Do While match1.Success
         ctr += 1
         match1 = match1.NextMatch()
      Loop
      sw.Stop()
      Console.WriteLine("{0}:\n   Found {1:N0} matches in {2}\n", 
                        backtrackingPattern, ctr, sw.Elapsed)

      sw = Stopwatch.StartNew()
      ctr = 0
      Dim noBacktrackPattern As String = "\b\p{Lu}(?>\w*)\b"
      Dim nonBacktrackingRegex As New Regex(noBacktrackPattern)
      Dim match2 As Match = nonBacktrackingRegex.Match(input)
      Do While match2.Success
         ctr += 1
         match2 = match2.NextMatch()
      Loop
      sw.Stop()
      Console.WriteLine("{0}:\n   Found {1:N0} matches in {2}", 
                        noBacktrackPattern, ctr, sw.Elapsed)
   End Sub
End Module

[C#]
using System;
using System.Diagnostics;
using System.IO;
using System.Text.RegularExpressions;

public class Example
{
    public static void Main()
    {
        Stopwatch sw;
        StreamReader sr = new StreamReader(@".\Dreiser_TheFinancier.txt");
        string input = sr.ReadToEnd();
        sr.Close();
        int ctr;

        sw = Stopwatch.StartNew();
        ctr = 0;
        string backtrackingPattern = @"\b\p{Lu}\w*\b";
        Regex backtrackingRegex = new Regex(backtrackingPattern);
        Match match1 = backtrackingRegex.Match(input);
        while (match1.Success)
        {
            ctr++;
            match1 = match1.NextMatch();
        }
        sw.Stop();
        Console.WriteLine("{0}:\n   Found {1:N0} matches in {2}\n",
                          backtrackingPattern, ctr, sw.Elapsed);
        sw = Stopwatch.StartNew();
        ctr = 0;
        string noBacktrackPattern = @"\b\p{Lu}(?>\w*)\b";
        Regex nonBacktrackingRegex = new Regex(noBacktrackPattern);
        Match match2 = nonBacktrackingRegex.Match(input);
        while (match2.Success)
        {
            ctr++;
            match2 = match2.NextMatch();
        }
        sw.Stop();
        Console.WriteLine("{0}:\n   Found {1:N0} matches in {2}",
                          noBacktrackPattern, ctr, sw.Elapsed);
    }
}

Because it does not have to keep track of previous saved states, the regular expression that explicitly disables backtracking executes approximately 40% faster than the one that uses unnecessary backtracking. The following output from this example and graph illustrate the difference in performance.

\b\p{Lu}\w*\b:
   Found 24,157 matches in 00:00:00.2465120

\b\p{Lu}(?>\w)*\b:
   Found 24,157 matches in 00:00:00.1483269

Note that disabling backtracking is not always an option, since it changes the behavior of the regular expression engine. In the following example, the regular expression pattern (a+)ab matches one or more “a” characters, followed by the substring “ab”. This pattern matches the input string “aaaaaaaab”. However, if we change the regular expression pattern to (?>a+)ab to disable backtracking, the input string no longer matches. This is because when backtracking disabled, the regular expression engine is greedy and will not give up the last matched “a” in the string so that the match with “ab” succeeds.

[Visual Basic]
Imports System.Text.RegularExpressions

Module Example
   Public Sub Main()
      Dim input As String = "aaaaaaaab"
      
      Dim backtrackingPattern As String = "(a+)ab"
      Console.WriteLine(Regex.IsMatch(input, backtrackingPattern))
      ' Displays True.
      
      Dim nonBacktrackingPattern As String = "(?>a+)ab"
      Console.WriteLine(Regex.IsMatch(input, nonBacktrackingPattern))
      ' Displays False.          
   End Sub
End Module

[C#]
using System;
using System.Text.RegularExpressions;
 
public class Example
{
   public static void Main()
   {
      string input = "aaaaaaaab";
      
      string backtrackingPattern = "(a+)ab";
      Console.WriteLine(Regex.IsMatch(input, backtrackingPattern));
      // Displays True.
      
      string nonBacktrackingPattern = "(?>a+)ab";
      Console.WriteLine(Regex.IsMatch(input, nonBacktrackingPattern));
      // Displays False.          
   }
}

Nested Optional Quantifiers and Backtracking

When developing regular expressions, developers typically focus on creating patterns that will match particular input. They rarely focus on creating patterns that will efficiently handle input that does not match the pattern. Frequently, this is justified. For example, if a regular expression is designed to extract substrings from text that follows a known format, there is no need to focus on non-matches. However, the regular expression engine can exhibit some of its poorest performance when handling text that does not conform to a specific regular expression pattern. In particular, this occurs when a regular expression that uses nested optional quantifiers (and therefore relies on extensive backtracking) is used to process text that does not conform to the pattern.

For example, in an order processing system, let’s say a regular expression is used to ensure that an item identifier conforms to a standard pattern. All item identifiers must contain at least one alphanumeric character; if they have multiple characters, the second to last character must also be alphanumeric; and the last character must be a dollar sign (“$”). Any intermediate characters must be word characters (that is, alphanumeric characters or underscores), periods, or hyphens. The regular expression pattern that was developed to reflect the format of the item identifier is ^[0-9A-Z]([-.\w]*[0-9A-Z])*\$$. A new employee who is being trained to do data entry enters one item identifier correctly, but then enters # instead of $ as the ending symbol for the next item. The following code illustrates this regular expression pattern and the data entry input. As the output shows, the regular expression performs efficiently when it is able to match the input identifier, but performance degrades drastically if it encounters an identifier that does not match the pattern.

[Visual Basic]
Imports System.Diagnostics
Imports System.Text.RegularExpressions

Module Example
   Public Sub Main()
      Dim pattern As String = "^[0-9A-Z]([-.\w]*[0-9A-Z])*\$$"
      Dim rgx As New Regex(pattern, RegexOptions.IgnoreCase)
      Dim sw As Stopwatch

      Dim items() As String = { "A163.1523C$", 
                                "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa#" }
      Dim toValidate, result As String
      
      For Each item In items
         For ctr As Integer = 1 To item.Length
            toValidate = item.Substring(0, ctr)     
            sw = Stopwatch.StartNew() 
   
            If rgx.IsMatch(toValidate) Then
               result = "Match"
            Else
               result = "No match"
            End If
            sw.Stop()
            Console.WriteLine("{0} with {1} characters in {2}", result, ctr, 
                              sw.Elapsed)   
         Next
         Console.WriteLine()               
      Next
   End Sub
End Module

[C#]
using System;
using System.Diagnostics;
using System.Text.RegularExpressions;

public class Example
{
   public static void Main()
   {
      string pattern = @"^[0-9A-Z]([-.\w]*[0-9A-Z])*\$$";
      Regex rgx = new Regex(pattern, RegexOptions.IgnoreCase);
      Stopwatch sw;

      string[] items= { "A163.1523C$", 
                        "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa#" };
      string toValidate, result;
      
      foreach (var item in items) {
         for (int ctr = 1; ctr <= item.Length; ctr++) {
            toValidate = item.Substring(0, ctr);     
            sw = Stopwatch.StartNew(); 
   
            if (rgx.IsMatch(toValidate))
               result = "Match";
            else
               result = "No match";

            sw.Stop();
            Console.WriteLine("{0} with {1} characters in {2}", result, ctr, 
                              sw.Elapsed);   
         }
         Console.WriteLine();               
      }
   }
}

The example displays the following output:

No match with 1 characters in 00:00:00.0000252
No match with 2 characters in 00:00:00.0000132
No match with 3 characters in 00:00:00.0000048
No match with 4 characters in 00:00:00.0000061
No match with 5 characters in 00:00:00.0000071
No match with 6 characters in 00:00:00.0000259
No match with 7 characters in 00:00:00.0000175
No match with 8 characters in 00:00:00.0000307
No match with 9 characters in 00:00:00.0000573
No match with 10 characters in 00:00:00.0001128
Match with 11 characters in 00:00:00.0000035

No match with 1 characters in 00:00:00.0000016
No match with 2 characters in 00:00:00.0000022
No match with 3 characters in 00:00:00.0000029
No match with 4 characters in 00:00:00.0000045
No match with 5 characters in 00:00:00.0000081
No match with 6 characters in 00:00:00.0000145
No match with 7 characters in 00:00:00.0000265
No match with 8 characters in 00:00:00.0000521
No match with 9 characters in 00:00:00.0001017
No match with 10 characters in 00:00:00.0002032
No match with 11 characters in 00:00:00.0004016
No match with 12 characters in 00:00:00.0007994
No match with 13 characters in 00:00:00.0015952
No match with 14 characters in 00:00:00.0032086
No match with 15 characters in 00:00:00.0065094
No match with 16 characters in 00:00:00.0129808
No match with 17 characters in 00:00:00.0270169
No match with 18 characters in 00:00:00.0553021
No match with 19 characters in 00:00:00.1070363
No match with 20 characters in 00:00:00.2208385
No match with 21 characters in 00:00:00.4289808
No match with 22 characters in 00:00:00.8553915
No match with 23 characters in 00:00:01.6993887
No match with 24 characters in 00:00:03.3938261
No match with 25 characters in 00:00:06.8359716
No match with 26 characters in 00:00:13.5125933
No match with 27 characters in 00:00:27.2257724
No match with 28 characters in 00:00:53.8205646
No match with 29 characters in 00:01:49.2090587
No match with 30 characters in 00:03:35.8028235
No match with 31 characters in 00:07:11.6145706
No match with 32 characters in 00:10:46.0044969

This example evaluates the performance of the regular expression engine based on the number of characters passed to it in the input string. As the following chart shows, for an input string that does not match the regular expression pattern, execution time approximately doubles each time another character is added to the string. It takes slightly over 10 minutes for the regular expression engine to recognize that an input string consisting of 32 characters does not match the regular expression pattern. Expanding the input to 35 characters would take approximately 1 hour and 20 minutes. Processing a string with 38 characters would take over 10 hours. Processing a string with 40 characters would take nearly two days. Visually scanning the string would be significantly faster.

This atrociously poor performance is tied to the behavior of the NFA regular expression engine in the .NET Framework. When the regular expression engine cannot find a match and the pattern requires backtracking, the engine must exhaust all possible paths through the data before it can conclude that a match has failed; so an unsuccessful match always represents a worst-case scenario. (For more information on the NFA engine and backtracking, see Details of Regular Expression Behavior and Backtracking in the MSDN Library.) Because of the nested quantifiers in our example, the number of possible matches that the regular expression engine has to examine before concluding that a match has failed increases exponentially with each character after the first character in the input string.

Not all nested quantifiers impact performance in this way. In our example, the real problem in the subexpression ([-.\w]*[0-9A-Z])* is that the last character to be matched (any numeric character from 0 to 9, or any alphanumeric character from A to Z) is a subset of the previous subexpression (any word character – that is, any numeric or alphanumeric character or an underscore -- along with a hyphen and a period). A partially successful match makes it difficult for the regular expression engine to determine whether a particular character belongs to the first subexpression, the ending character of the expression, or to a repetition of the first subexpression or the ending character. The inability to successfully match the entire regular expression results in excessive backtracking, so that the number of possible paths through the input string increases exponentially with each character that can be matched. (For a description of the additional comparisons performed by the regular expression engine when backtracking, see the “Backtracking with Nested Optional Quantifiers” section in the Backtracking topic in the MSDN Library.)

In cases where excessive backtracking causes performance problems, other language constructs can often be used in place of the nested quantifiers. In many cases, nested quantifiers can be replaced by a single quantifier and zero-width lookbehind or lookahead assertion. Zero-width assertions do not advance the position counter in the input string. They either look ahead of or immediately behind the current position to determine whether the pattern defined by the assertion is true. If it is not, the match fails. (For information on positive lookahead and positive lookbehind, see Grouping Constructs in the MSDN documentation.) For example, we can replace our previous regular expression to validate an item identifier with ^[0-9A-Z][-.\w]*(?<=[0-9A-Z])\$, which uses positive lookbehind when the $ symbol is matched to ensure that the character before $ is either numeric or alphanumeric. This regular expression is interpreted as follows:

Pattern	Description
^	Begin the match at the beginning of the string.
[0-9A-Z]	Match any numeric (0-9) or alphanumeric (A-Z) character.The comparison is not case-sensitive.
[-.\w]*	Match zero or more word, hyphen, or period characters.
\$	Match the $ character.
(?<=[0-9A-Z])	Determine whether the character before the current character is numeric (0-9) or alphanumeric (A-Z).The comparison is not case sensitive. This is a zero-width assertion; it does not cause the position counter in the input string to advance.

This produces the following output:

No match with 1 characters in 00:00:00.0498953
No match with 2 characters in 00:00:00.0002162
No match with 3 characters in 00:00:00.0000237
No match with 4 characters in 00:00:00.0000333
No match with 5 characters in 00:00:00.0000596
Match with 6 characters in 00:00:00.0000500
Match with 7 characters in 00:00:00.0000256
Match with 8 characters in 00:00:00.0000237
Match with 9 characters in 00:00:00.0000198
Match with 10 characters in 00:00:00.0000198
Match with 11 characters in 00:00:00.0000186
Match with 12 characters in 00:00:00.0000192
Match with 13 characters in 00:00:00.0000494
Match with 14 characters in 00:00:00.0000449

No match with 1 characters in 00:00:00.0000288
No match with 2 characters in 00:00:00.0000205
No match with 3 characters in 00:00:00.0000205
No match with 4 characters in 00:00:00.0000301
No match with 5 characters in 00:00:00.0000481
No match with 6 characters in 00:00:00.0000269
No match with 7 characters in 00:00:00.0000218
No match with 8 characters in 00:00:00.0000346
No match with 9 characters in 00:00:00.0000468
No match with 10 characters in 00:00:00.0000442
No match with 11 characters in 00:00:00.0000346
No match with 12 characters in 00:00:00.0000891
No match with 13 characters in 00:00:00.0000192
No match with 14 characters in 00:00:00.0000372
No match with 15 characters in 00:00:00.0000218
No match with 16 characters in 00:00:00.0000192
No match with 17 characters in 00:00:00.0000243
No match with 18 characters in 00:00:00.0000224
No match with 19 characters in 00:00:00.0000205
No match with 20 characters in 00:00:00.0000224
No match with 21 characters in 00:00:00.0000218
No match with 22 characters in 00:00:00.0000224
No match with 23 characters in 00:00:00.0000263
No match with 24 characters in 00:00:00.0000218
No match with 25 characters in 00:00:00.0000243
No match with 26 characters in 00:00:00.0000218
No match with 27 characters in 00:00:00.0000224
No match with 28 characters in 00:00:00.0000224
No match with 29 characters in 00:00:00.0000230
No match with 30 characters in 00:00:00.0000237
No match with 31 characters in 00:00:00.0000243
No match with 32 characters in 00:00:00.0000243

In other cases, some combination of alternation (the construct “|”) along with lookahead and lookbehind can be used to replace a regular expression that is subject to excessive backtracking. For example, regular expressions that are designed to match numbers are frequently troublesome if the numeric value can be either integral or floating point. A common pattern used to match such numeric values is ^\d+(|.\d+)*$, as shown in the following example.

[Visual Basic]
Imports System.Diagnostics
Imports System.Text.RegularExpressions

Module Example
   Public Sub Main()   
      Dim number As String = "111111111111111111111111111111!"
      Dim pattern As String = "^\d+(|.\d+)*$"
      
      Dim rgx As New Regex(pattern, RegexOptions.IgnoreCase)
      Dim sw As Stopwatch
      Dim str As String 
  
      For ctr As Integer = 1 To number.Length
         str = number.Substring(0, ctr)
         sw = Stopwatch.StartNew()
         If rgx.IsMatch(str) Then 
            Console.Write("Match with {0} characters ", str.Length)
         Else
            Console.Write("No match with {0} characters ", str.Length)
         End If  
         sw.Stop()
         Console.WriteLine("in {0}.", sw.Elapsed)
      Next
   End Sub
End Module

[C#] 
using System;
using System.Diagnostics;
using System.Text.RegularExpressions;

public class Example
{
   public static void Main()
   {
      string number = "111111111111111111111111111111!";
      string pattern = @"^\d+(|.\d+)*$";
      
      Regex rgx = new Regex(pattern, 
                  RegexOptions.IgnoreCase);
      Stopwatch sw;
      string str; 
  
      for (int ctr = 1; ctr <= number.Length; ctr++) {
         str = number.Substring(0, ctr);
         sw = Stopwatch.StartNew();
         if (rgx.IsMatch(str)) 
            Console.Write("Match with {0} characters ", str.Length);
         else
            Console.Write("No match with {0} characters ", str.Length);

         sw.Stop();
         Console.WriteLine("in {0}.", sw.Elapsed);
      }
   }
}

However, as the output from the example shows, this pattern is subject to excessive backtracking.

Match with 1 characters in 00:00:00.0011311.
Match with 2 characters in 00:00:00.0010926.
Match with 3 characters in 00:00:00.0011003.
Match with 4 characters in 00:00:00.0014570.
Match with 5 characters in 00:00:00.0002797.
Match with 6 characters in 00:00:00.0001565.
Match with 7 characters in 00:00:00.0012619.
Match with 8 characters in 00:00:00.0001456.
Match with 9 characters in 00:00:00.0011439.
Match with 10 characters in 00:00:00.0010682.
Match with 11 characters in 00:00:00.0011125.
Match with 12 characters in 00:00:00.0002053.
Match with 13 characters in 00:00:00.0001854.
Match with 14 characters in 00:00:00.0012485.
Match with 15 characters in 00:00:00.0010675.
Match with 16 characters in 00:00:00.0011137.
Match with 17 characters in 00:00:00.0012299.
Match with 18 characters in 00:00:00.0001937.
Match with 19 characters in 00:00:00.0012709.
Match with 20 characters in 00:00:00.0011323.
Match with 21 characters in 00:00:00.0010996.
Match with 22 characters in 00:00:00.0010188.
Match with 23 characters in 00:00:00.0011125.
Match with 24 characters in 00:00:00.0001770.
Match with 25 characters in 00:00:00.0001469.
Match with 26 characters in 00:00:00.0001687.
Match with 27 characters in 00:00:00.0004356.
Match with 28 characters in 00:00:00.0013370.
Match with 29 characters in 00:00:00.0010958.
Match with 30 characters in 00:00:00.0010983.
No match with 31 characters in 00:00:03.0872118.

The root of the problem is that the subexpression (|.\d+)* is intended to match either a group separator or a decimal separator, but it also matches any other character. The subexpression also includes a nested quantifier. In the event of a near-match with numeric digits, the regular expression engine will have difficulty apportioning numeric digits between the first occurrence of the \d* pattern or its subsequent possible repetitions.

A regular expression that eliminates this unnecessary backtracking and also addresses a number of limitations of the original regular expression is ^\d*(,\d{3})*(?((?=\.))\.\d+|(?<=\d))$. This regular expression uses the conditional construct with an expression (for more information, see Alternation Constructs in the MSDN Library. Note that this regular expression works with numeric values formatted using the conventions of the English (United States) culture. A regular expression that uses the group and decimal separators of the current culture could be built dynamically by retrieving these values from the current culture’s DateTimeFormatInfo object. The regular expression is interpreted as shown in the following table.

Pattern	Description
^	Begin the match at the beginning of the string.
\d*	Match zero or more decimal digits. This matches the integral portion of the numeric value.
(,\d{3})*	Match zero or more occurrences of a group separator followed by three decimal digits.
(?((?=\.))	Use positive lookahead to determine whether the next character is a decimal separator. (This is the condition clause of an alternation pattern.)
\.\d+	If the next character is a decimal separator, match the decimal separator followed by one or more decimal digits.
\|(?<=\d))	If the next character is not a decimal separator, look behind to determine whether the previous character is a decimal digit. This ensures that the number contains at least one integral digit.
$	Match the end of the input string.

If this regular expression pattern replaces the pattern in the previous example, the example produces the following output:

Match with 1 characters in 00:00:00.0012331.
Match with 2 characters in 00:00:00.0010586.
Match with 3 characters in 00:00:00.0010509.
Match with 4 characters in 00:00:00.0011676.
Match with 5 characters in 00:00:00.0002303.
Match with 6 characters in 00:00:00.0001892.
Match with 7 characters in 00:00:00.0005203.
Match with 8 characters in 00:00:00.0014442.
Match with 9 characters in 00:00:00.0010740.
Match with 10 characters in 00:00:00.0010573.
Match with 11 characters in 00:00:00.0002046.
Match with 12 characters in 00:00:00.0002309.
Match with 13 characters in 00:00:00.0012318.
Match with 14 characters in 00:00:00.0001411.
Match with 15 characters in 00:00:00.0010900.
Match with 16 characters in 00:00:00.0010785.
Match with 17 characters in 00:00:00.0002008.
Match with 18 characters in 00:00:00.0002149.
Match with 19 characters in 00:00:00.0012850.
Match with 20 characters in 00:00:00.0011137.
No match with 21 characters in 00:00:00.0010887.

Best Practices for Controlling Backtracking

To maximize the performance of regular expressions in the .NET Framework, we recommend that you do the following to limit or eliminate unnecessary backtracking:

In alternation constructs, order alternatives based on their approximate frequency of occurrence in the input.
Take the time to optimize alternation patterns.
Determine whether backtracking is required for the regular expression to successfully match a subexpression with a portion of the input string. If it is not necessary, disable backtracking with the (?>subexpression) language element.
If subexpressions are intended to validate input, replace nested quantifiers such as (\w*[A-Z])* with other language elements. The most useful language elements for this purpose are the positive lookahead assertion and the positive lookbehind assertion, the alternation construct, and the conditional construct.

More on Zip in .NET [Richard Lee]

BCL Team — Thu, 29 Jul 2010 23:21:29 GMT

First, I’d like to thank everybody for their comments on the Zip APIs. It’s great to know that I’m working on something that a lot of people will hopefully find useful. I’ll try to address the themes that came up in the comments.

Streams

A lot of the comments mentioned support for streams. The API does support creating a ZipArchive with a stream as the backing store, and input from streams. The following code sample takes the contents of instream and writes a Zip archive to outstream containing just that one file:

using (ZipArchive archive = new ZipArchive(outstream, ZipArchiveMode.Create))
{
	ZipArchiveEntry entry = archive.CreateEntry("data.dat");
	using (Stream entryStream = entry.Open())
	{
		instream.CopyTo(entryStream);
	}
}

This will write out the Zip archive directly to outstream, without buffering the entire contents of the archive in memory or writing to a temporary file. We really think of the methods used above as the core APIs. Methods like CreateEntryFromFile and ExtractToDirectory are purely convenience methods – their main purpose is to make some of the more common scenarios with files easier.

Compression and Encryption

Another common theme was custom encryption and compression algorithms. The vast majority of Zip archives that are meant to be interoperable with the widest range of libraries, tools, and applications use the Deflate compression algorithm without encryption. Our main goal for this API is to be able to read and write such archives. As such, we’re currently planning to support writing Zip archives with Deflate, reading Zip archives that use Deflate or no compression, and to not support encryption. Not only will this enable reading/writing interoperable Zip archives, it also scopes the work to something reasonable that can be delivered during my internship.

If we’re not providing built-in support for additional compression or encryption algorithms, an obvious question, then, is why not provide extensibility hooks so that custom compression or encryption algorithms can be plugged-in? We explored doing this and it turns out it’d be more complex than you might initially think.

A lot of people mentioned the CryptoStream and ICryptoTransform model as a powerful way to allow for extensibility. This works well because the ICryptoTransform only needs to do one thing – transform a series of bytes into another series of bytes. Unfortunately, encrypting Zip files is a much more complex operation. Fields and flags need to be set to appropriate values and headers need to be encrypted depending on which algorithm is used. Implementing either of the two secure encryption methods mentioned in the Zip specification would require access to essentially all of the metadata in the Zip file. The resulting interface for such an extension would be enormously complex.

Compression is a bit simpler, as only one entry is compressed at a time. However, there are still fields and flags that need to be set in the headers, depending on the algorithm used. Furthermore, for both encryption and compression, only a few compression/encryption methods are specified in the Zip specification. We don’t want to give the impression that any compression method can be used to produce a Zip file, when that isn’t the case. If you want to use a compression or encryption algorithm that isn’t specified in the Zip spec, you might as well just compress or encrypt the stream yourself.

So an extensibility interface would be substantially more complex than something like ICryptoTransform, and add significant complexity to the ZipArchive/ZipArchiveEntry class. We don’t think providing extensibility hooks in this way adds enough value to the API to justify the added complexity that they bring. However, we are open to adding built-in support for encryption or other compression algorithms in the future, based on customer demand. We’ve specifically designed the API in a way that would allow us to do this. If this is something you’re interested in, we’d love to better understand your needs.

Abstract Base Class

Another common question was around providing a base class for Archives. We explored this and have decided not to add such an abstraction at this time. The main reason we are holding off is because right now we’re only planning to provide support for Zip archives and have no plans to support other archive formats (such as CAB, RAR, etc.). As a design principle, we try to avoid adding abstractions when there is only one implementation, otherwise we risk getting the abstraction wrong. Then we’re either stuck with the bad abstraction or are forced to add another when adding another implementation. All of this adds additional complexity that we’d like to avoid.

The concerns about test-driven development are certainly valid, but there are ways around this. For example, because archives can be made with streams as the backing store, a FileStream could be mocked and put behind an archive. That may not be totally ideal, but these concerns don’t seem compelling enough to justify adding an abstract base class at this time.

Miscellaneous

There were a couple comments about treating the Zip archive like the filesystem, and supporting searching for certain kinds of files. We made the decision to treat the archive as a flat container of files because that is how they are actually stored. As a library rather than an application, we thought it made more sense to represent the archive as it exists on disk. Also, using LINQ on the entries means getting all of the files in a certain subdirectory that end in .txt is a relatively simple operation.

Another interesting comment was that MoveTo was confusing. MoveTo was intended to act like Rename, for renaming entries but keeping everything else about them the same. The method was named MoveTo to mimic the naming for the method on FileInfo. However, because it is confusing and probably has very few compelling usage scenarios, we’re thinking of cutting it from the API.

I hope I addressed some of your comments. We’d love to keep hearing from you, especially if you have extremely compelling use cases for some of the features that we’re not including in this version of the API. Being the designer of the API, I think it would be really cool, too, to support custom compression and encryption, or some of these other features. But at the risk of sounding like a broken record, I’m trying to build a simple, usable API, and these decisions are made with that in mind.

Working with Zip Files in .NET [Richard Lee]

BCL Team — Mon, 28 Jun 2010 21:37:00 GMT

Before getting started, I’ll introduce myself. My name is Richard Lee, and I’m a developer intern on the BCL for the summer. I’ve only been here for a few weeks, but it’s been great working here. The people, the environment, and my project are all great. Speaking of which, my project is to add general purpose .NET APIs for reading and writing Zip files, which we’re considering adding to the next version of the .NET Framework.

The most common Zip tasks are extracting to a directory and archiving a directory. For these mainline scenarios, we have static convenience methods. The following code takes all of the files in the Zip file, photos.zip, and extracts them to a folder on the file system:

ZipArchive.ExtractToDirectory("photos.zip", @"photos\summer2010");

This code does the reverse, putting all of the files in the folder into the Zip file:

ZipArchive.CreateFromDirectory(@"docs\attach", "attachment.zip");

For more sophisticated manipulations of Zip archives, there are two main classes. ZipArchive represents a zip archive, which is a collection of entries, and ZipArchiveEntry represents an archived file entry. The following code extracts only text files from the given archive.

using (var archive = new ZipArchive("data.zip"))
{
    foreach (var entry in archive.Entries)
    {
        if (entry.FullName.EndsWith(".txt", StringComparison.OrdinalIgnoreCase))
        {
            entry.ExtractToFile(Path.Combine(directory, entry.FullName));
        }
    }
}

Zip archives can also be created on-the-fly. This example creates a new archive with a readme file that is created without the need for a corresponding file on disk, and a file from the file system.

using (var archive = new ZipArchive("new.zip", ZipArchiveMode.Create))
{
    var readmeEntry = archive.CreateEntry("Readme.txt");
    using (var writer = new StreamWriter(readmeEntry.Open()))
    {
        writer.WriteLine("Included files: ");
        writer.WriteLine("data.dat");
    }

    archive.CreateEntryFromFile("data.dat", "data.dat");
}

The ZipArchive class supports three modes:

In Read mode, data is read from the file on demand, using only a small buffer.
In Create mode, data is written directly to the file using only a small buffer. Only one entry may be held open for writing at a time.
In Update mode it is possible to read and write from existing archives, as well as rename or delete entries. This mode requires loading the entire archive into memory, and as such we recommend that it be used only with small archives when this functionality is needed.

Below is our current thinking on what the public API listing will look like (note that this hasn’t been finalized yet).

namespace System.IO.Compression
{
    public enum ZipArchiveMode { Read, Create, Update }

    public class ZipArchive : IDisposable {
        // Constructors
        public ZipArchive(String path);
        public ZipArchive(String path, ZipArchiveMode mode); 
        public ZipArchive(Stream stream);
        public ZipArchive(Stream stream, ZipArchiveMode mode);
        public ZipArchive(Stream stream, ZipArchiveMode mode, Boolean leaveOpen);

        // Properties
        public ReadOnlyCollection<ZipArchiveEntry> Entries { get; }
        public ZipArchiveMode Mode { get; }
        
        // Instance methods
        public ZipArchiveEntry GetEntry(String entryName);
        public ZipArchiveEntry CreateEntry(String entryName);

        public void Dispose();
        protected virtual void Dispose(Boolean disposing);

        public override String ToString();

        // Instance convenience methods
        public ZipArchiveEntry CreateEntryFromFile(String sourceFileName, String entryName);

        public void ExtractToDirectory(String destinationDirectoryName);

        // Static convenience methods
        public static void CreateFromDirectory(String sourceDirectoryName, String destinationArchive);
        public static void CreateFromDirectory(String sourceDirectoryName, String destinationArchive, Boolean includeBaseDirectory);

        public static void ExtractToDirectory(String sourceArchive, String destinationDirectoryName);
    }

    public class ZipArchiveEntry {
        // Properties
        public DateTimeOffset LastWriteTime { get; set; }
        public String FullName { get; }
        public String Name { get; }
        public Int64 Length { get; }
        public Int64 CompressedLength { get; }
        public ZipArchive Archive { get; }

        // Methods
        public Stream Open();
        public void Delete();
        public void MoveTo(String destinationEntryName);

        // Convenience methods
        public void ExtractToFile(String destinationFileName);
        public void ExtractToFile(String destinationFileName, Boolean overwrite);

        public override String ToString();
    }
}

We would love to hear what you think of the APIs so far, and how you plan on using them.

Optimizing Regular Expression Performance, Part I: Working with the Regex Class and Regex Objects [Ron Petrusha]

BCL Team — Fri, 25 Jun 2010 08:29:00 GMT

The .NET Framework’s regular expression implementation is a traditional Nondeterministic Finite Automaton (NFA) engine. Perhaps the most significant feature of NFA engines is that they place the responsibility for crafting efficient, high-performance regular expressions on the developer. (For more information about the .NET Framework’s implementation of an NFA engine, see Details of Regular Expression Behavior in the MSDN Library.) With their support for such features as backreferences and backtracking, NFA engines often offer a trade-off between flexibility and power on the one hand, and speed and performance on the other. Many regular expression developers complain about the performance of the regular expression engine but are often unable to diagnose why an individual regular expression performs poorly. In this three-part article, we’ll discuss some of the techniques that regular expression developers can apply to maximize the performance of regular expressions while preserving their power.

At the heart of the .NET Framework’s regular expression object model is the Regex object, which represents the regular expression engine. Often, the single greatest factor that affects regular expression performance is the way in which the Regex engine is used. Defining a regular expression involves tightly coupling the regular expression engine with a regular expression pattern. That coupling process, whether it involves instantiating a Regex object by passing its constructor a regular expression pattern or calling a static method by passing it the regular expression pattern along with the string to be analyzed, is by necessity an expensive one. If the regular expression engine will be used in instance method calls, an instance of the regular expression engine must be created. All regular expressions, whether used by Regex instances or in static Regex method calls, must then be compiled (more on this later). This is analogous to recompiling an application each time that it is run.

Note: To measure regular expression performance, we make extensive use of the Stopwatch class. These measurements are used for illustrative purposes only; they reflect performance when a particular regular expression is used to parse a particular input string on a particular system. They are not intended to be construed as objective benchmarks.

Taking Advantage of the Regular Expression Cache

Since the process of coupling the regular expression engine with a particular pattern is expensive, you can improve performance by ensuring that you perform this coupling as few times as possible. The following example illustrates a fairly common scenario that offers poor performance. An IsValidEmail method is called whenever an application needs to validate an email address. The method instantiates a Regex object and calls its IsMatch method to process the email address. This means that a Regex object is instantiated with the same regular expression pattern with each method call.

Note: The practice of repeatedly instantiating a Regex object with the same regular expression, as the performance measurements presented later in this section show, represents a worst practice that significantly degrades performance. Do not replicate this practice in your own code.

[Visual Basic]
Imports System.Diagnostics
Imports System.Text.RegularExpressions

Module Example
  Public Sub Main()
    Dim inputs() As String = { "david.jones@proseware.com", _
                               "d.j@server1.proseware.com", _
                               "jones@ms1.proseware.com", _
                               "j.@server1.proseware.com", _
                               "j@proseware.com9", _
                               "js#internal@proseware.com", _
                               "j_9@[129.126.118.1]", _
                               "j..s@proseware.com", _
                               "js*@proseware.com", _
                               "js@proseware..com", _
                               "js@proseware.com9", _
                               "j.s@server1.proseware.com", _
                               """a*****""""@cohowinery.com", _
                         webmaster@aaaabbbbbbbbccccccccddddd!" }
    Dim sw As Stopwatch = Stopwatch.StartNew()
    For Each input As String In inputs
      If IsValidEmail(input) Then
        ' Handle valid data.
        Console.WriteLine("{0} is a valid email address.", _
                          input)
      Else
        ' Handle invalid data.
        Console.WriteLine("{0} is not a valid email address.", _
                          input)
      End If
    Next 
    sw.Stop()
    Console.WriteLine("Elapsed time: {0}", sw.Elapsed)
  End Sub
   
  Private Function IsValidEmail(input As String) As Boolean
    Dim pattern As String = "^(?("")("".+?""@)|(([0-9a-zA-Z]((\.(?!\.))|[-!#\$%&'\*\+/=\?\^`\{\}\|~\w])*)(?<=[0-9a-zA-Z])@))" + _
                             "(?(\[)(\[(\d{1,3}\.){3}\d{1,3}\])|(([0-9a-zA-Z][-\w]*[0-9a-zA-Z]\.)+[a-zA-Z]{2,6}))$"
      Dim rgx As New Regex(pattern)    ' BAD: Never reinstantiate the same object
      Return rgx.IsMatch(input)
   End Function
End Module

[C#]
using System;
using System.Diagnostics;
using System.Text.RegularExpressions;

public class Example
{
   public static void Main()
   {
      Example ex = new Example();
      string[] inputs = { "david.jones@proseware.com", 
                          "d.j@server1.proseware.com", 
                          "jones@ms1.proseware.com", 
                          "j.@server1.proseware.com", 
                          "j@proseware.com9", 
                          "js#internal@proseware.com", 
                          "j_9@[129.126.118.1]", 
                          "j..s@proseware.com", 
                          "js*@proseware.com", 
                          "js@proseware..com", 
                          "js@proseware.com9", 
                          "j.s@server1.proseware.com", 
                          @"""a*****""@cohowinery.com", 
                          "webmaster@aaaabbbbbbbbccccccccddddd!" };
      Stopwatch sw = Stopwatch.StartNew();
      foreach (string input in inputs) {
         if (ex.IsValidEmail(input))
            // Handle valid email data.
            Console.WriteLine("{0} is a valid email address.", input);
         else
            // Handle invalid data.
            Console.WriteLine("{0} is not a valid email address.",
                              input);
      } 
      sw.Stop();
      Console.WriteLine("Elapsed time: {0}", sw.Elapsed);
   }

   private bool IsValidEmail(string input)
   {
      string pattern = @"^(?("")("".+?""@)|(([0-9a-zA-Z]((\.(?!\.))|[-!#\$%&'\*\+/=\?\^`\{\}\|~\w])*)(?<=[0-9a-zA-Z])@))" + 
                       @"(?(\[)(\[(\d{1,3}\.){3}\d{1,3}\])|(([0-9a-zA-Z][-\w]*[0-9a-zA-Z]\.)+[a-zA-Z]{2,6}))$";
      Regex rgx = new Regex(pattern);   // BAD: Never reinstantiate the same object

      return rgx.IsMatch(input);   
   }
}

This example displays the following output:

david.jones@proseware.com is a valid email address.
d.j@server1.proseware.com is a valid email address.
jones@ms1.proseware.com is a valid email address.
j.@server1.proseware.com is not a valid email address.
j@proseware.com9 is not a valid email address.
js#internal@proseware.com is a valid email address.
j_9@[129.126.118.1] is a valid email address.
j..s@proseware.com is not a valid email address.
js*@proseware.com is not a valid email address.
js@proseware..com is not a valid email address.
js@proseware.com9 is not a valid email address.
j.s@server1.proseware.com is a valid email address.
"a*****""@cohowinery.com is a valid email address.
webmaster@aaaabbbbbbbbccccccccddddd! is not a valid email address.
Elapsed time: 00:00:00.4997749

Performance suffers here because each method call requires a Regex object to be instantiated with the same regular expression pattern. But instead of instantiating a Regex object in each method call, we can call the static Regex.IsMatch method, as the following example does.

[Visual Basic]
Imports System.Diagnostics
Imports System.Text.RegularExpressions

Module Example
  Public Sub Main()
    Dim inputs() As String = { "david.jones@proseware.com", _
                               "d.j@server1.proseware.com", _
                               "jones@ms1.proseware.com", _
                               "j.@server1.proseware.com", _
                               "j@proseware.com9", _
                               "js#internal@proseware.com", _
                               "j_9@[129.126.118.1]", _
                               "j..s@proseware.com", _
                               "js*@proseware.com", _
                               "js@proseware..com", _
                               "js@proseware.com9", _
                               "j.s@server1.proseware.com", _
                               """a*****""""@cohowinery.com", _
                         webmaster@aaaabbbbbbbbccccccccddddd!" }
    Dim sw As Stopwatch = Stopwatch.StartNew()
    For Each input As String In inputs
      If IsValidEmail(input) Then
        ' Handle valid data.
        Console.WriteLine("{0} is a valid email address.", _
                          input)
      Else
        ' Handle invalid data.
        Console.WriteLine("{0} is not a valid email address.", _
                          input)
      End If
    Next 
    sw.Stop()
    Console.WriteLine("Elapsed time: {0}", sw.Elapsed)
  End Sub
   
  Private Function IsValidEmail(input As String) As Boolean
    Dim pattern As String = "^(?("")("".+?""@)|(([0-9a-zA-Z]((\.(?!\.))|[-!#\$%&'\*\+/=\?\^`\{\}\|~\w])*)(?<=[0-9a-zA-Z])@))" + _
                             "(?(\[)(\[(\d{1,3}\.){3}\d{1,3}\])|(([0-9a-zA-Z][-\w]*[0-9a-zA-Z]\.)+[a-zA-Z]{2,6}))$"
      Return Regex.IsMatch(input, pattern)  ' GOOD: Take advantage of regex cache
   End Function
End Module

[C#]
using System;
using System.Diagnostics;
using System.Text.RegularExpressions;

public class Example
{
   public static void Main()
   {
      Example ex = new Example();
      string[] inputs = { "david.jones@proseware.com", 
                          "d.j@server1.proseware.com", 
                          "jones@ms1.proseware.com", 
                          "j.@server1.proseware.com", 
                          "j@proseware.com9", 
                          "js#internal@proseware.com", 
                          "j_9@[129.126.118.1]", 
                          "j..s@proseware.com", 
                          "js*@proseware.com", 
                          "js@proseware..com", 
                          "js@proseware.com9", 
                          "j.s@server1.proseware.com", 
                          @"""a*****""@cohowinery.com", 
                          "webmaster@aaaabbbbbbbbccccccccddddd!" };
      Stopwatch sw = Stopwatch.StartNew();
      foreach (string input in inputs) {
         if (ex.IsValidEmail(input))
            // Handle valid email data.
            Console.WriteLine("{0} is a valid email address.", input);
         else
            // Handle invalid data.
            Console.WriteLine("{0} is not a valid email address.",
                              input);
      } 
      sw.Stop();
      Console.WriteLine("Elapsed time: {0}", sw.Elapsed);
   }

   private bool IsValidEmail(string input)
   {
      string pattern = @"^(?("")("".+?""@)|(([0-9a-zA-Z]((\.(?!\.))|[-!#\$%&'\*\+/=\?\^`\{\}\|~\w])*)(?<=[0-9a-zA-Z])@))" + 
                       @"(?(\[)(\[(\d{1,3}\.){3}\d{1,3}\])|(([0-9a-zA-Z][-\w]*[0-9a-zA-Z]\.)+[a-zA-Z]{2,6}))$";
      return Regex.IsMatch(input, pattern);  // GOOD: Take advantage of regex cache
   }
}

The following output is displayed when the call to the instance Regex.IsMatch method is replaced by the equivalent call to the static Regex.IsMatch method:

david.jones@proseware.com is a valid email address.
d.j@server1.proseware.com is a valid email address.
jones@ms1.proseware.com is a valid email address.
j.@server1.proseware.com is not a valid email address.
j@proseware.com9 is not a valid email address.
js#internal@proseware.com is a valid email address.
j_9@[129.126.118.1] is a valid email address.
j..s@proseware.com is not a valid email address.
js*@proseware.com is not a valid email address.
js@proseware..com is not a valid email address.
js@proseware.com9 is not a valid email address.
j.s@server1.proseware.com is a valid email address.
"a*****""@cohowinery.com is a valid email address.
webmaster@aaaabbbbbbbbccccccccddddd! is not a valid email address.
Elapsed time: 00:00:00.0380847

The following chart illustrates the difference in performance between the first example, which relies on repeatedly instantiating a Regex object with the same regular expression pattern to call an instance matching method, and the second, which calls a static matching method.

The execution time of the first example is about 15 times the execution time of the second example. The difference in performance is due to the caching of regular expressions used in static method calls. Whereas the first example instantiates a regular expression object and converts the regular expression into opcodes in each of fourteen method calls (one for each element in the string array), the second example performs this conversion only once, on the first method call. Subsequently, it retrieves the interpreted regular expression.from the cache each time the expression is needed.

Only regular expressions used in static method calls are cached; regular expressions used in instance methods are not. The size of the cache is defined by the Regex.CacheSize property. By default, 15 regular expressions are cached, although this value can be modified if necessary. If the number of regular expressions exceeds the cache size, the regular expression engine discards the least recently used regular expression to cache the newest one.

Note that there is a breaking change in regular expression caching between versions 1.1 and subsequent versions of the .NET Framework. In version 1.1, both instance and static regular expressions are cached; in version 2.0 and all subsequent versions, only regular expressions used in static method calls are cached. For more information, see Regex Class Caching Changes between .NET Framework 1.1 and .NET Framework 2.0.

Options for Building Regular Expressions

So far, we’ve focused on using static regular expression methods rather than repeatedly re-instantiating the same regular expression object. You can obtain additional performance benefits by using compiled regular expressions when appropriate. (For more information, see Compilation and Reuse.)

The regular expression engine builds regular expressions in the .NET Framework in three different ways:

It interprets them. That is, when a regular expression object is instantiated, or when a static regular expression method is called and the regular expression cannot be found in the cache, the regular expression engine converts the regular expressions to opcodes. When a method is called, the opcodes are converted to MSIL and executed.by the JIT compiler. Interpreted regular expressions can be used in both static and instance method calls. Interpreted regular expressions reduce startup time at the cost of slower execution time.
It compiles them. That is, when a regular expression object is instantiated, or when a static regular expression method is called and the regular expression cannot be found in the cache, the regular expression engine converts the regular expressions to MSIL. When a method is called, the MSIL is executed by the JIT compiler. Compiled regular expressions can be used in both static and instance method calls. Compiled regular expressions increase execution speed at the cost of longer startup times.
It compiles them into a separate assembly. That is, the static Regex.CompileToAssembly method is used to compile one or more regular expression patterns as MSIL into a separate assembly, where the regular expression engine and each regular expression pattern is tightly coupled into a class that derives from Regex. When the regular expression object is instantiated, its MSIL is loaded from its assembly and executed by the JIT compiler. Regular expressions compiled to a special-purpose assembly can be used only in instance method calls. Regular expressions that are compiled to assemblies move much of a regular expression’s startup cost from runtime to design time. In a run-time environment, they maximize performance both at startup and during execution.

These three ways of using a regular expression represent three different tradeoffs between performance at startup and runtime execution. Let’s look at each of the three.

Interpreted Regular Expressions

Interpreted regular expressions sacrifice execution speed for reduced startup time, so they are best used when the regular expression is limited to a small number of method calls. This is illustrated by the following example, which compares the performance of an interpreted and a compiled regular expression when extracting a sentence from a file that contains a one-sentence string.

[Visual Basic]
Imports System.Diagnostics
Imports System.IO
Imports System.Text.RegularExpressions

Module Example
   Public Sub Main()
      Dim pattern As String = "\b(\w+((\r?\n)|,?\s))*\w+[.?:;!]"
      Dim input As String = New StreamReader(".\OneSentence.txt").ReadToEnd()
      Dim sw As Stopwatch

      Console.WriteLine("***Interpreted regular expressions***")
      ' Use Regex.Matches method to find all matches.
      sw = Stopwatch.StartNew()
      Dim instRegex As New Regex(pattern, RegexOptions.Singleline)
      Dim matches As MatchCollection = instRegex.Matches(input)
      For Each match As Match In matches
         ' Do nothing. If necessary, the MatchCollection will be populated using
         ' lazy evaluation.
      Next
      sw.Stop()
      Console.WriteLine("Found {0} matches in {1} using regex.Matches()",
                        matches.Count, sw.Elapsed)

      ' Use Regex.Match method followed by Match.NextMatch to find all matches.
      sw = Stopwatch.StartNew()
      Dim ctr As Integer = 0
      Dim instRegex2 As New Regex(pattern, RegexOptions.Multiline)
      Dim match2 As Match = instRegex2.Match(input)
      If match2.Success Then
         Do
            ctr += 1
            match2 = match2.NextMatch()
         Loop While match2.Success
      End If
      sw.Stop()
      Console.WriteLine("Found {0} matches in {1} using regex.Match()/NextMatch()",
                        ctr, sw.Elapsed)

      ' Use Regex.Match(String, Int32) to find all matches.
      sw = Stopwatch.StartNew()
      Dim pos As Integer = 0
      Dim ctr2 As Integer = 0
      Dim instRegex3 As New Regex(pattern, RegexOptions.Multiline)
      Dim match3 As Match
      Do
         match3 = instRegex3.Match(input, pos)
         If match3.Success Then
            ctr2 += 1
            pos = match3.Index + match3.Length
         End If
      Loop While match3.Success And pos <= input.Length
      sw.Stop()
      Console.WriteLine("Found {0} matches in {1} using regex.Match(String, Int32)",
                        ctr2, sw.Elapsed)

      Console.WriteLine()
      Console.WriteLine("***Compiled regular expressions***")

      ' Use compiled Regex.Matches method to find all matches.
      sw = Stopwatch.StartNew()
      Dim instRegexC As New Regex(pattern, RegexOptions.Multiline Or RegexOptions.Compiled)
      Dim matchesC As MatchCollection = instRegexC.Matches(input)
      For Each matchC As Match In matchesC
         ' Do nothing. If necessary, the MatchCollection will be populated using
         ' lazy evaluation.
      Next
      sw.Stop()
      Console.WriteLine("Found {0} matches in {1} using compiled regex.Matches()",
                        matchesC.Count, sw.Elapsed)

      ' Use compiled Regex.Match method followed by Match.NextMatch to find all matches.
      sw = Stopwatch.StartNew()
      Dim ctrC As Integer = 0
      Dim instRegex2c As New Regex(pattern, RegexOptions.Multiline Or RegexOptions.Compiled)
      Dim match2c As Match = instRegex2c.Match(input)
      If match2c.Success Then
         Do
            ctrC += 1
            match2c = match2c.NextMatch()
         Loop While match2c.Success
      End If
      sw.Stop()
      Console.WriteLine("Found {0} matches in {1} using compiled regex.Match/NextMatch",
                        ctrC, sw.Elapsed)

      ' Use compiled Regex.Match(String, Int32) to find all matches.
      sw = Stopwatch.StartNew()
      Dim posC As Integer = 0
      Dim ctr2c As Integer = 0
      Dim instRegex3c As New Regex(pattern, RegexOptions.Multiline Or RegexOptions.Compiled)
      Dim match3c As Match
      Do
         match3c = instRegex3c.Match(input, posC)
         If match3c.Success Then
            ctr2c += 1
            posC = match3c.Index + match3c.Length
         End If
      Loop While match3c.Success And posC <= input.Length
      sw.Stop()
      Console.WriteLine("Found {0} matches in {1} using compiled regex.Match(String, Int32)",
                        ctr2c, sw.Elapsed)
   End Sub
End Module

[C#]
using System;
using System.Diagnostics;
using System.IO;
using System.Text.RegularExpressions;

public class Example
{
   public static void Main()
   {
      string pattern = @"\b(\w+((\r?\n)|,?\s))*\w+[.?:;!]";
      string input = new StreamReader(@".\OneSentence.txt").ReadToEnd();
      Stopwatch sw;

      Console.WriteLine("***Interpreted regular expressions***");
      // Use Regex.Matches method to find all matches.
      sw = Stopwatch.StartNew();
      Regex instRegex = new Regex(pattern, RegexOptions.Singleline);
      MatchCollection matches = instRegex.Matches(input);
      foreach (Match match in matches)
      {
         // Do nothing. If necessary, the MatchCollection will be populated using
         // lazy evaluation.
      }
      sw.Stop();
      Console.WriteLine("Found {0} matches in {1} using regex.Matches();",
                        matches.Count, sw.Elapsed);

      // Use Regex.Match method followed by Match.NextMatch to find all matches.
      sw = Stopwatch.StartNew();
      int ctr = 0;
      Regex instRegex2 = new Regex(pattern, RegexOptions.Multiline);
      Match match2 = instRegex2.Match(input);
      if (match2.Success)
      {
         do
         {
            ctr++;
            match2 = match2.NextMatch();
         } while (match2.Success);
      }
      sw.Stop();
      Console.WriteLine("Found {0} matches in {1} using regex.Match();/NextMatch();",
                        ctr, sw.Elapsed);

      // Use Regex.Match(String, Int32); to find all matches.
      sw = Stopwatch.StartNew();
      int pos = 0;
      int ctr2 = 0;
      Regex instRegex3 = new Regex(pattern, RegexOptions.Multiline);
      Match match3;
      do
      {
         match3 = instRegex3.Match(input, pos);
         if (match3.Success)
         {
            ctr2++;
            pos = match3.Index + match3.Length;
         }
      } while (match3.Success & pos <= input.Length);
      sw.Stop();
      Console.WriteLine("Found {0} matches in {1} using regex.Match(String, Int32);",
                        ctr2, sw.Elapsed);

      Console.WriteLine();
      Console.WriteLine("***Compiled regular expressions***");

      // Use compiled Regex.Matches method to find all matches.
      sw = Stopwatch.StartNew();
      Regex instRegexC = new Regex(pattern, RegexOptions.Multiline | RegexOptions.Compiled);
      MatchCollection matchesC = instRegexC.Matches(input);
      foreach (Match matchC in matchesC)
      {
         // Do nothing. If necessary, the MatchCollection will be populated using
         // lazy evaluation.
      }
      sw.Stop();
      Console.WriteLine("Found {0} matches in {1} using compiled regex.Matches();",
                        matchesC.Count, sw.Elapsed);

      // Use compiled Regex.Match method followed by Match.NextMatch to find all matches.
      sw = Stopwatch.StartNew();
      int ctrC = 0;
      Regex instRegex2c = new Regex(pattern, RegexOptions.Multiline | RegexOptions.Compiled);
      Match match2c = instRegex2c.Match(input);
      if (match2c.Success)
      {
         do
         {
            ctrC++;
            match2c = match2c.NextMatch();
         } while (match2c.Success);
      }
      sw.Stop();
      Console.WriteLine("Found {0} matches in {1} using compiled regex.Match/NextMatch",
                        ctrC, sw.Elapsed);

      // Use compiled Regex.Match(String, Int32); to find all matches.
      sw = Stopwatch.StartNew();
      int posC = 0;
      int ctr2c = 0;
      Regex instRegex3c = new Regex(pattern, RegexOptions.Multiline | RegexOptions.Compiled);
      Match match3c;
      do
      {
         match3c = instRegex3c.Match(input, posC);
         if (match3c.Success)
         {
            ctr2c++;
            posC = match3c.Index + match3c.Length;
         }
      } while (match3c.Success & posC <= input.Length);
      sw.Stop();
      Console.WriteLine("Found {0} matches in {1} using compiled regex.Match(String, Int32);",
                        ctr2c, sw.Elapsed);
   }
}

As the following output and chart show, the interpreted regular expression is significantly faster than its compiled counterpart.

***Interpreted regular expressions***
Found 1 matches in 00:00:00.0014471 using regex.Matches()
Found 1 matches in 00:00:00.0000689 using regex.Match()/NextMatch()
Found 1 matches in 00:00:00.0000656 using regex.Match(String, Int32)

***Compiled regular expressions***
Found 1 matches in 00:00:00.0079332 using compiled regex.Matches()
Found 1 matches in 00:00:00.0053529 using compiled regex.Match/NextMatch
Found 1 matches in 00:00:00.0057524 using compiled regex.Match(String, Int32)

If you plan to call regular expression methods just a few times for a particular regular expression, or if the approximate number of calls to regular expression methods is unknown but is expected to be rather small, use interpreted regular expressions for best performance.

Compiled Regular Expressions

In contrast to interpreted regular expressions, compiled regular expressions increase startup time but execute individual pattern-matching methods faster. This means that the performance benefit that results from compiling the regular expression increases with the number of regular expression methods called.

If the previous example, which opens a file that contains a single sentence, is replaced with an example that extracts each sentence from the entire text of Theodore Dreiser’s The Financier, the performance benefit that results from compiling a regular expression becomes very clear. This variation of the example produces the following output, which shows that methods of compiled regular expressions execute about 30% faster than methods of interpreted regular expressions.

***Interpreted regular expressions***
Found 13679 matches in 00:00:01.9024852 using regex.Matches()
Found 13679 matches in 00:00:01.8579707 using regex.Match()/NextMatch()
Found 13679 matches in 00:00:01.8241908 using regex.Match(String, Int32)

***Compiled regular expressions***
Found 13679 matches in 00:00:01.1399927 using compiled regex.Matches()
Found 13679 matches in 00:00:01.1378395 using compiled regex.Match/NextMatch
Found 13679 matches in 00:00:01.1460928 using compiled regex.Match(String, Int32)

Static Interpreted and Compiled Regular Expressions

Just as instances of the Regex class can be either compiled or interpreted, so can regular expressions used in static method calls. In interpreted static regular expressions, the internal regular expression opcodes are cached. In compiled static regular expressions, the MSIL is cached. As a result, compiled static regular expressions are faster than interpreted static regular expressions in cases in which methods are called repeatedly with the same regular expression.

The previous two examples used a regular expression to extract sentences from a one-sentence string and from the complete text of Theodore Dreiser’s The Financier. If we combine the two examples but replace the calls to instance Regex methods with calls to the corresponding static methods, the following output results:

Reading file OneSentence.txt
***Interpreted regular expressions***
Found 1 matches in 00:00:00.0014642 using static Regex.Matches()
Found 1 matches in 00:00:00.0002807 using static Regex.Match()/NextMatch()

***Compiled regular expressions***
Found 1 matches in 00:00:00.0076935 using compiled static Regex.Matches()
Found 1 matches in 00:00:00.0089727 using compiled static Regex.Match/NextMatch

Reading file .\Dreiser_TheFinancier.txt
***Interpreted regular expressions***
Found 13679 matches in 00:00:02.1395809 using static Regex.Matches()
Found 13679 matches in 00:00:02.1683064 using static Regex.Match()/NextMatch()

***Compiled regular expressions***
Found 13679 matches in 00:00:01.2329685 using compiled static Regex.Matches()
Found 13679 matches in 00:00:01.1957317 using compiled static Regex.Match/NextMatch

This output shows the same relationship between interpreted and compiled static regular expressions that we observed for interpreted and compiled instance regular expressions. However, as the two charts that compare the performance of static and instance methods when making a single method call and when making multiple method calls to process a large block of text show, static regular expression methods offer performance that is generally inferior to instance regular expression methods. Their advantage lies in scenarios that use regular expression caching as an alternative to repeatedly re-instantiating the same regular expression object.

Regular Expressions Compiled to an Assembly

The .NET Framework also allows you to create an assembly that contains compiled regular expressions. This moves the performance hit of regular expression compilation from runtime to design time. However, it also involves some additional work: You must define the regular expressions in advance and compile them to an assembly. The compiler can then reference this assembly when compiling source code that uses the assembly’s regular expressions.

To compile regular expressions to an assembly, you call the Regex.CompileToAssembly method and pass it an array of RegexCompilationInfo objects that represent the regular expressions to be compiled, and an AssemblyName object that contains information about the assembly to be created.

For example, we can compile and store the regular expression that we’ve used to extract sentences from a string in a separate assembly. The following example does this:

[Visual Basic]
Imports System.Reflection
Imports System.Text.RegularExpressions

Module Example
   Public Sub Main()
      Dim SentencePattern As New RegexCompilationInfo("\b(\w+((\r?\n)|,?\s))*\w+[.?:;!]",
                                                      RegexOptions.Multiline,
                                                      "SentencePattern",
                                                      "Utilities.RegularExpressions",
                                                      True)
      Dim regexes() As RegexCompilationInfo = {SentencePattern}
      Dim assemName As New AssemblyName("RegexLib, Version=1.0.0.1001, Culture=neutral, PublicKeyToken=null")
      Regex.CompileToAssembly(regexes, assemName)
   End Sub
End Module
[C#]
using System;
using System.Reflection;
using System.Text.RegularExpressions;

public class Example
{
   public static void Main()
   {
      RegexCompilationInfo SentencePattern =
                           new RegexCompilationInfo(@"\b(\w+((\r?\n)|,?\s))*\w+[.?:;!]",
                                                    RegexOptions.Multiline,
                                                    "SentencePattern",
                                                    "Utilities.RegularExpressions",
                                                    true);
      RegexCompilationInfo[] regexes = { SentencePattern };
      AssemblyName assemName = new AssemblyName("RegexLib, Version=1.0.0.1001, Culture=neutral, PublicKeyToken=null");
      Regex.CompileToAssembly(regexes, assemName);
   }
}

When the example is compiled to an executable and run, it creates an assembly named RegexLib.dll. The regular expression is represented by a class named Utilities.RegularExpressions.SentencePattern that is derived from Regex, as the following MSIL Disassembler (IL DASM) window illustrates.

When you use this compiled regular expression to parse the text from the single-sentence file and from Theodore Dreiser’s The Financier, the following output is displayed. Calls to the SentencePattern pattern-matching methods produce execution times that are comparable to instance regular expressions that are called relatively few times, and to compiled regular expressions that are called repeatedly.

Reading file .\OneSentence.txt
***Regular expression compiled to an assembly***
Found 1 matches in 00:00:00.0062998 using Matches()
Found 1 matches in 00:00:00.0000396 using Match()/NextMatch()
Found 1 matches in 00:00:00.0000181 using Match(String, Int32)

Reading file .\Dreiser_TheFinancier.txt
***Regular expression compiled to an assembly***
Found 13679 matches in 00:00:01.1983633 using Matches()
Found 13679 matches in 00:00:01.1931376 using Match()/NextMatch()
Found 13679 matches in 00:00:01.1720003 using Match(String, Int32)

Regular expressions that are compiled to assemblies are typically considered less flexible than interpreted or compiled regular expressions. It is usually argued that they cannot support regular expressions that are built dynamically. Moreover, they are tightly coupled with the regular expression options that were provided to the regular expression’s RegexCompilationInfo object. For example, the case sensitivity of a pattern match must be specified at the time the assembly is created; it cannot be defined by an argument supplied to a regular expression pattern-matching method.

In fact, however, regular expressions can be compiled to an assembly dynamically, and regular expression objects can be defined dynamically as well. Although assemblies that contain compiled regular expressions are generally built separately from the application that uses them, this need not be the case. An assembly can be built dynamically by its application, the application can load the assembly dynamically by using reflection, and the regular expression object’s methods can also be called by using reflection. Note, however, that these techniques rely on reflection, which has a significant performance impact.

Best Practices for Regular Expression Object Usage

To maximize the performance of regular expressions in the .NET Framework, we recommend that you do the following when working with regular expression objects:

Call static regular expression matching methods instead of repeatedly instantiating the same regular expression object.
Use interpreted regular expressions when you expect to call pattern-matching methods a limited number of times to parse text.
Use compiled regular expressions to optimize performance for regular expression patterns that are known in advance and when the frequency of calls to the regular expression’s pattern-matching methods can vary extensively.

Part II of this series will examine how you can improve regular expression performance by crafting regular expressions that limit excessive backtracking.

Code Contracts Survey

BCL Team — Mon, 24 May 2010 22:38:06 GMT

The BCL Team has put together a survey about Code Contracts. We're trying to get data about how people are using the feature to help us figure out where to invest in further development. Even if you haven't used Code Contracts yet, your answers would be quite helpful. We have a few questions to help us determine the main reasons people haven't tried the feature. If you have a few minutes to spare, the survey can be found here: http://codecontracts.questionpro.com.

Thanks in advance for your help.

Constructors and Value Types [Ron Petrusha]

BCL Team — Wed, 28 Apr 2010 20:50:00 GMT

A number of recent customer comments have made it clear that some confusion surrounds the use of constructors with value types. Most common are the complaints that we've failed to document a constructor for a particular value type. In one case, though, a customer asked that we remove the documentation on constructors for value types, since he or she was certain that they are not supported by the .NET Framework.

Just as you can define parameterized constructors for reference types, you can define parameterized constructors for value types. For example, the following code defines a VehicleInfo structure that has two parameters: the number of wheels on a vehicle, and its number of doors.

[Visual Basic]

Public Structure VehicleInfo

Private nWheels As Integer

Private nDoors As Integer

Public Sub New(ByVal wheels As Integer, ByVal doors As Integer)

nWheels = wheels

nDoors = doors

End Sub

Public ReadOnly Property NumberOfWheels() As Integer

Get

Return nWheels

End Get

End Property

Public ReadOnly Property Doors() As Integer

Get

Return nDoors

End Get

End Property

End Structure

[C#]

using System;

public struct VehicleInfo

{

private int nWheels;

private int nDoors;

public VehicleInfo(int wheels, int doors)

{

nWheels = wheels;

nDoors = doors;

}

public int NumberOfWheels

{

get { return nWheels; }

}

public int Doors

{

get { return nDoors; }

}

The example compiles normally, and when we use IL DASM to examine our type, we can see that it includes the parameterized constructor.

So far, we haven't observed any differences between defining constructors for value types and defining them for reference types. However, this is where the similarity ends. While we can define a default or parameterless constructor for reference types, the attempt to define a default constructor for value types produces a compiler error. For example, if we modify the previous code by replacing the parameterized constructor with a default constructor and adding setters for the structure's two properties, the Visual Basic compiler displays the following error message:

error BC30629: Structures cannot declare a non-shared 'Sub New' with no parameters.

The equivalent C# code also fails to compile, with a slightly different error message:

error CS0568: Structs cannot contain explicit parameterless constructors

Not only can we not define a parameterless constructor for value types, but compilers don't supply one automatically if there is no defined constructor, as they do for reference types. For example, if we again modify our source code by deleting the parameterless constructor and then successfully compile, we can again use IL DASM to inspect the members of our type. As the following figure shows, a parameterless constructor is not a member of the structure.

So to summarize, there are two major differences in the definition of constructors between value types and reference types:

You can define a parameterless constructor for a reference type. You cannot define one for a value type.
If you do not define any constructors, the compiler supplies a parameterless constructor for a reference type. It does not supply one for a value type.

Given this, it may seem surprising that if we attempt to instantiate a VehicleInfo object by calling its parameterless constructor, as the following example does, our code compiles successfully.

[Visual Basic]

Module Example

Public Sub Main()

Dim bicycle As New VehicleInfo()

bicycle.Doors = 0

bicycle.NumberOfWheels = 2

End Sub

End Module

[C#]

using System;

public class Example

{

public static void Main()

{

VehicleInfo bicycle = new VehicleInfo();

bicycle.Doors = 0;

bicycle.NumberOfWheels = 2;

}

On the one hand, no parameterless constructor exists in our VehicleInfo type. On the other hand, we apparently are able to call this non-existent constructor. How can we explain this incongruous behavior?

Let's begin by using IL DASM to examine the MSIL for a "normal" call to a parameterless constructor from a class. For the purpose of the example, we can define a class with no members named TypicalClass and then instantiate it from another class that contains Main, the application entry point. The following is the MSIL emitted by a Visual Basic compiler for the Main method. Note the explicit call to the TypicalClass default constructor. (A C# compiler also emits an identical call to the TypicalClass default constructor.)

.method public static void Main() cil managed

{

.entrypoint

.custom instance void [mscorlib]System.STAThreadAttribute::.ctor() = ( 01 00 00 00 )

// Code size 7 (0x7)

.maxstack 1

.locals init (class TypicalClass V_0)

IL_0000: newobj instance void TypicalClass::.ctor()

IL_0005: stloc.0

IL_0006: ret

} // end of method Example::Main

We can then compare this with the IL generated by our call to the non-existent VehicleInfo parameterless constructor. The following is the IL emitted by the C# compiler, although for our purposes the IL emitted by the Visual Basic compiler is identical.

.method public hidebysig static void Main() cil managed

{

.entrypoint

// Code size 28 (0x1c)

.maxstack 2

.locals init (valuetype [NoConstructor1]VehicleInfo V_0)

IL_0000: nop

IL_0001: ldloca.s V_0

IL_0003: initobj [NoConstructor1]VehicleInfo

IL_0009: ldloca.s V_0

IL_000b: ldc.i4.0

IL_000c: call instance void [NoConstructor1]VehicleInfo::set_Doors(int32)

IL_0011: nop

IL_0012: ldloca.s V_0

IL_0014: ldc.i4.2

IL_0015: call instance void [NoConstructor1]VehicleInfo::set_NumberOfWheels(int32)

IL_001a: nop

IL_001b: ret

} // end of method Example::Main

The call to the non-existent parameterless constructor in our source code has been replaced by a call to Initobj, an MSIL opcode that is documented at http://msdn.microsoft.com/en-us/library/system.reflection.emit.opcodes.initobj(VS.100).aspx. Initobj initializes each field in a value type to its default value.

This feature - the ability to instantiate a value type by using a language's constructor syntax to invoke a default value type even though one does not exist -- is known as the implicit default constructor. In C# in particular, it is necessary to use the implicit default constructor to instantiate value types that do not define parameterized constructors, since otherwise the C# compiler generates error CS0165: "Use of unassigned local variable 'bicycle'", because C# does not allow variables to be used before they're explicitly initialized.

On the other hand, Visual Basic does not forbid the use of unassigned value types, since it provides a call to the implicit default constructor if none is present in source code. For example, the following Visual Basic code is similar to the previous example, except that it omits the call to the VehicleInfo default constructor.

[Visual Basic]

Module Example

Public Sub Main()

Dim bicycle As VehicleInfo

bicycle.Doors = 0

bicycle.NumberOfWheels = 2

End Sub

End Module

The following is the MSIL emitted by a Visual Basic compiler for this version of the Main method.

.method public static void Main() cil managed

{

.entrypoint

.custom instance void [mscorlib]System.STAThreadAttribute::.ctor() = ( 01 00 00 00 )

// Code size 17 (0x11)

.maxstack 2

.locals init (valuetype [NoConstructor1]VehicleInfo V_0)

IL_0000: ldloca.s V_0

IL_0002: ldc.i4.0

IL_0003: call instance void [NoConstructor1]VehicleInfo::set_Doors(int32)

IL_0008: ldloca.s V_0

IL_000a: ldc.i4.2

IL_000b: call instance void [NoConstructor1]VehicleInfo::set_NumberOfWheels(int32)

IL_0010: ret

} // end of method Example::Main

In this case, the Initobj opcode has been replaced by init, but the result is the same: each field of a value type is initialized to its default value. In Visual Basic, an implicit default constructor is invoked whether or not that constructor is actually used in source code.

BCL CodePlex Site Launch

BCL Team — Tue, 30 Mar 2010 21:03:00 GMT

Today the BCL Team is launching a CodePlex site to host samples, previews, and prototypes. You can find it at http://bcl.codeplex.com.

This is a site for the BCL Team to get features to customers to try out without requiring a Beta or CTP of the .NET Framework. Our goal is to put generally useful functionality on the CodePlex site, and to get feedback on it and have the chance to iterate on the design. Having a feature there does not mean that it will eventually end up in the BCL. Some items are samples that build on top of existing classes, and some features might be ones we were considering for the .NET Framework but decided not to include for one reason or another.

We'd love to get your feedback in the form of comments, bug reports, and feature requests, but please note that we cannot take code submissions. We plan to release updates with new features and updates to existing features on a quarterly basis, but we're just getting started with this and may adjust our schedule depending on how things go.

And just to avoid confusion, you will not find the source code for the whole BCL on this CodePlex site. The site is for things we're thinking about for the future, not our existing classes.

The first four projects up on the site are BigRational, Long Path, PerfMonitor, and TraceEvent. Go check them out and let us know what you think.

I just installed Visual Studio 2010, now how do I get Code Contracts? [Melitta Andersen]

BCL Team — Wed, 27 Jan 2010 00:57:00 GMT

This article is to show you how to go from a clean install of Visual Studio 2010 to running the Code Contract tools for the first time. Since the tools aren’t included in VS install, we thought we’d provide some extra information on where to find them. If you’re already using Code Contracts in Visual Studio 2010, this blog post probably won’t have anything new for you. If you want an overview of the System.Diagnostics.Contracts.Contract class, please see earlier blog posts or documentation. This post is just about getting set up for the very first time.

This article might also be useful to you if you’ve tried to run a project and gotten an assert stating that “This assembly must be rewritten using the binary rewriter.” That can happen if you’ve used certain methods on the System.Diagnostics.Contracts.Contract class in your code and defined the appropriate symbol to have them compiled into your code, but haven’t run the tools mentioned here.

If you want the really quick version of this post, the answer is to go download the appropriate installer linked to on the right side of this page: http://msdn.microsoft.com/en-us/devlabs/dd491992.aspx. Then use the Code Contracts pane in the Properties of your project to perform runtime contract checking. Without this setting, contracts are not checked - we don’t compile them into your code without this option. If you want a few more details, read on.

The Contract class is included in the Microsoft .NET Framework 4, but the tools that take advantage of them are found on DevLabs (more on that later). There are two primary tools: ccrewrite.exe, a binary rewriter that moves your contracts to the correct place in the code and checks whether the contracts are written correctly, and cccheck.exe, a static analysis tool that will find places where contracts have been violated without running your code. Think of cccheck like FxCop for finding bugs, using the rules you establish via your own contracts. You cannot do runtime checking of your contracts without using ccrewrite.exe. Because of this, most of the methods on the Contract class are conditionally compiled, which means that calls to them won’t be included in your code without a special symbol being defined. The tools provide an easy way to define that symbol in a new tab on the properties pane of your project and run ccrewrite immediately after compiling your assemblies.

Downloading from DevLabs

DevLabs is a Microsoft site for sharing prototypes of tools for developers and getting feedback on them from the community. The Code Contract tools started out as a project in Microsoft Research. We’re working on getting them incorporated into products, but to get them out where our customers can use them in this release, we’re hosting them on DevLabs. In addition to being a place to host the tools, the Code Contracts page also has a forum, FAQ, and documentation for the tools.

There are two different installers for the Code Contracts tools. The one that includes the static checker requires VS 2010 Premium or VS 2010 Ultimate. The version without the static checker will work on any edition of VS 2010 other than the Express Edition, and is what the DevLabs site refers to (for now) as the “Standard Edition” of the tools. Pick which version works for you, and you’ll get the usual prompt asking if you want to run or save the file.

Enabling Contracts in Projects

The installer adds a tab to the properties pane for projects in Visual Studio. It has quite a few options that are described in the aforementioned documentation for the tools. The two most important, though, are the checkboxes to Perform Runtime Contract Checking and (if you have the static checker) to Perform Static Contract Checking. The option to perform runtime checking will define the symbol to put contracts in your code and enable a post-build step to put the checks for those contracts in the correct place (for instance, placing checks for post-conditions at every exit of a method). If those contracts are violated, an assertion or exception will happen, depending on your selections for the other options on the tab. If you enable static contract checking, cccheck.exe will run in the background to try to prove all of the contracts you have written in your code.

I hope this helps you to get started using Code Contracts. If you’re just getting started with the feature, I strongly encourage you to spend some time on the forum or reading the documentation on MSDN.