Say wwhhhaaaat?

Windows Communication Foundation - The Basics (Part 1)

2011-08-03T10:07:00Z

No matter how much I read about WCF in online forums and blog posts, I still couldn’t grasp the whole idea behind the extensibility model until I was able to play with it myself. What I discovered along the way is how the entire channel model works and how the binding and endpoint address fit into the equation. Although this may seem like I’m diving in details quickly, I personally find it very useful to illustrate real world scenarios to really show you the power of the API. On that note, I’d like to dive right in to how WCF generates your proxy channel for you at runtime and how the binding fits into this model. With this knowledge we will be able to create our own custom channel implementation that provides a sort of “message replay” ability in the face of connectivity issues.

Binding

The way I think about the binding class is that it is nothing more than the actual networking stack implementation. If you have any background in computer networking, you should be fully aware that the low-level TCP/IP network protocols are layered on top of each other. In this example, IP is the core transport protocol and TCP packets are simply carried as a payload. If we view this layering of protocols by analyzing the flow of packets, we find something like this:

Sender: TCP –> IP ……. Receiver: IP –> TCP

As you can see, on the way out the IP layer is the last to see the packets, and on the way in the IP layer is the first to receive the packets. In order for this system to work the two layers need to play by the rules, meaning they conform to a common set of rules and restrictions. Much in this same way, the Binding object provides the ability to layer protocols on top of one another, just at a higher level than the previous example. This provides a generic way to model a networking stack, where each layer performs some function and typically delegates further behavior to the next layer down in the stack. From a binding, you can build a IChannelFactory<TChannel> instance where TChannel is one of the built-in channel shapes (e.g. IRequestChannel, IOutputChannel, etc). Depending on how you order the binding elements in the binding, you wind up with a different layering of the channels involved.

Conclusion

The main purpose of the Binding object is to construct the lower level networking stack, which by itself this is not completely useful since we are still communicating with remote services at a very low level. We still need to handle formatting incoming and outgoing messages manually, which typically isn’t the most desirable thing to do. This is where the next layer of the WCF model comes in, which provides this mapping of service methods to messages and understands how to format both requests and responses to and from the remote service.

Next time we’ll take a look at the ServiceChannel object and how it uses the Binding to provide us with a usable client service.

Windows Communication Foundation–The Journey Begins ..

2010-09-09T00:19:12Z

The 2010 release provided a dramatic shift in the build platform. Previous releases were built on top of MSBuild, providing integration through custom tasks and a special logger for tracking events during a build. Current and future releases are built on Windows Workflow Foundation, providing a design experience in Visual Studio and the ability to orchestrate a build process across multiple machines. During the product development cycle I learned quite a bit about workflow (having access to the architect didn’t hurt), since I was essentially forced to do so by the collective direction of my team. Once the product cycle was over I took some of my down time to post knowledge gained during the process to (hopefully) help others learn.

Now we’re developing the next release and I have been driven down a similar, yet different, path of making extensive use of the Windows Communication Foundation libraries. Through many long nights (sadly enough I actually found that I missed them) I have been slowly expanding my knowledge base in this area of the .NET framework. I must admit that upon first glance the API seems extremely complicated. The size of the library is enormous and there are so many customization points it becomes tough to determine the appropriate hook. However, once you understand the pieces and how they fit together you have infinite possibilities at your disposal.

Sorry to spoil the fun, but this isn’t the first installment …. soon though, very soon.

Busy busy busy ...

2010-06-01T18:45:09Z

I apologize to my reader or so for the lack of updates recently. I have been knee deep in work for our next release and have not had much spare time. I will also be fixing my code segments soon which are now cut off due to the re-theming of our blogs.

Patrick

Windows Workflow 4.0 – Workflow Instance Extensions

2010-01-19T19:01:00Z

Update 01/20/2009: Since I failed to mention this before I wanted to make something clear about these workflow posts. All information in these posts is based on the knowledge I gained through developing a framework based on Windows Workflow 4.0. I am not a member of the team nor is this information in any way endorsed by the team.

In the previous posts we covered one way you could implement a producer-consumer pattern in workflow. This approach used composition and attempted to hide the complexities of the underlying operations inside of the respective activities through composition. However, one scenario in particular this model does not allow you to implement involves modeling of existing .NET events in a workflow. For instance, if we want to implement a FileSystemWatcher activity we need to provide some way for consumers to hook into the events exposed by the .NET object. The only way to accomplish our goal is to hook up to the .NET events and schedule workflow actions from our event handler – but how would this be implemented? Let’s take a look at some more pieces of the workflow framework that will allow us to accomplish this goal.

Workflow Extensions

Each workflow instance has a collection of objects known as extensions that are able to interface with both the workflow and the host. By default no extensions exist, but they may be added to an instance by the host prior to calling the run method using the WorkflowInstanceExtensionManager. Both WorkflowApplication and WorkflowInvoker provide access to the extension manager by a property named Extensions. There are two distinct ways to attach an extension to a workflow instance using this property:

WorkflowInstanceExtensionManager.Add(Object singletonExtension) – Allows you to add a singleton instance of an object to the workflow instance. This is useful if you intend to share a particular extension across instances. It is up to the host to manage the lifetime of the service, although the workflow will remove references to your service.
WorkflowInstanceExtensionManager.Add<T>(Func<T> extensionCreationFunction) – Allows you to provide an anonymous function to create your extension on demand. Services created through this mechanism are owned by the workflow instance so the lifetime is managed by the instance as well. If the service object implements IDisposable then the dispose method will be invoked when the instance is unloaded.

There is a third way to add extensions to a workflow instance, and is typically used when an activity requires an extension for correct function. If you have been following through these posts, you may recall our usage of the method NativeActivity.CacheMetadata(NativeActivityMetadata) when implementing the ParallelItemScheduler<T>. In addition to providing a location to declare your arguments and variables, you may also use the method for validation warnings/errors and more importantly providing extension creation functions for required extensions.

NativeActivityMetadata.AddDefaultExtensionProvider<T>(Func<T> extensionProvider) where T : class – Allows an activity to inform the runtime to create the specified extension using the provided function if the extension does not already exist. The host may or may not provide this particular extension, so the runtime ensures that only one instance of this extension type is created.

Another related scenario for the CacheMetadata method is to inform the runtime that you require an extension that is not provided by you. The following two APIs allow you to specify extension types that you require for an activity to function correctly. If the extension does not exist at runtime a validation error is created informing the user that a required extension does not exist.

NativeActivityMetadata.RequireExtension<T>() – Informs the runtime that an extension of type T is required for this activity to work.
NativeActivityMetadata.RequireExtension(Type extensionType) – Informs the runtime that an extension of type extensionType is required for this activity to work.

If you take a quick look back at the API for adding extensions you’ll notice that an extension is just an Object and is not restricted to any particular base class or type. However, there are some scenarios where the extension may need to interact with the workflow via bookmarks, and in order to do that we need some way to get the WorkflowInstanceProxy (a trimmed down version of WorkflowApplication that provides limited method access from inside a workflow) so we can resume bookmarks. This is where the interface IWorkflowInstanceExtension comes in handy! This interface is defined as follows:

namespace System.Activities.Hosting
{
    public interface IWorkflowInstanceExtension
    {
        IEnumerable<Object> GetAdditionalExtensions();
        void SetInstance(WorkflowInstanceProxy instance);
    }
}

Any time an extension is attached (though either mechanism) that implements the above interface the runtime will invoke both methods. The first method provides an entry point for creating additional extensions. The second method is the more interesting one for our current discussion, as it provides the extension with the owning workflow instance proxy. A typical implementation for this method is to simply store the provided parameter into a member variable for later use by the extension. For illustration purposes, and since it leads us to our next destination, I have pasted an example of how we might design an item scheduling extension for interfacing with our ParallelItemScheduler<T> activity.

public sealed class ItemSchedulerExtension<T> : IWorkflowInstanceExtension
{
    internal ItemSchedulerExtension()
    {
    }

    IEnumerable<Object> IWorkflowInstanceExtension.GetAdditionalExtensions()
    {
        return null;
    }

    void IWorkflowInstanceExtension.SetInstance(WorkflowInstanceProxy instance)
    {
        m_instance = instance;
    }

    public void Schedule(Bookmark bookmark, T item)
    {
        m_instance.BeginResumeBookmark(bookmark, item, s_timeout, new AsyncCallback(OnEndResumeBookmark), null);
    }

    void OnEndResumeBookmark(IAsyncResult result)
    {
        try
        {
            BookmarkResumptionResult resumptionResult = m_instance.EndResumeBookmark(result);
            Debug.Assert(resumptionResult == BookmarkResumptionResult.Success, "Error resuming bookmark. Reason: " + resumptionResult);
        }
        catch (Exception ex)
        {
            Debug.Fail("Exception occurred while resuming bookmark.", ex.ToString());
        }
    }

    private WorkflowInstanceProxy m_instance;
    static readonly TimeSpan s_timeout = TimeSpan.FromMinutes(2);
}

As you can see the implementation of this particular extension is minimal and straightforward. We do not have additional extensions so we can simply return null. Our SetInstance implementation simply stores the workflow instance proxy into our member variable so we can use it in our Schedule implementation. Finally, the schedule method simply resumes a bookmark using the asynchronous pattern for the ResumeBookmark method.

Note: I would like to point out that the synchronous versions of the bookmark methods are intentionally missing from the proxy object, since resuming a bookmark requires the workflow thread to be free. If an activity attempts to resume a bookmark on an asynchronous thread (using this extension) and the workflow thread is busy, the synchronous call blocks until the workflow thread becomes available to schedule the bookmark resumption in the queue. In order to encourage better programming practices and reduce confusion regarding which method to invoke the synchronous methods have been removed.

Conclusion

In this article we discussed workflow extensions, how the lifetime is managed, and how to attach them to a workflow instance. We also discussed how an activity can provide an extension by providing information in the CacheMetadata(*ActivityMetadata) family of methods. If you recall from the introduction I alluded to modeling a .NET event-driven object in a workflow activity. So far we have covered the first piece of the puzzle – next post I plan on revising the producer-consumer model, illustrating a different way to implement the CopyDirectory activity using this new model.

Windows Workflow 4.0 – CopyDirectory (Part 3 of 3)

2010-01-08T20:10:00Z

So far we have written ParallelItemScheduler<T>, which is a generic activity that goes dormant and schedules actions based on input via bookmark resumption. We then used this activity to design ProducerConsumer<T>, which takes this idea a little further and provides a generic mechanism for running a producer thread that can pump data into a consumer thread. In the last part of this series we finally have all required building blocks to design our highly efficient and parallel copy directory activity!

As with any activity, the first thing we need to determine is what the inputs and outputs should be. Since this is simply a copy operation, there will be no outputs. The inputs I have chosen are as follows:

InArgument<String> Source
- The source directory for the copy operation.
InArgument<String> Target
- The target directory for the copy operation.
InArgument<Int32> FileCopyBufferSize
- The buffer size which should be used for individual copy operations.
InArgument<Int32> FileCopyThreadCount
- The number of concurrent copies that should be allowed.

Now that we have our inputs, we can move onto the producer and the consumer for our directory copy activity.

Implementing CopyDirectory

The producer of our copy directory activity is what will actually walk the source hierarchy and accumulate files for consumption. I will not be posting the code in-line here, but I will describe the implementation so the rest of this makes sense without having to download an attachment. Basically the producer maintains a queue or stack (depending on how the copy operation should work, breadth-first or depth-first respectively) and begins by pushing the root directory onto the list. It then begins a loop that retrieves the next directory from the list, finds the files and directories that are immediate children, pushes the directories onto the aforementioned list and then queues the files for consumption by our consumer (which will be discussed next). In order to help illustrate this concept, I have rewritten it into an outline form.

Get the next directory
Calculate the target directory by removing Source and replacing it with Target
Create the target directory
Asynchronously walk the current directory to find all immediate child objects
For-each directory
- Push directory into our list
For-each file
- Queue a Tuple<String, String> with Item1 set to the source file and Item2 set to the target file

Although it’s not entirely necessary the directory/file discovery activity is asynchronous to allow for maximum parallelism between the producer and the consumer. Just like our activity needs a source and target for the operation, so does our consumer which will be the CopyFile activity. For this reason we will use the ProducerConsumer<Tuple<String, String>> in our implementation below.

Note: The activities Enqueue, Dequeue, CreateDirectory and GetFilesAndDirectories have been omitted for length. I have attached a full project that includes all code and supporting classes that you may download.

public sealed class CopyDirectory : Activity
{
    public CopyDirectory()
    {
        base.Implementation = () => CreateBody();

        m_fileCopyBufferSize = new InArgument<Int32>(new VisualBasicValue<Int32>("1024 * 1024 * 8"));
        m_fileCopyThreadCount = new InArgument<Int32>(new VisualBasicValue<Int32>("System.Environment.ProcessorCount * 2"));
    }

    [RequiredArgument]
    [DefaultValue(null)]
    public InArgument<String> Source
    {
        get;
        set;
    }

    [RequiredArgument]
    [DefaultValue(null)]
    public InArgument<String> Target
    {
        get;
        set;
    }


    public InArgument<Int32> FileCopyBufferSize
    {
        get
        {
            return m_fileCopyBufferSize;
        }
        set
        {
            m_fileCopyBufferSize = value;
            m_fileCopyBufferSizeSet = true;
        }
    }

    public InArgument<Int32> FileCopyThreadCount
    {
        get
        {
            return m_fileCopyThreadCount;
        }
        set
        {
            m_fileCopyThreadCount = value;
            m_fileCopyThreadCountSet = true;
        }
    }

    [EditorBrowsable(EditorBrowsableState.Never)]
    public Boolean ShouldSerializeFileCopyBufferSize()
    {
        return m_fileCopyBufferSizeSet;
    }

    [EditorBrowsable(EditorBrowsableState.Never)]
    public Boolean ShouldSerializeFileCopyThreadCount()
    {
        return m_fileCopyThreadCountSet;
    }

    Activity CreateBody()
    {
        Variable<String> bookmarkName = new Variable<String>("bookmarkName");
        Variable<String> sourceDirectory = new Variable<String>("sourceDirectory");
        Variable<String> targetDirectory = new Variable<String>("targetDirectory");
        Variable<Queue<String>> pendingDirectories = new Variable<Queue<String>>("pendingDirectories");
        Variable<IEnumerable<String>> childDirs = new Variable<IEnumerable<String>>("childDirs");
        Variable<IEnumerable<String>> childFiles = new Variable<IEnumerable<String>>("childFiles");

        DelegateInArgument<String> childDirArgument = new DelegateInArgument<String>("childDirArgument");
        DelegateInArgument<String> childFileArgument = new DelegateInArgument<String>("childFileArgument");
        DelegateInArgument<Tuple<String, String>> fileCopyArgument = new DelegateInArgument<Tuple<String, String>>("fileCopyArgument");
        DelegateOutArgument<ICollection<Tuple<String, String>>> childFileCollection = 
            new DelegateOutArgument<ICollection<Tuple<String, String>>>("childFileCollection");

        return new Sequence
        {
            Variables = 
            {
                pendingDirectories,
            },
            Activities =
            {
                new Assign<Queue<String>>
                {
                    Value = new InArgument<Queue<String>>(env => new Queue<String>(new String[] { Source.Get(env) })),
                    To = new OutArgument<Queue<String>>(pendingDirectories),
                },
                new ProducerConsumer<Tuple<String, String>>
                {
                    ConsumerCount = new InArgument<Int32>(env => FileCopyThreadCount.Get(env)),
                    ContinueProducing = ExpressionServices.Convert<Boolean>(env => pendingDirectories.Get(env).Count > 0),
                    Consumer = new ActivityAction<Tuple<String, String>>
                    {
                        Argument = fileCopyArgument,
                        Handler = new CopyFile
                        {
                            Source = new InArgument<String>(env => fileCopyArgument.Get(env).Item1),
                            Target = new InArgument<String>(env => fileCopyArgument.Get(env).Item2),
                            BufferSize = new InArgument<Int32>(env => FileCopyBufferSize.Get(env)),
                        },
                    },
                    Producer = new ActivityFunc<ICollection<Tuple<String, String>>>
                    {
                        Result = childFileCollection,
                        Handler = new Sequence
                        {
                            Variables = 
                            {
                                childDirs,
                                childFiles,
                                sourceDirectory,
                                targetDirectory,
                            },
                            Activities = 
                            {
                                new Dequeue<String>
                                {
                                    Queue = new InArgument<Queue<String>>(pendingDirectories),
                                    Result = new OutArgument<String>(sourceDirectory),
                                },
                                new Assign<String>
                                {
                                    Value = new InArgument<String>(env => System.IO.Path.Combine(Target.Get(env),
                                                                 sourceDirectory.Get(env).Remove(0, Source.Get(env).Length).TrimStart('/', '\\'))),
                                    To = new OutArgument<String>(targetDirectory),
                                },
                                new If
                                {
                                    Condition = new InArgument<Boolean>(env => !System.IO.Directory.Exists(targetDirectory.Get(env))),
                                    Then = new CreateDirectory
                                    {
                                        Directory = new InArgument<String>(targetDirectory),
                                    },
                                },
                                new GetFilesAndDirectories
                                {
                                    Directory = new InArgument<String>(sourceDirectory),
                                    Files = new OutArgument<IEnumerable<String>>(childFiles),
                                    Directories = new OutArgument<IEnumerable<String>>(childDirs),
                                },
                                new ForEach<String>
                                {
                                    Values = new InArgument<IEnumerable<String>>(childDirs),
                                    Body = new ActivityAction<String>
                                    {
                                        Argument = childDirArgument,
                                        Handler = new Enqueue<String>
                                        {
                                            Queue = new InArgument<Queue<String>>(pendingDirectories),
                                            Item = new InArgument<String>(childDirArgument),
                                        },
                                    },
                                },
                                new Assign<ICollection<Tuple<String, String>>>
                                {
                                    To = new OutArgument<ICollection<Tuple<String, String>>>(childFileCollection),
                                    Value = new InArgument<ICollection<Tuple<String, String>>>(env => new List<Tuple<String, String>>()),
                                },
                                new ForEach<String>
                                {
                                    Values = new InArgument<IEnumerable<String>>(childFiles),
                                    Body = new ActivityAction<String>
                                    {
                                        Argument = childFileArgument,
                                        Handler = new AddToCollection<Tuple<String, String>>
                                        {
                                            Collection = new InArgument<ICollection<Tuple<String, String>>>(childFileCollection),
                                            Item = new InArgument<Tuple<String, String>>(env => 
                                                new Tuple<String, String>(
                                                    childFileArgument.Get(env), 
                                                    System.IO.Path.Combine(targetDirectory.Get(env), System.IO.Path.GetFileName(childFileArgument.Get(env))))),
                                        },
                                    },
                                },
                            },
                        },
                    },
                },
            },
        };
    }

    private Boolean m_fileCopyBufferSizeSet;
    private Boolean m_fileCopyThreadCountSet;
    private InArgument<Int32> m_fileCopyBufferSize;
    private InArgument<Int32> m_fileCopyThreadCount;
}

As you can see from our implementation, the code matches exactly (almost) what was outlined in the bulleted list. When this activity executes it will begin walking the source directory hierarchy and discover files and directories. While it’s performing this discovery operation it can have FileCopyThreadCount files being copied at any given time. Since this implementation uses a Queue<String> for the directories the files will be copied in breadth-first fashion. Another thing I would like to point out about this activity is its inherent ability to be canceled. Since everything is modeled in workflow, the framework will handle cancelation of our composite activity for us in between any of the workflow “statements” above. We also modeled our CopyFile activity to support cancelation in between reading from/writing to the file (in blocks of size FileCopyBufferSize), so if the activity is canceled we can gracefully and quickly respond to the request.

There are quite a few more improvements we could add to this activity to make it much more robust, such as the ability to specify behavior when the target file exists or provide the ability to retry a failed file copy operation up to some configurable amount of times. I will leave this as an exercise to the reader as it does nothing to further your knowledge on workflow.

Conclusion

Over the past 3 posts we have developed a much more efficient directory copy activity that may be reused in the workflow. Along the way we also developed CopyFile, ParallelItemScheduler<T>, ProducerConsumer<T>, and various other supporting activities. In the beginning we started out very low level as we required scheduling functionality that was not built into the framework. After that initial hump, however, you should notice that we very rarely dropped back to code and typically composed all of our logic using standard workflow activities. There are numerous benefits to building activities this way, one of which I have tried to harp on, such as:

Automatic support for cancelation by the framework (if a cancel requests comes in during a short running code activity then when it finishes the parent will automatically clean up)
We verified our lower level activities by reusing them in our own activity library
We have small modular pieces of workflow logic that is now reusable in other, similar activities (this will most likely not be the only producer-consumer model you will need to implement)

Hopefully as you read through this 3 part series you got a better feel for how to design activities as well as a taste of the more advanced behaviors you can model with Windows Workflow. I have attached a zip file that includes this activity library along with a sample executable that links to and utilizes the CopyDirectory activity that we just developed. Enjoy!

Windows Workflow 4.0 – CopyDirectory (Part 2 of 3)

2010-01-07T21:40:00Z

In my previous post I introduced you to NativeActivity and walked through the design and implementation of a class named ParallelItemScheduler<T>. In part 2 of this series we will be utilizing the aforementioned activity to build a new composed activity named ProducerConsumer<T>. Once we have completed this task we will have a framework for building our final activity, CopyDirectory.

Building Activities Through Composition

Unlike the previous activity, the ProducerConsumer<T> activity may be built using composition. This means that we will be using Activity as a sub-class and providing an implementation using initializer syntax and numerous built-in and custom activities that we have been working on. Before we start designing our activity, however, lets go back to the producer-consumer diagram from the previous post to make sure we get it right.

Parallel

Producer
- While continue-producing
  - Invoke producer to generate items
  - For each of the items produced, notify the consumer that a new item is available
- Notify the consumer to shut down
Consumer
- Wait for next item
- If consumer is available
  - Then
    - Invoke consumer to handle item
  - Else
    - Queue the item and wait for a consumer to complete

The pieces in bold are the configurable properties, so lets break down what we will need to provide and what workflow classes will support our implementation.

InArgument<Int32> ConsumerCount
- Allows users to specify how many concurrent consumers should be scheduled for handling items
ActivityAction<T> Consumer
- Allows users to specify custom consumer logic for processing items produced by the producer
Activity<Boolean> ContinueProducing
- Allows users to specify a custom condition to determine when to stop running the producer loop
ActivityFunc<ICollection<T>> Producer
- Specifies custom producer logic for feeding items into the consumer

What is ActivityFunc<T>?

Much like ActivityAction<T> is synonymous to the delegate type Action<T>, ActivityFunc<T> is synonymous to the delegate type Func<T>. Providing a hook for an ActivityFunc<> class provides a customizable hook for users of the activity to provide custom logic that returns a value and may or may not take parameters. In our particular scenario, we need to provide a custom mechanism for producing items to queue up for our consumer and do not require any arguments, so we use ActivityFunc<ICollection<T>> as our producer definition. As you will see in the implementation that follows, you can schedule ActivityAction/ActivityFunc objects using the InvokeAction/InvokeFunc activities in composition or by using the methods NativeActivityContext.ScheduleAction/NativeActivityContext.ScheduleFunc if you are scheduling from a native activity.

Implementing the ProducerConsumer

When using composition there is one major difference to how you build the activity vs. another type of activity. Rather than providing an Execute method to determine the logic, you provide what is called the activity Implementation, which is defined as:

protected virtual Func<Activity> Implementation { get; set; }

We developed a pattern internally that will be shown below, but essentially every activity that uses Activity as a base class has a private method called CreateBody() that returns the logic as workflow generated with initializer syntax. In the constructor we simply set the implementation to the result of an anonymous delegate that invokes this method. Keeping this in mind I have pasted the implementation of the activity below, with an explanation that follows.

public sealed class ProducerConsumer<T> : Activity
{
    public ProducerConsumer()
    {
        base.Implementation = () => CreateBody();
    }

    public InArgument<Int32> ConsumerCount
    {
        get;
        set;
    }

    public ActivityAction<T> Consumer
    {
        get;
        set;
    }

    public Activity<Boolean> ContinueProducing
    {
        get;
        set;
    }

    public ActivityFunc<ICollection<T>> Producer
    {
        get;
        set;
    }

    protected override void CacheMetadata(ActivityMetadata metadata)
    {
        base.CacheMetadata(metadata);

        if (Consumer == null)
        {
            metadata.AddValidationError(new ValidationError("Consumer must be set", false, "Consumer"));
        }

        if (ContinueProducing == null)
        {
            metadata.AddValidationError(new ValidationError("ContinueProducing condition must be set", false, "ContinueProducing"));
        }

        if (Producer == null)
        {
            metadata.AddValidationError(new ValidationError("Producer must be set", false, "Producer"));
        }
    }

    Activity CreateBody()
    {
        Variable<String> queueName = new Variable<String>();
        DelegateInArgument<T> itemToProduce = new DelegateInArgument<T>();
        Variable<ICollection<T>> itemsProduced = new Variable<ICollection<T>>();

        return new Sequence
        {
            Variables = 
            {
                queueName,
            },
            Activities =
            {
                new Assign<String>
                {
                    Value = new InArgument<String>(env => Guid.NewGuid().ToString("D")),
                    To = new OutArgument<String>(queueName),
                },
                new Parallel
                {
                    Branches = 
                    {
                        new ParallelItemScheduler<T>
                        {
                            QueueName = new InArgument<String>(queueName),
                            MaxConcurrency = new InArgument<Int32>(env => ConsumerCount.Get(env)),
                            Body = this.Consumer,
                        },
                        new TryCatch
                        {
                            Try = new While
                            {
                                Condition = this.ContinueProducing,
                                Body = new Sequence
                                {
                                    Variables =
                                    {
                                        itemsProduced,
                                    },
                                    Activities =
                                    {
                                        new InvokeFunc<ICollection<T>>
                                        {
                                            Func = this.Producer,
                                            Result = new OutArgument<ICollection<T>>(itemsProduced),
                                        },
                                        new If
                                        {
                                            Condition = new InArgument<Boolean>(env => itemsProduced.Get(env) != null && itemsProduced.Get(env).Count > 0),
                                            Then = new ForEach<T>
                                            {
                                                Values = new InArgument<IEnumerable<T>>(itemsProduced),
                                                Body = new ActivityAction<T>
                                                {
                                                    Argument = itemToProduce,
                                                    Handler = new ParallelItemScheduler<T>.Enqueue
                                                    {
                                                        QueueName = new InArgument<String>(queueName),
                                                        Value = new InArgument<T>(itemToProduce),
                                                    },
                                                },
                                            },
                                        },
                                    },
                                },
                            },
                            Finally = new ParallelItemScheduler<T>.Shutdown
                            {
                                QueueName = new InArgument<String>(queueName),
                            },
                        },
                    },
                },
            },
        };
    }
}

As you can see above, for the most part the workflow framework provides the building blocks for our implementation. If you compare the outline for the activity above with the construction of the activity logic, you should see that it lines up almost exactly. What we have done is create a parallel activity with two branches, one is the producer and the other is the consumer. The consumer thread simply waits for input and then schedules the provided consumer each time an available slot is available. The producer thread loops and continually calls the provided callback, pushing the produced items into the consumer for scheduling. Once the producer is finished, we have the entire block wrapped in a TryCatch so we can ensure to invoke the ParallelItemScheduler<T>.Shutdown activity.

For all intents and purposes, the queue name (really the bookmark name) simply has to be unique. In an attempt to keep things simple I have chosen to create a queue name by generating a new GUID. There are many other ways to generate queue names, but this was the easiest way to ensure there are not any conflicts. One important thing about the queue, however, is that the consumer is first in the parallel and the producer is second. This activity may not work the other way around since the consumer is what creates the bookmark that the producer will be resuming it. As it turns out, the built-in Parallel activity is not strictly parallel. What I mean by that is since the workflow scheduler works off of a single thread, the ability of concurrent work to occur is completely dependent on the child activities that it is invoking. If we did not design our ParallelItemScheduler<T> to use the bookmark and go idle, for instance, then the producer would never run since the scheduler would never have a chance to execute it while it’s blocked on the aforementioned activity. Using this knowledge, we can guarantee that the first branch in the parallel will invoke its first child BEFORE the second child is invoked, and so on down the list in sequential order. The parallelism occurs when an activity goes idle using bookmarks or AsyncCodeActivity, which allows the scheduler to move onto the next item in the queue while execution of the previous activity continues off of the workflow thread.

Note: See Nate Talbert's post on the WF 3.5 ParallelActivity and how it works for a more complete explanation of how parallelism works in a workflow instance.

Validation Errors

The previous post introduced one of the mechanisms for CacheMetadata, and this one introduces another use for overriding this method. Workflow validation occurs against the workflow definition and not against the runtime inputs, so all you can validate are things like the activity hierarchy, check for the existence of properties, and other things that belong to the activity definition and are not runtime-specific. What I have done here is add validation errors to the workflow if any properties required for property execution are not set. You can test the behavior in this scenario by trying to run this activity without one, two, or all three of the required properties set.

Conclusion

This part of the 3 part series introduced you to activity composition, activity functions, and basic activity validation. At this point in time we have all of the building blocks necessary to build more activities that follow the model of producer-consumer. As you probably guessed, the CopyDirectory activity that I will be going over in part 3 is one such activity. Given the proper inputs a directory copy operation that walks the hierarchy while simultaneously copying discovered files N at a time can provide large performance gains when network latency is large. Since we’re using workflow, we have not and will not need to make any use of synchronization primitives or do anything to handle cancelation of the operation! Hopefully, if not yet, by the next post you will start seeing the benefits of writing workflow in small and reusable modules rather than trying to implement everything in code.

Windows Workflow 4.0 – CopyDirectory (Part 1 of 3)

2010-01-07T20:04:00Z

In my last couple of posts I have lead you through some basic introduction to workflow as well as a real world example of how to implement a cancelable CopyFile activity. The next thing I would like to do is dive into a much more involved example utilizing the previous activity. The purpose of this post is to introduce the class NativeActivity, since we will need to implement a custom activity for driving the consumers. We will also be working toward using our producer-consumer activity to eventually develop a parallelizable CopyDirectory activity, which I plan to cover in a subsequent posts.

Producer-Consumer

The first thing I want to do is step back and determine how we might go about modeling this as a reusable model that may be extended through composition. Our basic structure will be something along the lines of the following:

Parallel

Producer
- While continue-producing
  - Invoke producer to generate items
  - For each of the items produced, notify the consumer that a new item is available
- Notify the consumer to shut down
Consumer
- Wait for next item
- If consumer is available
  - Then
    - Invoke consumer to handle item
  - Else
    - Queue the item and wait for a consumer to complete

Typically the coordination between the producer and the consumer would be handled through mutexes, monitors, or some other cross-thread synchronization mechanism. The nice thing about workflow, however, is that we do not need to worry about this synchronization between threads as long as we make sure that all access to the common objects are handled on the workflow thread. If you were worried by my earlier posts that a single thread is dedicated to running a workflow, perhaps now it will make more sense. What this allows us to do is model complex, multi-threaded operations without having to deal with thread synchronization issues! However, there is one critical piece of the puzzle that is not already done for us – modeling the consumer scheduling behavior across threads. In order to run the consumer scheduling in parallel to the producer we will need to dive into the wonderful world of NativeActivity and Bookmark.

Activity classes which derive from NativeActivity are typically attempting to provide some sort of custom scheduling logic that is not available through composition of existing framework activities. In our scenario, for instance, we need an activity that can:

Wait for items to be pushed into a queue (this is what we need the bookmark for)
Schedule some maximum number of handlers in parallel and ensure that maximal use of these handlers is obtained

The remainder of the post will be dedicated to walking through how to design this activity, using NativeActivity.

What is a Bookmark?

A bookmark, for lack of a better definition, is exactly what it sounds like. For those of you unfamiliar with the previous workflow framework, a bookmark is essentially like a piece of paper you put inside of a book to mark your place. When an activity creates a bookmark, it is creating a named “resumption point” so it may go dormant and other activities may “wake it up” by resuming the bookmark. If you are familiar with Workflow from the .NET 3.5 Framework, a bookmark provides similar functionality to a queue. A bookmark is how we will handle making the consumer scheduling thread go idle while waiting for items in the queue.

Implementing the ParallelItemScheduler

The parallel item scheduler is a generic and reusable class we will be developing to add to our slowly growing library of custom activities. This will be the class (well, the main class) that I will be discussing which derives from NativeActivity. As we already mentioned, the purpose of this activity is to invoke a user-defined callback for each item in the queue, with up to N running simultaneously. We will need some mechanism for pushing items from the producer into the consumer, which means we will need some sort of a queue (represented by a bookmark). We should also allow users of our activity to control how many consumers may run in parallel. Using this information we can come up with a stub for our activity.

public sealed class ParallelItemScheduler<T> : NativeActivity
{
    public ParallelItemScheduler()
    {
        m_maxConcurrency = new InArgument<Int32>(new VisualBasicValue<Int32>("1"));
    }

    public ActivityAction<T> Body
    {
        get;
        set;
    }

    public InArgument<Int32> MaxConcurrency
    {
        get
        {
            return m_maxConcurrency;
        }
        set
        {
            m_maxConcurrency = value;
            m_maxConcurrencySet = true;
        }
    }

    [RequiredArgument]
    [DefaultValue(null)]
    public InArgument<String> QueueName
    {
        get;
        set;
    }

    protected override Boolean CanInduceIdle
    {
        get
        {
            return true;
        }
    }

    [EditorBrowsable(EditorBrowsableState.Never)]
    public Boolean ShouldSerializeMaxConcurrency()
    {
        return m_maxConcurrencySet;
    }

    protected override void CacheMetadata(NativeActivityMetadata metadata)
    {
        base.CacheMetadata(metadata);
        metadata.AddImplementationVariable(m_itemQueue);
        metadata.AddImplementationVariable(m_hasCompleted);
        metadata.AddImplementationVariable(m_concurrencyCount);
    }

    protected override void Execute(NativeActivityContext context)
    {
        context.CreateBookmark(QueueName.Get(context), new BookmarkCallback(OnItemAdded), BookmarkOptions.MultipleResume);

        m_concurrencyCount.Set(context, 0);
        m_itemQueue.Set(context, new Queue<T>());
    }

    void OnItemAdded(NativeActivityContext context, Bookmark bookMark, Object data)
    {
        throw new NotImplementedException();
    }

    void OnChildCompleted(NativeActivityContext context, ActivityInstance activity)
    {
        throw new NotImplementedException();
    }

    public sealed class Enqueue : NativeActivity { … }

    public sealed class Shutdown : NativeActivity { … }

    private static Object s_shutdownItem = new Object();

    private Boolean m_maxConcurrencySet;
    private InArgument<Int32> m_maxConcurrency;
    private Variable<Queue<T>> m_itemQueue = new Variable<Queue<T>>("itemQueue");
    private Variable<Boolean> m_hasCompleted = new Variable<Boolean>("hasCompleted");
    private Variable<Int32> m_concurrencyCount = new Variable<Int32>("concurrencyCount");
}

Before I progress any more I would like to step back and break down some new pieces of this activity definition.

ActivityAction<T> is the workflow version of the framework delegate type Action<T>. This type allows you provide a pluggable mechanism for a single-argument callback, just like the framework counterpart (there are also ActivityAction<T1, T2…> types up to 16 arguments or so).
MaxConcurrency and its use of the Boolean field. Since the type of the property is InArgument<Int32> you cannot use the DefaultValueAttribute to specify the default and have the serializer automatically pick it up. Instead, if you want to provide a default value that should not be serialized to XAML then you need to do it the XAML way by keeping track of whether or not the property has been set and providing a method called ShouldSerializeXXXX (where XXXX is the name of the property) that returns a value indicating whether or not to serialize the property.
CacheMetadata and why it exists. Typically when designing activities you do not need to provide your own implementation since the default behavior uses reflection to discover the arguments, variables, activities, etc., of your activity. However, in certain scenarios like the one we are dealing with, you may have the requirement to create private variables for data storage (workflow calls these implementation variables). Since the reflection logic only picks up public properties of certain types we need to manually inform the runtime of these variables so we can use them. Why do we need variables? If you recall my previous post about data flow, C# fields and workflow variables are not the same – one is activity definition storage, the other is activity instance storage. We need the latter, since there may be multiple of this type of activity running within a workflow or the same definition may be shared across multiple workflow instances.
CanInduceIdle is a necessary override if you are writing an activity that can cause the workflow scheduler to become idle (this allows the runtime to perform certain validations to ensure correctness when composing activities together). In this case, since we are using a bookmark to “sleep,” we have the potential to induce an idle event on the workflow instance so we need to return true here.
The Execute method of our activity has an easy task. It simply needs to create the bookmark used to notify it of items (sort of like a mutex being signaled) and set up some initial state for the private variables. You’ll notice that we pass BookmarkOptions.MultipleResume when creating it, which means that we expect to have the bookmark resumed more than one time and we will handle cleaning it up on shutdown. As there is a bookmark active for our activity the workflow runtime is smart enough to leave this activity running even when the Execute method returns, so by doing this we have effectively gone to sleep waiting for input.

The basic logic of OnItemAdded, which is invoked when an item is placed into our bookmark queue, is to store the item into our implementation variable and then determine if we have any available slots for scheduling. While we are not at our maximum consumer count we should schedule a new consumer for each item that exists in the queue. Once we have finished scheduling items, we store the current concurrency count and return. It should be noted that the call to NativeActivityContext.ScheduleAction here does not actually begin execution of the action – it merely places an entry into the workflow instance’s scheduler queue and returns. The only other thing we can do in this method is remove the bookmark if the producer has signaled completion by queuing our “shutdown item.”

void OnItemAdded(NativeActivityContext context, Bookmark bookMark, Object data)
{
    if (m_hasCompleted.Get(context))
    {
        return;
    }

    Int32 concurrencyCount = m_concurrencyCount.Get(context);
    Int32 maxConcurrentCount = MaxConcurrency.Get(context);

    if (data is T)
    {
        Queue<T> itemQueue = m_itemQueue.Get(context);
        itemQueue.Enqueue((T)data);

        while (itemQueue.Count > 0 && concurrencyCount < maxConcurrentCount)
        {
            concurrencyCount++;
            context.ScheduleAction<T>(Body, itemQueue.Dequeue(), new CompletionCallback(OnChildCompleted));
        }

        m_concurrencyCount.Set(context, concurrencyCount);
    }
    else if (Object.ReferenceEquals(data, s_shutdownItem))
    {
        context.RemoveBookmark(QueueName.Get(context));
    }
}

The last order of business for this particular activity is the OnChildCompleted callback which is hooked above. This method is invoked when a child completes execution, either successfully or via cancelation. Much like the previous method this also handles consumer scheduling. As consumers complete we need to see if there are any more items which may be scheduled, and if there are we should schedule the new item using the slot of the consumer that just finished. The major difference between this method and OnItemAdded is the handling of activity cancelation. If a child activity is canceled and we have a pending cancellation request then we mark ourselves complete by storing it into our private variable and then mark the activity itself canceled via NativeActivityContext.MarkCanceled. Once we have done this once, as other children are canceled or complete they will see that the activity has already completed and immediately return.

void OnChildCompleted(NativeActivityContext context, ActivityInstance activity)
{
    if (m_hasCompleted.Get(context))
    {
        return;
    }

    if (activity.State != ActivityInstanceState.Closed && context.IsCancellationRequested)
    {
        m_hasCompleted.Set(context, true);
        context.MarkCanceled();
    }
    else
    {
        Int32 concurrencyCount = m_concurrencyCount.Get(context);
        Int32 maxConcurrencyCount = this.MaxConcurrency.Get(context);

        // Subtract one since an activity just finished
        concurrencyCount--;

        Queue<T> itemQueue = m_itemQueue.Get(context);
        if (concurrencyCount < maxConcurrencyCount && itemQueue.Count > 0)
        {
            concurrencyCount++;
            context.ScheduleAction<T>(Body, itemQueue.Dequeue(), new CompletionCallback(OnChildCompleted));
        }

        m_concurrencyCount.Set(context, concurrencyCount);
    }
}

The final classes that have not been shown here are ParallelItemScheduler<T>.Enqueue and ParallelItemScheduler<T>.Shutdown. The former simply resumes the bookmark with the specified data and the latter queues the item stored in s_shutdownItem so the call to Object.ReferenceEquals will succeed and remove the bookmark.

Conclusion

Today we have learned how to write custom scheduling logic by extending NativeActivity. We designed a custom class, ParallelItemScheduler<T>, that may be used in a workflow to model a consumer scheduler which schedules N concurrent consumers. The consumer is an attachable ActivityAction<T> class that is pluggable by consumers of the activity, and it is generic so highly reusable in a variety of scenarios. Although I would have liked to cover it all in a single post, this one is already becoming quite long and I would like to provide the ability to allow this information to soak in before continuing onto the next piece. Once we design our ProducerConsumer<T> activity you will hopefully gain a better understanding of how this activity is intended to be used. Until then, enjoy your newfound knowledge of the workflow framework and happy coding!

Windows Workflow 4.0 - CopyFile

2010-01-04T20:50:00Z

It has recently come to my attention that there is unrest regarding our inclusion of a CopyDirectory activity without a CopyFile activity. Therefore, I decided to take this opportunity to introduce everyone to the sub-class AsyncCodeActivity and how you can utilize it with a real world example – a cancelable file copy operation. As you may recall from my previous workflow posts, CodeActivity does provide a mechanism for cancelation while the AsyncCodeActivity does facilitate this need. If you are not familiar with asynchronous programming in .NET then you should probably read the following article to brush up before continuing.

.NET API Overview

The .NET API for System.IO.File does not provide a way to asynchronously copy a file, so we will need to provide our own mechanism using the asynchronous APIs provided on System.IO.FileStream (BeginRead/BeginWrite and EndRead/EndWrite respectively). I have provided the documentation for the aforementioned APIs below to aid in development of our activity.

public IAsyncResult BeginRead(Byte[] array, Int32 offset, Int32 numBytes, AsyncCallback userCallback, Object stateObject)
public Int32 EndRead(IAsyncResult asyncResult)
public IAsyncResult BeginWrite(Byte[] array, Int32 offset, Int32 numBytes, AsyncCallback userCallback, Object stateObject)
public void EndWrite(IAsyncResult asyncResult)

Just so we’re all on the same page, the parameters serve the following purpose:

array – The array of bytes that should be written to the file stream or a user allocated buffer in which to place bytes read from the file stream
offset – The offset in the array that reading or writing should occur
numBytes – The number of bytes from the offset that should be read or written

Also, it’s probably important to mention that the return value from EndRead returns the number of bytes that were actually read from the stream, which may be equal to or less than the range provided by (numBytes – offset). When the return value is less than the range provided to BeginRead you are at the end of the file. Now that we have this basic introduction out of the way, lets see how this all fits into the AsyncCodeActivity framework.

AsyncCodeActivity API Overview

The workflow framework provides the base classes, AsyncCodeActivity/AsyncCodeActivity<T>, which we will be using in this post. This activity provides 3 important methods for overriding that I have broken down.

IAsyncResult BeginExecute(AsyncCodeActivityContext context, AsyncCallback callback, Object asyncState)
- The synchronous entry point at which time all values required for asynchronous execution should be extracted using the provided context and stored into a custom result or state.
void EndExecute(AsyncCodeActivityContext context, IAsyncResult result)
- The synchronous exit point at which time all values for the workflow, such as OutArgument<T> and InOutArgument<T>, should be set using the provided context.
void Cancel(AsyncCodeActivityContext context)
- The synchronous entry point invoked from workflow to begin the cancelation process.

In the next section we will be diving a little deeper into how each of these may be implemented in a real world activity.

Writing the CopyFile Activity

The first thing we should do when designing out custom activity is to determine the type of flexibility we would like to provide to consumers of our activity. At the very least, copying a file will require a Source and a Target. This is where we will start, but we will continue evaluating other values we may like to promote to configurable properties. So, to begin we have the following activity outline:

public sealed class CopyFile : AsyncCodeActivity
{
    [RequiredArgument]
    public InArgument<String> Source
    {
        get;
        set;
    }

    [RequiredArgument]
    public InArgument<String> Target
    {
        get;
        set;
    }

    protected override IAsyncResult BeginExecute(AsyncCodeActivityContext context, AsyncCallback callback, Object state)
    {
        throw new NotImplementedException();
    }

    protected override void Cancel(AsyncCodeActivityContext context)
    {
        base.Cancel(context);
    }

    protected override void EndExecute(AsyncCodeActivityContext context, IAsyncResult result)
    {
        throw new NotImplementedException();
    }
}

Note: For the remainder of this section we will be investigating the implementation of the activity and simply refer to the individual method we are currently writing.

The first caveat I would like to point out is that the BeginRead API takes a buffer and an offset/length to determine how much data to read. Since there is no possible way we can know up-front how much data we will be streaming from the file, we will most likely be making multiple calls back into BeginRead each time we have copied the pending buffer into the target location. On top of this, when the callback provided to BeginExecute is invoked the workflow framework queues an operation on the workflow execution thread to invoke the EndExecute method – once this method completes the activity has completed execution. With this knowledge, we know that we will need to provide our own callback(s) to BeginRead/BeginWrite and only invoke the callback provided by the workflow framework once the entire copy operation has completed. The other important caveat you may have noticed is that there is no way to cancel an ongoing call to BeginRead/BeginWrite, as the built-in IAsyncResult interface does not provide a mechanism to get this feedback to the asynchronous operation. So, in order to properly support cancellation we should just read/write in a small enough buffer such that we can check a flag on our asynchronous copy state and stop the process once the ongoing read or write operation completes.

So, in summary, we will follow a pattern similar to the following:

Set up our async state in BeginExecute and start the first read operation by invoking FileStream.BeginRead
When the first read operation completes, if we have not been cancelled then begin the first write operation using the buffer from (1)
When the write operation completes, if we have not been cancelled begin the next read operation if we are not at the end of the file. If we are at the end of the file, invoke the callback provided by the workflow framework to trigger the EndExecute method.

I realize that this was a lot of information to ingest but hopefully it will start making sense as we start writing the method implementations.

Asynchronous State

The first thing we will need to define is our asynchronous state, which I have included for reference below. As it turns out, all we need is the original callback and state, a buffer, and the source and target file streams opened with the appropriate permissions and flags (some of the fields required by IAsyncResult have been omitted for space).

sealed class CopyFileAsyncResult : IAsyncResult
{
    public Exception Exception { get; set; }
    public Boolean Canceled { get; set; }
    public Byte[] Buffer { get; set; }
    public FileStream ReadStream { get; set; }
    public FileStream WriteStream { get; set; }
}

Implementation

Now that we have defined the data we will need to retain for our operation, we can start implementing the logic of our activity. Using steps 1-3 above, we come to something similar to the following:

protected override IAsyncResult BeginExecute(AsyncCodeActivityContext context, AsyncCallback callback, Object state)
{
    String source = Path.GetFullPath(Source.Get(context));
    String target = Path.GetFullPath(Target.Get(context));

    Int32 bufferSize = BufferSize.Get(context);

    CopyFileAsyncResult copyFileResult = new CopyFileAsyncResult
    {
        AsyncState = state,
        AsyncCallback = callback,
        Buffer = new Byte[bufferSize],
        ReadStream = new FileStream(source, FileMode.Open, FileAccess.Read, FileShare.Read, bufferSize, true),
        WriteStream = new FileStream(target, FileMode.CreateNew, FileAccess.Write, FileShare.None, bufferSize, true),
    };

    context.UserState = copyFileResult;
    copyFileResult.ReadStream.BeginRead(copyFileResult.Buffer,
                                        0,
                                        copyFileResult.Buffer.Length,
                                        new AsyncCallback(OnEndRead),
                                        copyFileResult);

    return copyFileResult;
}

protected override void Cancel(AsyncCodeActivityContext context)
{
    ((CopyFileAsyncResult)context.UserState).Canceled = true;
}

protected override void EndExecute(AsyncCodeActivityContext context, IAsyncResult result)
{
    CopyFileAsyncResult copyFileResult = result as CopyFileAsyncResult;
    if (copyFileResult == null)
    {
        return;
    }

    if (copyFileResult.AsyncWaitHandle != null)
    {
        ((IDisposable)copyFileResult.AsyncWaitHandle).Dispose();
    }

    if (copyFileResult.Exception != null)
    {
        throw copyFileResult.Exception;
    }
}

static void OnEndRead(IAsyncResult result)
{
    CopyFileAsyncResult copyFileResult = result.AsyncState as CopyFileAsyncResult;
    if (copyFileResult == null)
    {
        return;
    }

    Int32 bytesRead = 0;

    try
    {
        bytesRead = copyFileResult.ReadStream.EndRead(result);
    }
    catch (Exception ex)
    {
        copyFileResult.Exception = ex;
        OnCopyCompleted(copyFileResult);
    }

    if (copyFileResult.Canceled || bytesRead == 0)
    {
        OnCopyCompleted(copyFileResult);
    }
    else
    {
        // Start the next write operation using the current write offset and the bytes read.
        copyFileResult.WriteStream.BeginWrite(copyFileResult.Buffer,
                                              0,
                                              bytesRead,
                                              new AsyncCallback(OnEndWrite),
                                              copyFileResult);
    }
}

static void OnEndWrite(IAsyncResult result)
{
    CopyFileAsyncResult copyFileResult = result.AsyncState as CopyFileAsyncResult;
    if (copyFileResult == null)
    {
        return;
    }

    try
    {
        copyFileResult.WriteStream.EndWrite(result);

        if (copyFileResult.Canceled ||
            (copyFileResult.WriteStream.Length == copyFileResult.ReadStream.Length &&
             copyFileResult.ReadStream.Position == copyFileResult.ReadStream.Length))
        {
            OnCopyCompleted(copyFileResult);
        }
        else 
        {
            copyFileResult.ReadStream.BeginRead(copyFileResult.Buffer, 
                                                0,
                                                copyFileResult.Buffer.Length, 
                                                new AsyncCallback(OnEndRead), 
                                                copyFileResult);
        }
    }
    catch (Exception ex)
    {
        copyFileResult.Exception = ex;
        OnCopyCompleted(copyFileResult);
    }
}

static void OnCopyCompleted(CopyFileAsyncResult copyFileResult)
{
    copyFileResult.ReadStream.Dispose();
    copyFileResult.WriteStream.Dispose();

    copyFileResult.AsyncWaitHandle.Set();

    if (copyFileResult.AsyncCallback != null)
    {
        copyFileResult.AsyncCallback(copyFileResult);
    }
}

As you can see from the code above, essentially we loop back and forth between OnEndRead and OnEndWrite allowing the framework to handle threading for us. We determine the process is completed in a few ways – an exception is thrown, a cancellation request is made, or the read stream and write stream have equal lengths and our position in the read stream is at the end (as seen in OnEndWrite). Asynchronous APIs should not throw exceptions on the background thread, including callbacks, since an unhandled exception on a background thread may have the unintended consequences of shutting down the application domain or the entire process. What should occur instead, as you can see by the try/catch blocks in OnEndRead and OnEndWrite, is that an exception simply ends the read/write loop and completes the operation. Once the EndExecute method is invoked by the framework we check for an exception that was saved in our state, and if one exists we throw it on the foreground thread provided by the workflow runtime. Something else that’s important to point out is that we store our state into the AsyncCodeActivityContext.UserState property, which is how we retrieve it when the activity is canceled (see the Cancel method above).

For those of you paying close attention you may be noticed the introduction of an InArgument<Int32> BufferSize which is used in the BeginExecute method to construct our in-memory buffer. Although I didn’t cover this initially I wanted to provide this functionality as a configurable property so we do not lose flexibility provided by the standard file copy APIs.

Conclusion

Although this article is quite wordy, hopefully it introduced the workflow APIs and provided an understandable and realistic example of how to utilize asynchronous APIs and cancellation in the workflow framework. Since this activity is asynchronous, it may be used as a branch of a Parallel or the body of a ParallelForEach<T> for copying multiple files in parallel to more efficiently saturate the network when high latency is involved. In fact, I plan on illustrating how you might use this activity to build a more efficient CopyDirectory than what we ship in the box for copying a configurable number of files in parallel. However, this will involve a more complex implementation since the standard ParallelForEach<T> provides no mechanism for capping the parallel branches at a maximum number, and providing this functionality requires us to derive from NativeActivity.

I hope this article serves as a starting point for the development of more complex activities, perhaps using this as a building block to develop that parallel directory copy I referred to above!

I'm a ramblin' wreck from Georgia Tech and a helluva engineer!

2009-12-06T08:47:00Z

This particular post has nothing to do with technology. I'm an avid college football fan and have always cheered for my alma mater, the Georgia Tech Yellow Jackets. Tonight, for the first time since I have been actively following them, we beat Clemson for the 2nd time this season (39-34) and earned a berth in the BCS Orange Bowl!!

Not to downplay Clemson, however, as they put up a really good fight both times and it came down to who had the ball last. The first game was equally amazing to watch and ended with a field goal to beat the Tigers 30-27. This game we won on a touchdown by Jonathan Dwyer! He's the man! Speaking of the man, I think C. J. Spiller deserves to go to New York and be in the running for the Heisman. Amazing athlete who I'm sure will do well in the NFL.

Ok, I'm done gloating ... hopefully this will earn the ACC more respect. Now on to the Orange Bowl!

One more thing .. of course I can't leave Paul Johnson hanging high and dry in recognition. He has done an amazing job as head coach of GT, taking a team that went 7-5 to 9-4 and now 11-2 with a BCS berth, first appearance in the Orange Bowl since the early 60's!

Introduction to Windows Workflow 4.0

2009-12-04T18:54:00Z

Note: I apologize in advance if this first post seems unorganized. There is a lot of material to cover and I couldn’t find a really good way to order the topics. Hang in there and hopefully it will start making sense.

In order to help everyone get up to speed with build customization, I wanted to take a few posts to share our collective knowledge on the Workflow framework in .NET 4.0. In this first post I plan to tackle the different base classes that may be used when designing your own custom activities, along with typical usage scenarios. Other topics, such as extensions, tracking, and more advanced customization will be covered later. I will also be keeping an eye on comments to help point me in the right direction. So without further adieu, on with the code!

The Core Activity Classes

There are 4 base classes that you should become very familiar with in the framework. Each class has a corresponding generic version, which informs the workflow runtime that it returns a result of the specified type (more on this later). I have listed out the 4 base classes below, along with general information on when you might choose to utilize each one.

System.Activities.Activity

This class serves as the base class for all of the other activities in included in the framework, as well as the class you will be extending for activities composed in code. Most of our activities in TFS 2010 are derived from this activity or Activity<T>, which is simply an activity that returns a result of type T.

System.Activities.CodeActivity

CodeActivity, along with CodeActivity<T>, provide a simple mechanism for providing the implementation of an activity as .NET code. Classes that utilize this as a base class should not perform long running operations as there is no way to cancel the execution of this activity.

System.Activities.AsyncCodeActivity

AsyncCodeActivity and AsyncCodeActivity<T> are very similar to the CodeActivity classes but are intended to perform long running operations. If you need perform I/O or would like to provide the ability to cancel the execution of your activity, you should derive from the asynchronous variety. Most of the code activities in TFS 2010 utilize the AsyncCodeActivity classes so a build may be stopped cleanly during an operation.

System.Activities.NativeActivity

This particular activity, in conjunction with NativeActivity<T>, are not intended for heavy use. In fact, the workflow framework even makes somewhat limited use of this activity as a base class. Typically you should only use this activity if you would like to define your own scheduling behavior or need to interact with bookmarks and external services.

The Core Data Classes

System.Activities.Variable

A variable in represents the same concept as it does in code when you declare an integer or string. The purpose of variables is to act as a data storage mechanism within a particular visibility scope. Much like you cannot declare two variables of the name ‘foo’ in C# without a compiler error, you cannot successfully run a workflow that attempts to define two variables in the same scope with the same name.

System.Activities.Argument

An argument represents the flow of data, and does not actually have the ability to store anything (the actual storage mechanism for an argument is typically a variable or top-level input to the workflow). There are 3 important sub-classes of argument that you will need to become familiar with:

System.Activities.InArgument<T>
- The value is meant to flow into the method. Much like a pointer in C, the pointer itself cannot be changed but if a complex object is passed as an argument then fields of that object may be changed.
System.Activities.OutArgument<T>
- The value is meant to flow out of the method. This argument type is synonymous to the ‘out’ keyword in C#.
System.Activities.InOutArgument<T>
- The value is meant to flow in and possibly be modified as an output. This argument type is synonymous to the ‘ref’ keyword in C#.

Designing Your First Activity

For simplicity, we’re going to start with the age-old example of “Hello, World!” but accomplish the task with workflow. For this introduction we just need to load up visual studio and create a new C# ‘Console Application’ project. Normally you would select a ‘Workflow Console Application’ for the project type, but I’d rather go through the steps manually to hopefully aid in your understanding of the framework. Once you have the project created, add a reference to System.Activities using the ‘.NET’ tab. Now we need to create our first activity for interacting with the console for output. Create a new C# class called ‘WriteLine’ and paste the following code into the file.

using System;
using System.Activities;
using System.ComponentModel;
using System.IO;

namespace HelloWorkflow
{
    public sealed class WriteLine : CodeActivity
    {
        [Browsable(true)]
        [RequiredArgument]
        [DefaultValue(null)]
        public InArgument<TextWriter> TextWriter
        {
            get;
            set;
        }

        [Browsable(true)]
        [RequiredArgument]
        [DefaultValue(null)]
        public InArgument<String> Text
        {
            get;
            set;
        }

        protected override void Execute(CodeActivityContext context)
        {
            String text = Text.Get(context);
            TextWriter textWriter = TextWriter.Get(context);

            if (textWriter == null)
            {
                throw new ArgumentException("The provided text writer is invalid");
            }

            textWriter.WriteLine(text);
        }
    }
}

What we have done is created an activity, called WriteLine, that takes a TextWriter and some Text as arguments and writes the text to the writer. I know this may seem overly simple but it works pretty well for explaining the general concepts of data flow. If you think about this a little bit, you may notice that this mimics code quite a bit! For instance, we could write this using C# instead in the following way:

public static void WriteLine(TextWriter textWriter, String text)
{
    if (textWriter == null)
    {
        throw new ArgumentNullException("textWriter");
    }

    textWriter.WriteLine(text);
}

Taking a moment to analyze the correlation between these two approaches, we notice a couple of similarities. First, the CodeActivity.Execute(CodeActivityContext) method is nothing more than the private implementation of the C# method. Second, the input arguments are nothing more than properties on the activity of type System.Activities.InArgument<T>. Although there isn’t always a direct 1-to-1 mapping as illustrated by this simple example, you should approach your development of activities in much the same way. Activities are meant to be modular, reusable pieces of code that may be composed into more complex functionality, exactly like functions/methods in written code.

Now that we have our activity, let’s see how we can use it. Go back to your Program.cs file, or where your Main method exists, and make it look similar to the following.

static void Main(String[] args)
{
    WorkflowInvoker invoker = new WorkflowInvoker(new WriteLine());
    
    Dictionary<String, Object> inputs = new Dictionary<String, Object>();
    inputs.Add("TextWriter", Console.Out);
    inputs.Add("Text", "Hello, Workflow!");

    invoker.Invoke(inputs);
}

If you now run this program you should see the text “Hello, Workflow!” output to the standard output stream of the console window! Ok, I know that wasn’t all that exciting, but you always need to start somewhere to understand the basic concepts before we can move on to the better stuff. Let’s take a second to go through the main method and explain what it does. First we allocate a WorkflowInvoker object, which is very useful for running simple workflows (most hosting environments, including TFS Build 2010, will most likely utilize the WorkflowApplication class instead), specifying the activity definition we would like to use. Next, we set up some inputs which map to the argument names on our activity, which define the execution environment for the runtime. Last, we invoke the workflow which will execute our activity with the provided input environment.

Activity Definition vs. Activity Instance

One particularly interesting area I would like to point out is the separation of the activity definition from the activity instance. When you load a workflow for execution and provide an instance of a C# class, in this case the WriteLine activity, you are actually supplying what is known as the activity definition. The definition defines the inputs, outputs, control flow, and everything else you would expect that remains static, much like a class definition in code. Each time the activity is executed it reuses the same C# class instance of the definition – once again, each time the activity is executed it reuses the same C# class instance of the definition. Why did I say this twice? I want to make it extremely clear why it’s important to differentiate between properties that belong to the definition and properties that belong to the individual execution instance. Instance-level storage is accomplished with the mechanisms provided by the workflow runtime (Variable<T>, InArgument<T>, InOutArgument<T>, etc), while definition-level storage is accomplished via standard class properties. If you take a look back at our WriteLine activity, the execute method obtains the values for the inputs using the provided CodeActivityContext. Along with providing mechanisms to interact with the tracking participant and the ability to get extensions, the activity context also provides the storage for the current instance environment visible to the activity. So, if you want to retrieve the value for one of the inputs to the activity you must have access to the activity context to retrieve the value. If you would like to experiment with this more, try adding an integer as a regular field or property to the class and increment it each time the workflow is run.

Wrapping It Up

Reading this post should introduce you to basic concepts of workflow, how data flows in a workflow, and how to get started writing your own activity. I plan on continuing this series with regular updates on progressively more advanced and cumulative topics, so if you’re looking to make use of this framework in your own programs or are simply attempting to extend your build process keep an eye out! My current plan is to cover AsyncCodeActivity and composition through Activity in the following posts, but feel free to steer me in a different direction.

TFS 2010 – Customizing Build Information (Part 2)

2009-11-19T19:18:00Z

My previous post introduced the basics of how the tracking participant may be used to create custom build information through the activity context. However, as pointed out at the end of the article, the best practice for creating custom activities in Workflow 4.0 is to do so through composition, meaning that you should derive from Activity or Activity<T>. In compliance with this model we supply activities that allow you to perform the same custom information creation through activity composition rather than from code. The intention of this article is to lead you through how you might make this type of functionality available, as well as introduce you to composing custom build information into your own build processes.

Tracking Custom Build Information using Activity Composition

As you may recall, we previously introduced a custom tracking record, BuildInformationRecord<T>, for sending instructions to the tracking participant to create custom information. In order to keep things generic, we want to allow users of our activity to have the same flexibility with our new activity that they have when interacting directly with the activity context. Since our activity will need to perform the interaction with the activity context on behalf of the workflow writer, we need to create a custom CodeActivity, which may be defined similarly to the class below.

public sealed class WriteBuildInformation<T> : CodeActivity
{
    public WriteBuildInformation()
    {
    }

    [RequiredArgument]
    [Browsable(true)]
    [DesignerSerializationVisibility(DesignerSerializationVisibility.Visible)]
    public InArgument<T> Value { get; set; }

    protected override void Execute(CodeActivityContext context)
    {
        context.Track(new BuildInformationRecord<T>() { Value = Value.Get(context) });
    }
}

As you can see, our activity is simply a thin wrapper around the code we previously explored. However, there are a couple of advantages I’d like to point out when using this approach over the strict code approach.

Since execution is very short lived, the tracking participant gets a chance to receive the record and take action once the method completes.
Gives consumers of the activity a chance to incorporate custom information by building activities through composition in either XAML or code.

This activity should only be considered a utility activity that serves as the base for special purpose activities. The reason I propose limited use of this activity directly is because the XAML for it can get relatively tedious fairly quickly. For example, if we step back and model our example of writing a build message in XAML using this activity, we will end up with something very similar to the XAML below.

<my:WriteBuildInformation x:TypeArguments="my:BuildMessage" xmlns:my="http://mycustomschema.company.com" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml">
  <my:WriteBuildInformation.Value>
    [New BuildMessage() With {.Importance = BuildMessageImportance.Normal, .Message = "This is a custom message" }]
  </my:WriteBuildInformation.Value>
</my:WriteBuildInformation>

Judging by the XAML, the verbosity could be greatly reduced if we didn’t need to specify the x:TypeArguments or construct the Value parameter with a visual basic expression; this is where activity composition comes in handy. We can provide any number of specialized wrapper activities that simplify the usage of this activity for particular information types. Once again, for the sake of consistency, we will revisit the BuildMessage for an example. What we would ultimately like to provide for users of our activity library is a single, specialized activity for writing build messages. Users should be able to set the Importance and the Message as arguments to the activity, just as they would when constructing the XAML above. Given this small set of requirements, we might end up with an activity that looks similar to the following:

public sealed class WriteBuildMessage : Activity
{
    public WriteBuildMessage()
    {
        base.Implementation = () => CreateBody();
    }

    [RequiredArgument]
    [Browsable(true)]
    [DesignerSerializationVisibility(DesignerSerializationVisibility.Visible)]
    public InArgument<String> Message { get; set; }

    [Browsable(true)]
    [DesignerSerializationVisibility(DesignerSerializationVisibility.Visible)]
    public InArgument<BuildMessageImportance> Importance { get; set; }
 
    Activity CreateBody()
    {
        return new WriteBuildInformation<BuildMessage>
        {
            Value = new InArgument<BuildMessage>(ctx => new BuildMessage() { Message = Message.Get(ctx), Importance = Importance.Get(ctx) }),
        };
    }
}

There a few things I would like to point out about the code above, just in case not everyone is familiar with lambda syntax and activity composition in general.

The single line in the constructor is one of the most important lines. The way composed activities work is you provide a Func<Activity>, exposed as the property Implementation, that returns the private implementation of your activity (synonymous to the implementation of a method or function in code). If you forget to set this, the activity will not produce an error and will produce no output.
The InArgument<T> type has a few constructors. In this particular scenario we want to construct an object at runtime based on the incoming arguments. The syntax we must use in this case is the constructor that takes a Func<ActivityContext, T> so we can utilize the activity context handed to us by the runtime to extract the values of the arguments for our BuildMessage. Since Workflow 4.0 is very new and still in beta, I plan on covering general topics regarding activity design and best practices we learned through our use of the framework in TFS 2010 in coming posts. If you don’t quite understand what I’m talking about just be patient, for soon you will be a Workflow Master (probably not, but you’ll understand it better than you do now)!

Now that we have our specialized activity for writing build messages, we can express the equivalent XAML we saw earlier with the following:

<my:WriteBuildMessage xmlns:my="http://mycustomschema.company.com" Importance="[BuildMessageImportance.Normal]" Message="This is a custom message" />

As you can see the syntax is much more concise than the previous example. It also takes the burden off the user of your activity to create an instance of the appropriate type and use initializer-syntax to set the property values. Although this simplifies the XAML you should probably use good judgment when writing these specialized activities to keep your library to a reasonable size. You will find the following activities exposed in the library that shipped with TFS 2010 Beta 2:

WriteBuildError
WriteBuildMessage
WriteBuildWarning

Using these activities you can, you guessed it, write messages, warnings, and errors. These particular information types are special in that they are well understood by our build report, however, and will show up with warning or error icons next to the message automatically when viewing the build log. Jason Prickett should be making a follow-up post soon describing how you can extend the build report to display your custom information. I’ll be sure to notify everyone once this happens so we can bring all of the customizations together and show off the entire scenario.

After writing these past 2 articles I have come to the realization that it would probably be very helpful to go over Workflow 4.0 in general so everyone can get up to speed on these topics. My next few posts will focus on development of activities and design practices with the 4.0 APIs, so keep reading for more in-depth explanations of the underlying framework.

TFS 2010 – Customizing Build Information (Part 1)

2009-11-17T23:10:00Z

Before reading this post, you should take a moment to read Aaron Hallberg’s post on build process customization in TFS 2010 Beta 1. Now that you fully understand custom activities in Windows Workflow 4.0 (just kidding), I would like to focus on how we have made it much easier to customize the information created and stored along with the build. There are currently multiple mechanisms for getting additional information into the build process, made possible by some activities, extension methods, and integration with the tracking participant (synonymous to the MSBuild logger for those familiar with that framework). This post will focus on using the activity context to create custom information from code.

Build Information (VERY quick primer)

Build information is simply a hierarchical data store that is extensible. Each node in the hierarchy has a Type and a set of Fields (name/value pairs as strings). All information created since TFS 2008, including task logging, build steps, and compilation details, are stored using this mechanism. Since its introduction into the API this hierarchy has always been extensible, but with the introduction of Windows Workflow 4.0 we have tried to make it easier.

Tracking Custom Build Information using the ActivityContext

Tracking in Windows Workflow is the mechanism that a workflow may use to communicate information to the tracking participant(s). For each workflow run in TFS Build 2010 we hook in a custom implementation of a tracking participant, the BuildTrackingParticipant. Along with tracking general activity in the workflow, such as activity execution, we also provide a way to inject custom build information during the workflow in a safe manner by interaction with the tracking participant. In order to facilitate this we have provided a custom tracking record defined as follows (certain classes in the hierarchy have been omitted):

public sealed class BuildInformationRecord<T>
{
    public T Value { get; set; }
}

This specialized record should be used for writing all custom build information during a workflow. The tracking participant creates information nodes from these records by analyzing the enclosed type via designer-based reflection and converts the object into an IDictionary<String, String> for storage. If complete customization of the data conversion is desired you may provide a custom TypeConverter for T which can take the object and convert it to an IDictionary<String, String> (in most instances this level of customization will not be necessary, but the hook exists if the need arises). The information node type name is extracted from the name of the managed type (e.g. typeof(T).Name), so there is no need to provide this explicitly. This mechanism of writing build information is how all built-in information nodes are written to the build log, excluding the activity tracking nodes themselves.

Illustration by example is typically the easiest way to understand, so next I will describe how we write information nodes of type BuildMessage to the log using this tracking record. The first thing we will need to do is define a class that describes the data we will be storing for a message: Message and Importance. You will find the definition of this class below.

public sealed class BuildMessage
{
    public String Message { get; set; }
    public BuildMessageImportance Importance { get; set; }
}

As you can see, all we had to do is define a class with a type name and set of properties matching the information type we would like to create. Now, in order to inject a BuildMessage information node into the build log all we need to do is call the appropriate CodeActivityContext.Track(CustomTrackingRecord) or NativeActivityContext.Track(CustomTrackingRecord) method from code, depending on which activity type your custom activity derives. For instance, if you would like to track a message from a code activity you could do the following:

public sealed class WriteMessage : CodeActivity
{
    protected override void Execute(CodeActivityContext context)
    {
        context.Track(new BuildInformationRecord<BuildMessage>()
        {
            Value = new BuildMessage()
            {
                Importance = BuildMessageImportance.Normal,
                Message = "This is a custom information node",
            },
        });
    }
}

When the tracking participant receives this record, it will essentially perform the following steps:

Find the closest activity tracking node in scope (more details on how this is managed in a later post), using the root node of the current IBuildDetail if none is found.
Create a new IBuildInformationNode as a child of the node from (1) with a type name of “BuildMessage” (derived from typeof(T).Name as described earlier).
Retrieve the TypeDescriptor for T. If the type converter can convert directly to an IDictionary<String, String>, it simply allows the converter to do the heavy lifting. If the converter cannot perform this conversion, an information field will be created from each PropertyDescriptor retrieved from a call to TypeDescriptor.GetProperties(Object), converting each value to a String using the converter of the PropertyDescriptor. In most cases this will simply invoke the ToString() method of the type. However, we provide explicit converter implementations which cannot be overridden for DateTime and Byte[] properties to ensure appropriate and consistent conversions of these data types.

While this mechanism works fine there is a pretty important caveat with tracking custom information from within a CodeActivity that you should be aware of. With the exception of the AsyncCodeActivity, every operation in a single workflow instance is executed on the same thread, including tracking. So, if you have a long-running execute method where you are attempting to track progress by logging multiple messages you will be surprised when nothing shows up until the activity completes. This behavior is due to the single thread; since your execute method is using the thread the tracking records will not be flushed to the tracking participant until your method returns. As a general rule of thumb, we have tried to keep away from creating long-running and complex code activities in favor of producing much smaller, more modular behavior that we then build into an activity through composition. Most of our activities in the shipping library derive from Activity or Activity<T>, with workflow driving the majority of the logical constructs. Due to this design we needed a way to create build information through composition, which drove us to create some custom activities for exactly this purpose. I plan to explore these activities and how to use them in a future post. Stay tuned …

TFS 2010 - An Introduction to Gated Check-in

2009-06-30T01:50:00Z

In TFS 2008 we made quite a few improvements to build, such as scheduled builds and continuous integration. The continuous integration feature is nice for detecting failures in the build as soon as possible to ensure minimal downtime and reduce the chance of generating a bad nightly build. However, it’s not a perfect solution since even when the build fails it is possible for others to submit unrelated changes and keep the build broken. As a first measure to aid this scenario, Buck Hodges wrote a check-in policy that would determine which, if any, build definitions would be affected by the changes being checked in – if the last build for the definition failed, the policy would not allow the user to check-in without overriding it. But there are quite a few issues with this approach, the main one being that it requires the users to install the policy and only override it when honestly submitting fixes for the build break. At this point you may be asking yourself “what if there were a way to stop people from submitting broken source code into the repository on the server?” This is exactly what gated check-in offers, and I intend to explore the features around gated check-in in the following paragraphs.

Setting Up Gated Check-in

Gated check-in is merely an extension of the check-in trigger for a build definition, so it should be easy to setup for those familiar continuous integration. In TFS 2008, we provided ‘Manual’, ‘Continuous Integration’, ‘Rolling builds’, and ‘Schedule’ (along with the option to force builds on that schedule even if no changes occurred since the last build). When viewing the build definition properties in TFS 2010, you will see the same options (with the addition of ‘Gated Check-in’):

When this trigger is selected for a build definition, any check-in that is made to a file that is mapped in the associated workspace will trigger a verification build. If the build succeeds, the changes will be submitted to the repository. If the build fails, the changes are not allowed to be submitted and must be fixed and resubmitted.

Submitting a Gated Check-in

When a user attempts to submit changes to a version control path that is mapped by a build definition with the ‘Gated Check-in’ trigger, they will be presented with the following dialog:

The first thing to point out here is that the check-in been automatically converted to a shelveset. In this particular scenario, the shelveset has been named ‘Gated_2009-06-29_02.49.00.9801’. The check-in notes, associated work items, etc., are all replicated into the shelveset and will be included in the changeset upon a successful build. The second piece of the dialog to pay attention to is the build definition selection. If there were only a single build definition that was considered affected by the changes, the combo box would be disabled and you would not be provided any options. If, however, there is more than one build definition that maps the changes being submitted, the user you will be presented with the list of build definitions as shown in the following image:

It’s important to note here that the actual definition selected to perform the check-in does not matter here, as both have been determined to map at least some subset of changes included in the check-in. However, it’s possible that ‘Gated Definition 1’ only maps a single file in the check-in, while ‘Gated Definition 2’ could map all the files or a larger subset. Currently we do not provide any indication as to what percentage of the changes are mapped by any given build definition. Once you have selected a build definition, you have the ability to specify some extra options. For example, it is entirely possible that you are a build administrator and wish to submit a fix directly rather than waiting for the verification build. However, this option will only be enabled for a selected build definition if you have the appropriate permission, ‘Override Check-in Validation by Build’. If you have the appropriate permission, we have provided the ability to bypass build verification as shown in the following image (although it’s disabled in this particular shot since no build definition is currently selected):

The other option available in the ‘Show Options’ section of the dialog is ‘Preserve my pending changes locally’. This allows the user to decide whether or not the changes being submitted should be left in the workspace or undone once the ‘Build Changes’ button is clicked (if the changes are left in the workspace, there is an entry point that we will explore later in the article to reconcile the submitted changes with those in your workspace). Once you are satisfied with the options on this dialog and ‘Build Changes’ is clicked, you will be redirected to the build explorer where the build just queued will be highlighted for easy discovery as shown here:

Now I know there is only one build in this particular queue so it wouldn’t be very difficult to locate the build, but imagine there is a team of 50-60 people all submitting changes to the same build definition, or even multiple definitions all sharing the same build controller. :-) Anyway, moving along to the successful gated check-in, you can see from the following build report that the submitted changeset is associated with the gated build, just like changesets would be associated with a non-gated build by comparing the labels and analyzing the differences.

Now that the build has completed successfully, you may have some further actions to perform if you chose not to preserve pending changes in your workspace. The entry point for this functionality is in build explorer, which is available via the context menu of the build detail entry in the ‘Completed Builds’ tab or the queued build entry in the ‘Queued Builds’ tab. If you only have a single workspace on the current machine then a workspace will be automatically selected for you. If, however, you have more than one workspace on the machine, you will be prompted to select the workspace which you would like to reconcile changes with. We currently do not support storing the source workspace from which the shelveset originated, so we cannot automatically determine the workspace used to create the shelveset. However, this feature is on a backlog and we are looking at including this functionality in a future version. The second entry point is from the gated check-in notification window in the build notification system tray application, which is included with the product in TFS 2010.

This has been a very targeted and simple walk-through of the new gated check-in functionality available in TFS 2010. I will be covering more advanced scenarios surrounding permissions, integration with the build notification system tray application, and more in future postings. Stay tuned!

The access control list is not canonical

2009-02-09T16:02:00Z

Developing a program that does not require administrative privileges requires quite a bit of work up-front during the install. You need to make sure you do things like write registry keys to HKLM, create file system entries, and most importantly ensure that your ACLs (Access Control List) are correct to ensure smooth operation of your program while running under that non-admin account. However, sometimes this last step fails with a rather cryptic error, mentioning something along the lines of "The Access Control List is not canonical". After searching the web for solutions to this error, I couldn't find a definitive answer. One thing I did notice is if you open the 'security' dialog of the object that exhibits this issue, the Windows UI will actually recognize that there is an issue and ask if you would like to correct it. So obviously there is a way to fix this problem, but for my scenario requiring user intervention was just not good enough .. so I returned to searching. I finally came across an algorithm, buried in the depths of MSDN, which described how to fix the ordering of a DACL (Discretionary Access Control List). There was only one problem: it was written in VB. So, after a lot of hard work converting the code (just kidding, it was quite simple to convert), I produced the following helper method to attempt fixing the order of a DACL.

   internal static void CanonicalizeDacl(NativeObjectSecurity objectSecurity)
   {
       if (objectSecurity == null)
       {
           throw new ArgumentNullException("objectSecurity");
       }

       if (objectSecurity.AreAccessRulesCanonical)
       {
           return;
       }

       // A canonical ACL must have ACES sorted according to the following order:
        //   1. Access-denied on the object
        //   2. Access-denied on a child or property
        //   3. Access-allowed on the object
        //   4. Access-allowed on a child or property
        //   5. All inherited ACEs 
        RawSecurityDescriptor descriptor = new RawSecurityDescriptor(objectSecurity.GetSecurityDescriptorSddlForm(AccessControlSections.Access));

       List<CommonAce> implicitDenyDacl = new List<CommonAce>();
       List<CommonAce> implicitDenyObjectDacl = new List<CommonAce>();
       List<CommonAce> inheritedDacl = new List<CommonAce>();
       List<CommonAce> implicitAllowDacl = new List<CommonAce>();
       List<CommonAce> implicitAllowObjectDacl = new List<CommonAce>();

       foreach (CommonAce ace in descriptor.DiscretionaryAcl)
       {
           if ((ace.AceFlags & AceFlags.Inherited) == AceFlags.Inherited)
           {
               inheritedDacl.Add(ace);
           }
           else
            {
               switch (ace.AceType)
               {
                   case AceType.AccessAllowed:
                       implicitAllowDacl.Add(ace);
                       break;

                   case AceType.AccessDenied:
                       implicitDenyDacl.Add(ace);
                       break;

                   case AceType.AccessAllowedObject:
                       implicitAllowObjectDacl.Add(ace);
                       break;

                   case AceType.AccessDeniedObject:
                       implicitDenyObjectDacl.Add(ace);
                       break;
               }
           }
       }

       Int32 aceIndex = 0;
       RawAcl newDacl = new RawAcl(descriptor.DiscretionaryAcl.Revision, descriptor.DiscretionaryAcl.Count);
       implicitDenyDacl.ForEach(x => newDacl.InsertAce(aceIndex++, x));
       implicitDenyObjectDacl.ForEach(x => newDacl.InsertAce(aceIndex++, x));
       implicitAllowDacl.ForEach(x => newDacl.InsertAce(aceIndex++, x));
       implicitAllowObjectDacl.ForEach(x => newDacl.InsertAce(aceIndex++, x));
       inheritedDacl.ForEach(x => newDacl.InsertAce(aceIndex++, x));

       if (aceIndex != descriptor.DiscretionaryAcl.Count)
       {
           System.Diagnostics.Debug.Fail("The DACL cannot be canonicalized since it would potentially result in a loss of information");
           return;
       }

       descriptor.DiscretionaryAcl = newDacl;
       objectSecurity.SetSecurityDescriptorSddlForm(descriptor.GetSddlForm(AccessControlSections.Access));
   }

I hope this code saves someone out there a lot of trouble and time. Until next time …

Hello, World

2009-02-06T20:16:00Z

What software developer can start anything new without the proverbial "Hello, World". Since I'm new to this side of things, this will serve as my introduction. My name is Patrick Carnahan, and I'm currently a developer on a product called Team Foundation Server, of which I focus on the Build Automation functionality. More specifically, I mainly design and work on the server code, including web services, data access, and SQL. This isn't where I started my almost 4 year career at Microsoft, however. I was originally hired on to the Version Control team within the same server product in June of 2005. At that point in time I knew nothing about .NET application development, ASP.NET web services, and very little about SQL development. This product has allowed me to interact with these technologies much more than I ever would have working on projects for myself. I have become proficient in SQL, including query performance analysis and debugging. My knowledge of .NET libraries continues to grow every day. I deal with distributed programming models all the time, which keeps my job extremely interesting.

I am very passionate about the product that I work on and hope to deliver compelling releases in the future. Hopefully this spot will serve as a more general discussion of programming topics, but I will most likely include targeted posts helping users of Team Foundation Server get the most out of the product. I have learned quite a bit since starting my tenure with this company, and hopefully I can begin to share my knowledge with others.