Darren Jefford

Software in the cloud: Cloud Workflow

2008-10-02T12:31:30Z

As promised in my last posting let's take a look at the new "Workflow in the cloud" functionality offered in the July drop of the BizTalk Services

This cloud based workflow system leverages Windows Workflow and is hosted in our datacenter and extended as appropriate through "cloud-scale" runtime services and infrastructure. The idea being that you can run your workflows on our infrastructure which will scale to your throughput needs - subject to appropriate payments. These workflows will be durable and resilient to any failures - all of this will be transparent to you, simply deploy your workflows and away you go.

As it currently stands you only get one workflow type to use, a sequential workflow which is of course well suited to straight-forward predictable machine workflows. Within this workflow you get to use Activities but you can't however use the complete palette of WF activities that you get in the .NET framework.

Your restricted to a number of new cloud activities provided which are as follows: CloudHttpSend, CloudHttpReceive, CloudIfElse, CloudSequence, CloudServiceBusSend, CloudDelay and CloudWhile.

All of these do exactly what they say on the tin, the CloudHttpSend activity enables you to send an HTTP GET or POST to a remote URI, the CloudHttpReceive enables something to send data to a running workflow instance via an HTTP POST.

The CloudServiceBusSend activity may be a new concept to you as it refers to invoking a Service exposed via the Internet Service Bus using the relay service which I covered in this posting. This means you can have a WCF service hosted within your corporate network but still enable it to be invoked from parties outside of your corporate boundary provided they have the appropriate permissions. In the case of a cloud based workflow, it can invoke services hosted within a corporate boundary from a Microsoft data-center without having to punch new holes in your firewall but still maintain security.

So, let's take a look at how you can create your own workflow, deploy and execute it using the BizTalk Services release that you can all use. To start you'll need to create an account at http://biztalk.net and download/install the BizTalk Services SDK.

The first step is to create a new Sequential Workflow, all cloud workflows must be Sequential Workflows and XOML only. There is no way to have "code behind" files with these workflows, everything has to be expressed using a workflow, activities and as appropriate rules. Once this has been created you should consider deleting the automatically created .cs file to avoid you accidently adding code which can't be used.

Once you've done this you should see the new cloud based activities in your toolbox as shown below, if not you may need to right click the toolbox, choose "Choose Items.." and then ensure that the cloud prefixed workflow activities are selected. These new activities are held within the System.ServiceBus.Workflow assembly under the %PROGRAMFILES%\Microsoft BizTalk Services SDK\Assemblies directory.

For the purposes of this simple Hello World cloud workflow we'll just use one activity. The CloudHttpReceive activity will be used to demonstrate how we can communicate with a running workflow instance. You shoud have a workflow as shown below

The usual exclamation marks indicate that we have some configuration to do on this activity, for the cloudHttpReceive the only configuration we need to supply is a response that that will be returned as part of the remote HTTP post to our workflow, you can also configure the HTTP Status code to return as required. For the purposes of this workflow and the code you'll see in a bit, set the name of this activity to helloWorldHttpReceive and supply a response body of your choosing.

Now that our simple workflow is done let's deploy it to the Cloud Workflow infrastructure hosted by Microsoft, right now there isn't a Deploy addin to Visual Studio although there is an underlying Web Service. For now we need to perform a simple copy/paste deployment model.

Firstly we need to get the XOML representation of the workflow, a quick way of doing this is to right Click the workflow in Solution Explorer, Chose "Open With.." and then choose "XMl Editor", you can now copy the XOML representation into the clipboard.

We now need to deploy the workflow via the BizTalk Services website, browse to https://workflow.biztalk.net/ and supply your BizTalk Services credentials to view the workflow management portal as shown below.

Click "Manage Types" on the right hand side to manage the workflow types under your account, then click Add New to access the Workflow Deployment page which is shown below. For the purposes of this demo we'll call our Workflow "Hello World" and you can now paste the XOML representation into the supplied box.

Click Save changes and your cloud workflow is now ready to go! Note the "Rules" tab in the previous screenshot, this is where you can also provide any rules associated with your cloud workflow.

Now our workflow is created how can we go about kicking it off within the Microsoft datacenter? It's pretty straight forward once you know how to do it and the BizTalk Services SDK has a sample to help you out. Those of you familiar with BizTalk Orchestrations you'd expect that we could now perform a simple HTTP POST at a special URI and have a workflow instance created automagically? Not right now, I'm afraid - you have to manually crete an instance, then start it before you can submit an HTTP post, this as I understand it will be changed as the product teams progress.

The code to create, start and then HTTP POST to a workflow instance is shown below, it's all pretty straight forward and a lot of it is demonstrated in the SDK sample. The key thing in the code is how to target your HTTP POST of a running workflow instance. The URI is made up as follows:

http://workflow.biztalk.net/servicesHttp/YOURBIZTALKSERVICESUSERNAME/workflows/YOURWORKFLOWNAME/instances/YOURWORKFLOWINSTANCEID/HTTPRECEIVEACTIVITYNAME

And an example for my scenario here is as follows:

http://workflow.biztalk.net/servicesHttp/darrenj/workflows/HelloWorld/instances/83734f1c-d08f-4cc9-b396-b76ebdb60979/helloWorldHttpReceive

You'll need to add assembly references to your project for System.ServiceBus.dll, System.ServiceBus.Workflow.dll and System.ServiceModel.dll

[TestMethod]
public void TestMethod1()
{
    // Create a new Workflow Instance in the cloud, we then get the InstanceID back
   string instanceId = CreateWorkflowInstanceInTheCloud();

    // The Workflow Type
   string workflowTypeName = "HelloWorld";

    // The PlaceOrder Workflow starts with a cloudHttpReceive activity called "placeOrderHttpReceive"
    // We must target this directly when posting our Order information to the cloud workflow
   string httpActivityName = "helloWorldHttpReceive";

    // Build up the URL to directly target our newly created Workflow Instance and to post XML directly to the
    // waiting cloudHttpReceive activity

   Uri serviceUri = new Uri(
        string.Format(WorkflowClientConfig.HttpUri + "{1}/workflows/{2}/instances/{3}/{4}",
        WorkflowClientConfig.WorkflowHostName, _bizTalkServicesUsername, workflowTypeName, instanceId, httpActivityName));

    string httpPostBody = "<HelloWorldWorkflow><Message>Hello World</Message></HelloWorldWorkflow>";

    // Prepare to perform an HTTP POST to the waiting workflow
   HttpWebRequest request = (HttpWebRequest)WebRequest.Create(serviceUri);
    request.Method = "POST";
    request.ContentType = "text/xml";
    request.ContentLength = httpPostBody.Length;
    // Add the special header which contains our STS token retrieved from the cloud
   request.Headers.Add("X-MS-Identity-Token", GetTokenFromCloudSts());

    using(StreamWriter writer = new StreamWriter(request.GetRequestStream()))
    {
        // Write the OrderInformation XML representation into the stream
       writer.Write(httpPostBody);
       writer.Flush();

        // Wait for a response from the Cloud Workflow, the cloudHttpReceive activity is configured to respond with a fixed
        // Response if succesful;
       HttpWebResponse response = (HttpWebResponse)request.GetResponse();

        using(StreamReader reader = new StreamReader(response.GetResponseStream()))
        {
            string returnMessage = reader.ReadToEnd();

            Trace.WriteLine("Response message from the cloud is: " + returnMessage);

            // For some reason all responses from the cloudHttpReceive activity are wrapped into a <string> element
            Assert.IsTrue(returnMessage.Contains("Hello World Response"));
        }
    }
}

private string GetTokenFromCloudSts()
{
    // Using a tokenised URL let's build up a valid URL given the supplied BizTalk Services credentials
    // This URL when invoked will return an authentication token we can then use to perform administrative operations on
    // Our Workflow Configuration (Create, Start, etc.)
    string tokenUrl = string.Format(
            "https://{0}/issuetoken.aspx?u={1}&p={2}",
            WorkflowClientConfig.StsHostName,
            _bizTalkServicesUsername,
            _bizTalkServicesPassword);

    // Create an HttpWebRequest
    HttpWebRequest tokenRequest = (HttpWebRequest)WebRequest.Create(tokenUrl);
    Trace.WriteLine(tokenRequest.RequestUri);

    // No Request to send, just invoke the URL and get the authentication token
    HttpWebResponse tokenResponse = (HttpWebResponse)tokenRequest.GetResponse();

    // Get the token from the Response Stream
    byte[] tokenBody = new byte[500];
    int tokenBodyLength = tokenResponse.GetResponseStream().Read(tokenBody, 0, 500);

    // Convert to a UTF8 representation which we can then use in subsequent requests
    string authenticationToken = Encoding.UTF8.GetString(tokenBody, 0, tokenBodyLength);
    Trace.WriteLine(String.Format("Authentication Token: {0}", authenticationToken));

    return authenticationToken;
}

private string CreateWorkflowInstanceInTheCloud()
{
    // New up an instance of the provided System.ServiceBus.Workflow.WorkflowClient which provides a wrapper
    // around the cloud workflow SOAP Web Servics        
    WorkflowClient workflowClient = new WorkflowClient();

    // Provide BizTalk Services credentials
    workflowClient.CredentialType = TransportClientCredentialType.UserNamePassword;
    workflowClient.Credentials.UserName.UserName = _bizTalkServicesUsername;
    workflowClient.Credentials.UserName.Password = _bizTalkServicesPassword;

    workflowClient.Open();

    // Parameters can't be supplied in R12 of Cloud Services so we this empty
    // A shame as it would enable us to pass the OrderInformatoin directly
    Dictionary<string, object> parameters = new Dictionary<string, object>();

    // Create a workflow within the BizTalk Services username namespace 
    // The Workflow Type name is "HelloWorld"
    // No parameters in R12
    // Retrieve an authentication token from the BizTalk Services Cloud STS
    string instanceId = workflowClient.CreateWorkflowInstance(_bizTalkServicesUsername, "HelloWorld", parameters, GetTokenFromCloudSts());
    Trace.WriteLine("Workflow Instance Created:" + instanceId);

    // Once a workflow is created it must be explicitly started otherwise it won't active any activity shapes
    // In R12 there's no ability to create/start a workflow based on an external activity action such as an Http Send
    // The workflow instance was created within the BizTalk Services username namespace
    // The workflow Type name is "HelloWorld"
    // The instanceId was returned by the previous CreateWorkflowInstance call
    workflowClient.StartWorkflowInstance(_bizTalkServicesUsername, "HelloWorld", instanceId);
    Trace.WriteLine("Workflow Instance Started:" + instanceId);

    // Our work is done, close the WorkflowClient
    workflowClient.Close();

    // Return the InstanceId so we can identify the WorkflowInstance we've just created when we perform an HTTP post to it next
    return instanceId;
}

All pretty straight forward, and it's a bit more complex then it should be due to the requirement to create and start a workflow instance before being able to perform an HTTP post but this should come in time.

If you navigate back to the https://workflow.biztalk.net site then you can review running workflow instances and check they've completed successfully.

That's it for this post, this post has got rather large so I'll do another post on calling ServiceBus services via the cloudServiceBusSend activity and also detail some limitations in this release of Cloud Workflow (it's the first release for the team so don't expect everything to be there straight away!). I'll also cover a handy little ISBTraceTool that I put together to aid with visualizing cloud workflow activity.

More soon!

Dublin!

2008-10-02T00:43:01Z

This announcement came sooner then I expected, I'd assumed PDC was going to be the first time Dublin was mentioned in public! The most detailed information before the PDC can be found here.

A number of internal people in the field (myself included) have been working with the product team for a number of years now to help shape some of these new technologies and ensure they will address real-world customer scenarios based on our customer experience. It's great to be able to start discussing what's coming down the line, although the real detail won't be available until the PDC.

So Dublin! Windows Server is our application server today and we're now expanding it's capabilities to deploy and manage .NET based applications (using WCF and WF) - no more roll your own!

If you appreciate the way BizTalk Server provides enterprise capabilities to host your Integration solutions today, you'll see a lot of similarities in how these hosting capabilities are being introduced in "Dublin" for WCF and WF. For those who might have shied away from WF because of having to roll their own infrastructure, this is great news!

Note that in the press-releases that BizTalk is formally being referred to as an integration server, e.g:

Q: Will "Dublin" work with BizTalk Server's enterprise connectivity services?
A: Yes. The integration server and application server workloads are distinct but complementary; customers want to be able to deploy them separately as needed to support their distinct requirements. For example, customers that don't need the rich LOB or B2B connectivity provided by an integration server, will deploy the Windows Server application server to host and manage middle-tier applications. Likewise, customers that need to connect heterogeneous systems across an enterprise, but don't need to develop and run of custom application logic, will deploy BizTalk Server. When customers need both capabilities, "Dublin" and BizTalk Server will work together nicely.

BizTalk is by no means "dead", in fact Microsoft committed to future versions including BizTalk Server 2009 recently for the integration server workload, BizTalk = Integration, Dublin = Application.

So if you want to expose [WF] workflows via [WCF] services but ensure performance and scalability (up to enterprise scale), you can now do this without having to write the code required to host these apps on Windows Server. Ensuring performance and scale of WCF services and WF is hard to do today, hence it's not done very often at least in my experience and sometimes causes a tendency to twist BizTalk into doing something it wasn't necessarily designed to do which causes problems of their own (coupling Web Sites/UI's directly to BizTalk for synchronous processing springs to mind).

We don't want customers in this situation to be forced into writing huge amounts of hard plumbing code to achieve this, we need a server product to do this for you, which is where Dublin comes in. Note some of the server features announced which will be familiar to BizTalk developers (content based routing, compensation, etc.).

If however you need to use the extensive adapter support, B2B, EDI, RFID, BAM, BizTalk Mapper style features then you'll still be wanting to use BizTalk. Both products will work together seamlessly through the WCF communication options so you can combine as appropriate.

Remember though that a number of BizTalk adapters have been re-written as WCF bindings and are available through the BizTalk Adapter Pack which offers some key LOB adapters. A new SQL adapter is in the works along with new Oracle adapters which you can find information on here. As these new adapters are exposed as WCF bindings you can leverage them with WCF and Dublin.

Dublin will also be the first and best consumer of the Oslo modeling platform, they'll be more detail on this at the PDC but trust me - this is going to blow your minds!

WCF and WF get a big makeover as already announced, we get new workflow types in WF and an extension library of Activities out of the box. If you want to call a SQL stored procedure, why write half a page of code when you can just configure a database activity? Combine that with the 10x improvement in performance and things are looking good! Imagine a world when a typical software solution can be implemented (modeled) exclusively through a workflow and out-of-the-box activities? ;-)

Notice also the subtle new feature in WF, "persistence control". Low Latency scenarios with BizTalk are achievable but has to be carefully designed, we don't currently have the ability in BizTalk to control when an Orchestration persists but imagine if we had this feature in WF - low-latency potentially becomes easier to achieve ;-)

That's enough for now, once more detail starts to emerge I'll post some more information and will also work internally on locking down some clear "where to use what" style scenarios for BizTalk and Dublin using some real-world customer scenarios.

Exciting times!

Professional BizTalk Server 2009?

2008-09-29T13:34:00Z

I've been looking into the possibility of updating the highly successful Professional BizTalk Server 2006 book to take into account new features and capabilities in BizTalk Server 2006 R2 and the recently announced BizTalk Server 2009 product.

A fair bit has changed to warrant new chapters and updating of various chapters but of course the main body of content is still accurate and will be brought forward into any new edition.

The current book is of course still absolutely usable for both BizTalk Server 2006, R2 and even 2009 as the main engine, features and best practices are unchanged.

I've listed some of my thoughts around what needs to be covered to expand the reach of the book given the new version but would love to hear of topics and sections that you would like to see added if you've already read the current edition.

Please send me a mail with your thoughts via my blog site or add comments to this posting, I'd really appreciate your views on what we should consider.

I haven't got formal commitment from Wiley yet as I'm still at the Table of Contents stage, here are my outline thoughts as it currently stands.

Technology Primer - Update

· Cover WCF basics

· Cover the penalty of XmlDocument vs Serializable Classes vs XPath with real-examples of overhead

BizTalk Architecture - Update

· Emphasise further how important testing is, don’t expect performance and how critical the performance of supporting systems is. Detail techniques for isolating and testing performance of supporting systems

· Cover any changes with BTS2009

· Potentially position BTS2009 vs Oslo at the architectural level

Adapters - Update

· Include drill-down on WCF Adapter topics

· Overview

· Different WCF-* adapters

· Walkthrough

· Demonstrate how a custom binding can be developed using the WCF Adapter SDK (null adapter?)

· Cover the new WCF Adapter bindings (SQL, MQS, etc)

Business Activity Monitoring - Update

· Provide implementation of BAM Latency Timer pipeline component

· Provide more real-world and common questions/issues guidance

Business Rules Engine - Update

· Position BRE vs the WF rules engine and futures

RFID - New Chapter

· Drill-down chapter into the RFID features of BizTalk complete with real-world examples

Testing - Update

· Provide more real-world “case studies” of how it’s been done right and the benefits..

· New BizUnit features (Excel)

· Update with any new VS2008 (and maybe future release features)

Performance and Scalability - Update

· PAL for analysis of performance logs

· Discuss SAN technology improvements

· ‘n’ step plan for identifying where bottlenecks lie within a BizTalk rig

· Expand throttling “plain” English guide, step-by-step diagnosis guide

Low Latency - Update

· Cover new options such as NetTcp and NetNamedPipe WCF bindings

· Include real-world perf-test results, step by step guide

Administration - Update

· Virtualization (HyperV)

· Best Practice Analyser

· MSBuild

· SQL2008 differences

BizTalk Best Practices - Update

· Add further best practices, small and large – with evidence as to why

Cloud Services (BizTalk Services, SSDS, etc.) - New Chapter

· Overview

· Connectivity

· Identity

· Cloud Workflow

· SSDS

· Cloud Commerce scenario code and walkthrough

First look at Oslo futures - New Chapter

· More soon ;-)

Software in the cloud: The Relay Service

2008-09-24T16:39:00Z

I’m now back from a trip to Seattle a few weeks back where I attended a huge week-long Microsoft internal technical training event. I presented three times during the week, two of the presentations were “Software in the cloud” sessions which received fantastic feedback.

The premise of these sessions was to paint a view of how software solutions may look in the future if they were to leverage cloud based services such as SQL Server Data Services (SSDS) and BizTalk Services (Cloud Workflow, Identity and Connectivity).

I presented a fictional but realistic architecture using today’s technologies and discussed the challenges, I then presented a forward view on how you might approach the same problem using cloud based technologies and how this solves a number of challenges today and opens up completely new opportunities which for the most part aren’t possible today.

I’m going to present this scenario and the resulting solution in a series of blog posts, this being the first.

Before we get started I want to introduce the BizTalk Services Relay which by itself is incredibly powerful but often overlooked. There is an SDK to download and play with via http://www.biztalk.net

So, if I have a Service offering today, say Credit Scoring functionality, it’s hard to enable business partners to invoke this service, especially if your organisation doesn’t host things out on the internet regularly.

You’ve got any number of options generally involving any combination of leased lines, VPNs, proxy configuration, etc. The key point here is that you must put in infrastructure and/or specific configuration for each business partner that wishes to leverage your service, and of course vice versa, each business partner will have to do the same to enable messages to pass out of their corporate boundary.

So, imagine a technology that enabled you to avoid all of this infrastructure/configuration whilst still maintaining security and integrity of messages passing between the organisations?

This is where the Relay comes into play; the first step is for your WCF service to register it’s endpoint with the cloud based relay service, a valid BizTalk Services account will be required to register an endpoint under the users namespace. This step is shown below as step number 1.

The endpoint address will use a specific “service bus” prefix, e.g: sb://connect.biztalk.net/services/darrenj/OrderService/

Once an endpoint has been registered with the relay service it must be kept alive otherwise the socket could of course timeout. This is done through a series of “ping” messages automatically passed under the covers between the relay service and the endpoint.

Once an endpoint is registered a client can then invoke the service, within Visual Studio you can simply type in the sb:// prefixed address via the usual Add Service Reference dialog and the proxy will be automatically created and configured – neat!

The proxy can then use the contract as usual, once invoked the message will be passed to the endpoint registered in the cloud (shown as number 3 in the diagram above) which will in-turn pass the message on to the service (shown as number 4 in the diagram above), the reverse then happens with the response (shown as number 4 in the diagram above).

In essence this is a straight forward message relay pattern with all messages passing via the relay, enabling two parties to exchange messages where they otherwise wouldn’t have been able to.

This works fine, but it would be nicer (and probably quicker) if we could enable both parties to communicate directly but the firewalls won’t allow this. However the relay service knows quite a bit about the two parties.

When the service registered itself with the relay, the relay can see which dynamic port number the hosting organisation’s firewall is using for this communication session. When a client communicates with the relay, the relay can again see the dynamic port number being used.

By sharing the port numbers with each party the relay can effectively step out of the way and enable both parties to communicate with each other after this initial port handshaking, this is depicted below and is explained in better detail in this great article

This process is called NAT traversal and is used by many things today including MSN Messenger, Groove, Skype, etc. Some firewalls may not allow NAT traversal at which point the relay will fall back to the original relay pattern. As long as both parties can see the internet and therefore the relay service then we’re in business!

This is a big step forward from today where you have to provision and configure network infrastructure such as Proxies, Firewalls, Private Network Links, VPNs, etc. whenever you wish to expose invoke services between organisations.

With this relay service you can break down your corporate barriers to communicate across organisations and through the relay you can also utilise non-durable publish/subscribe messaging effectively publishing business “events” which others can subscribe to.

So how about security? Firewalls are there for a reason and you typically aren’t allowed to punch holes through them. Access to any services exposed through the relay is restricted through the BizTalk Services provided implementation of a Security Token Service (STS) and messages can be secured as you require using encryption of the payload and digital signatures.

So, it must be hard to host a service through the relay? Well not really, it’s just a bit of WCF configuration on your service an example of which is shown below:

<system.serviceModel>

<binding name="default" allowBrowsing="true" sendTimeout="01:00:00"

connectionMode="RelayedDuplexSession">

</binding>

</relayBinding>

</bindings>

<endpoint

address=""

binding="relayBinding"

contract="PaymentService.IPayment"

behaviorConfiguration="relayBehavior"

bindingConfiguration="default"/>

</clientCredentials>

</transportClientEndpointBehavior>

</behavior>

</endpointBehaviors>

</behaviors>

</system.serviceModel>

You don’t need to include the custom endpointBehavior in all cases, if you don’t then CardSpace will pop up to authenticate you and validate that you have permission to register a service at the specified endpoint – not ideal for non-interactive services J

Hopefully that’s given you a good flavour of what the relay service can do, it’s a fantastic piece of technology and opens up the opportunity to break down the enterprise SOA barriers whereby your services can only be consumed or exposed within your corporate boundaries.

To my mind, this opens up some great new opportunities for solutions moving forward, as we’ll see in my next few blog posts J

Cloud Services

2008-07-17T01:03:00Z

CSD have just announced Release 12 (R12) of Codename BizTalk Services, Clemens has a good low-down on what’s new here. Remember BizTalk Services != BizTalk. BizTalk isn’t used in the cloud or the client – it’s just a branding thing.

The Connectivity service (relay) is an amazing piece of technology breaking down corporate boundaries and enabling true communication between organisations and people despite all of the firewalls and proxies put in place. It’s been available for a while but I’ll do a blog post on my take soon in case you haven’t seen it in action.

The big news in this release is the addition of cloud based Workflows, we are now hosting Windows Workflow in our datacenter which can run your own Sequential Workflows. You can’t use the full toolbox of usual WF activities though but a cut-down selection that the team have supplied.

These workflows as Clemens discusses on his blog are aimed towards the “service orchestration”, so a business process that you have can be modelled using a workflow and invoke any number of Internet Service Bus (ISB) services and also perform HTTP requests, workflows can also be communicated with using HTTP.

The current list of cloud activities are as follows

· CloudHttpSend

· CloudHttpReceive

· CloudIfElse

· CloudSequence

· CloudServiceBusSend

· CloudDelay

· CloudWhile

Only XOML workflows are allowed (so no code behinds here!) and if you need to do any coding you must push this into a Service that your workflow can invoke using the CloudServiceBusSend or Http activities.

You must copy/paste the XOML from Visual Studio into a HTTP form available when you sign into www.biztalk.net, clunky but effective for now!

Workflows can’t be activated through say an HTTP post but instead an instance has to be first created and then started before any activities will be executed, so if you had a workflow with a CloudHttpReceive activity at the top you would have to follow those previous steps before you can send an HTTP post to it.

There is a handy WorkflowClient class provided that means you can automate the creation and starting of a workflow instance, I’ve got a unit test that does this before communicating with the created workflow instance.

I’ll post a little sample shortly to get you going straight away but in the meantime check out the samples provided in the BizTalk Services SDK.

Long time, no blog!

2008-07-17T00:50:00Z

Yes, It’s been quite a while since my last blog post! A lot has been happening internally including a job move for me to the Solution Architecture team in MCS as a Solution Architect from the fantastic Application Development Consulting (ADC) team.

I still retain my strong technical focus you’ll be pleased to know and I don’t think I’ll be getting away from BizTalk anytime soon J

I’ve been doing a lot of work in the “fluffy cloud” space in recent months and will be presenting a rather complex session at our internal technical event later this month.

This work has included pulling together a fictional scenario which demonstrates stitching together SQL Server Data Services (SSDS), Cloud Services (Connectivity, Identify and Workflow) and good old BizTalk to address challenges with “today’s” technologies and enable new possibilities.

It’s been an interesting experience, not everything can be done right now as both SSDS and the Cloud Services are still in the CTP phase and missing certain features but the future looks very interesting.

More on this over the next few days and weeks plus after my presentation and demo is over I’ll post the code and more details on my fictional scenario so you can take a look and see what I've been tinkering with.

My other pet project has been a BizUnit DSL tool to enable graphical visualization and editing of BizUnit tests, again more soon hopefully

64bit support for the BizTalk Orchestration Profiler and BizTalk Documenter

2007-11-28T14:36:00Z

A number of you have contacted me in recent months to report that the BizTalk Orchestration Profiler and BizTalk Documenter didn't run on 64bit machines, this seems to have been down to compiler settings which I've now resolved.

I've uploaded a planned release for both tools to codeplex - if you've got a 64bit machine I'd appreciate it if you could install and test the tool to see if it works on your servers? Please let me know either way - I'll then make the releases as the default released versions once I've got validation.

You can find the releases at the following URLs:

http://www.codeplex.com/BizTalkDocumenter/Release/ProjectReleases.aspx?ReleaseId=8689

http://www.codeplex.com/BiztalkOrcProfiler/Release/ProjectReleases.aspx?ReleaseId=8688

I've done some tidying up across the entire codebase to make it easier to support and also reworked and re-released the Word output option for the BizTalk Documenter thus allowing Word documents to be generated as well as CHM files. Let me know if the Word output doesn't work as expected for you when compared to CHM output.

Please keep any feature suggestions coming...

How everyone should test their BizTalk based solutions: Presentation

2007-11-01T21:06:00Z

I've managed to find somewhere to store the presentation deck I delivered yesteday and you can now download it from here. I've also added the slides I had to cut from the presentation to enable it to fit within the SOA & BPM conference breakout length - these cover the throttling semantics at a high level and detail some common symptoms you might see during performance testing and what you should look at to resolve them.

There's of course more detail in the book! Enjoy.

Codename “Oslo”

2007-11-01T02:43:00Z

Microsoft has just made the first public announcement of “Oslo” at the SOA and BPM conference, at a high level “Oslo” is an overarching initiative across multiple products and Microsoft divisions, in fact the first release as announced today will be made up of BizTalk Server “6”, BizTalk Services “1”, Visual Studio “10”, System Center “5” and .NET Framework “4.0”. Therefore Oslo != Just BizTalk.

Official press release information can be found here and here along with a good overview by Directions on Microsoft which you can find here, but here’s my take for what’s it worth! J

Oslo is driven out of Connected Systems Division (CSD) which is the almagamtion of the BizTalk, WCF, WF, ASMX, MSMQ, .NET Remoting, etc. teams and has around 1000 people, Oslo as highlighted above is a huge program of work within CSD and across multiple divisions within Microsoft (AD, System Center, Developer Division, etc.). It's a huge investment.

Two key themes for Oslo were annouced during the keynote, Models (Making models a mainstream part of development) and Services (Extending services from the client to the cloud).

Those of you familiar with BizTalk Orchestration and Windows Workflow or even “Whitehorse” style designers such as those created using the DSL Tools or the Class Designer in VS are well versed with how a modelling tool works effectively enabling you to view and modify an underlying “thing” with an abstracted graphical language.

There has been lots of modelling attempts across the years across the industry; two things stand out for me as to why. Firstly the model is typically a representation at a point in time, you draw a nice pretty picture either for reference purposes or to then generate some physical representation like code.

Sometimes you get round-tripping which enables the model to be up-to-date but this becomes increasingly hard in most scenarios to the point of not being possible so invariably the model becomes out of date with the actual reality and therefore its value is minimal.

Secondly you have lots of “models” within a typical project, perhaps you have a business process modelled using BizTalk Orchestration, the physical datacenter modelled using the VS Team Architect tools and operational management within System Center. Nothing is shared between these models and therefore everyone has their own version of reality – often out of date and there’s no knowledge transfer between the various software development lifecycle roles.

In short, the modelling vision is a great thing but has a number of serious limitations today which limits its values to small isolated parts of your solution, imagine though having one model to describe your solution that could be used across your entire solution development lifecycle (Business Analyst, Developer, Tester, Operations, etc.).

This model of course needs to be rendered into different views for each of these stakeholders, a business analyst has a different “view” requirement to a developer for instance.

All of this is fine but don’t we end up with a stale description of a solution? Not in Oslo as the model is executed; therefore the model has to be up-to-date as its being used to execute the solution.

These models then have to “live somewhere”, firstly to make them broadly discoverable by all the stakeholders involved in the development process and secondly to enable them to be executed by some form of host. This is where the Metadata repository comes in which leverages SQL Server.

The viewing/editing experience will be provided through a new tool offering an “office like experience”, this will be able to interrogate the repository and retrieve/persist models. Precise technical detail of how all this hangs together isn’t available right now but more information will be available publically in due course.

So Modelling is a big focus for Oslo, basically taking the modelling concept but making it mainstream and used across the entire development lifecycle and thus connecting up all roles.

The next big focus for Oslo is around services and messaging; BizTalk today provides an incredibly powerful and flexible messaging platform through Adapters, Pipelines and the MessageBox. We have publish and subscribe, correlation, transformation all in the box.

Through the use of adapters we can easily integrate with a variety of technologies provided by different vendors running on different platforms but this typically speaking is all about integration/messaging within your corporate boundaries.

You can of course implement B2B scenarios where you can do cross business integration but any communication between B2B partners typically requires specific and dedicated communication links such as a leased line, VPN, SSL, etc. This is expensive and normally only implemented by the larger organisations.

Oslo is set to address this and more through the use of what Microsoft is calling a “Internet Service Bus”, imagine being able to connect two organisations together without requiring anything more than a internet connection, no need for leased lines, NAT traversal?

This in part is offered today through the BizTalk Labs preview of the BizTalk Connectivity Services, this offers the concept of a relay which can patch two applications together even if they’re behind their own respective firewalls and would normally have no way to directly communicate with each other.

The relay offers NAT traversal technologies and once it’s patched you together it can then completely step out of the communication path enabling you to communicate together. Now this might not see that useful on the surface but it opens up a whole new breed of solutions for the small-medium market but of course any size of user or customer.

In short the relay enables communicate between users and applications even though there is no direct communication link between the two – of course this NAT traversal solution has been down before in silos by software such as Groove and MSN Messenger. The key here is that the Connectivity Services are doing all the hard NAT stuff for you and the service is hosted by Microsoft in the cloud.

A Security Token Service (STS) is also supplied through the BizTalk Labs preview which offers an identity provider to enable you to offload identity and authentication functionality. Moving forward there is a commitment to provide workflow hosting.

All of these services are cloud hosted meaning that a Microsoft datacenter somewhere provides the hardware, resilience, software, etc. to expose these services, you don’t need to provision datacenter space, hardware, etc. but instead rely on Microsoft of another third party. How this will be charged for hasn’t been agreed at this time.

So combining models and these cloud based services is interesting as suddenly you can develop your solution using Oslo tools and then make a deployment-time decision to host within your datacenter or to host within the cloud. This is an interesting choice for organisations that currently maintain large datacenters and have to worry about monitoring, resilience and the like. It also opens up completely new possibilites about how users/services can be contacted regardless of where they are, as long as they have a internet connection and can "see" the cloud they can communicate regardless of what network infrastructure is in the way.

So, all of that aside – your probably thinking about BizTalk today? Should you continue to use it, will any new investment in BizTalk Orchestrations, Rules Engine, etc. be redundant in a few years time? There’s also a slightly scary statement in the Directions on Microsoft report that advises that you may wish to limit new development using BizTalk Server.

I don’t speak for the product team, but can say that enabling you to run BizTalk Server 2006 artefacts within the Oslo release will absolutely be supported – Microsoft can’t turn away from the significant investment that customers have made today and aren’t planning to.

So, in short carry on using Biz Talk. Oslo is a way away yet and there’s an incredible amount you can do with our technology stack today (BizTalk Server 2006 R2, Office 2007 Sharepoint, WCF, WF....)

That’s it for now, will provide some more thoughts and info as time goes on.

How everyone should test their BizTalk based solutions presentation links

2007-11-01T02:32:00Z

Thanks to everyone who came to my presentation today, I was humbled by the huge turnout! The presentation deck will be broadly shared to attendees which contains hyperlinks to all the tools showcased but in the meantime the links are below if you want to get started right away. I'll look to share the entire deck via the blog shortly if I'm allowed!

A new release of LoadGen has very recently been made available which I didn't mention today, this adds WCF support for injecting load amonst other things.

PerfMon Counter spreadsheet
http://media.wiley.com/product_ancillary/22/04700464/DOWNLOAD/Chapter9_Code.zip

Orchestration Profiler
http://www.codeplex.com/biztalkorcprofiler

BizTalk Documenter
http://www.codeplex.com/biztalkdocumenter

LoadGen 2007 (now includes WCF, Sharepoint 2007 support amongst others)

http://www.microsoft.com/downloads/info.aspx?na=22&p=1&SrcDisplayLang=en&SrcCategoryId=&SrcFamilyId=&u=%2fdownloads%2fdetails.aspx%3fFamilyID%3dc8af583f-7044-48db-b7b9-969072df1689%26DisplayLang%3den

Old version of LoadGen 2004 (recommend you use the 2007 release moving forward)
http://go.microsoft.com/fwlink/?LinkId=59841

BizUnit
http://www.codeplex.com/bizunit

Microsoft SOA & BPM Conference

2007-10-29T19:51:00Z

I've just arrived in Seattle for the Microsoft SOA & BPM conference that kicks off on Tuesday 30th October in Seattle, I'm presenting a best-practice session on Wednesday afternoon titled "How everyone should test their BizTalk Server based solutions", it's a great deck with some compelling demos that went down very well at a internal technical conference earlier this year. If we can get BizTalk solutions tested properly - the world will be a much better place!

More as the week goes on, should be some interesting presentations....... In the meantime Sam Gentile has a great running commentary of the week.

Orchestration Profiler for BizTalk Server 2006 and BizTalk Documenter

2007-07-05T14:53:00Z

Hi,

I've recently taken ownership of the source for the BizTalk Orchestration Profiler and BizTalk Documenter tools that Jason Birth originally developed, the documenter has worked fine with BizTalk Server 2006 but the Profiler was broken in a number of places. I have just finished a new build of the Profiler which now works with 2006 and fixed a few other niggles, both tools are now available and I'm open to feature suggestions for both tools moving forward.

If you haven't seen or used the tools before then do take a look - they are invaluable in many circumstances and I for one use them one way or another on every BizTalk engagement, the Profiler is here: http://www.codeplex.com/BizTalkOrcProfiler and the Documenter is here: http://www.codeplex.com/BizTalkDocumenter

Information on both tools is copy/pasted below from the respective CodePlex sites, enjoy!

BizTalk Orchestration Profiler

Creates CHM report files illustrating the level of coverage for specified BizTalk orchestrations. This tool can be run to gain a consolidated view of orchestration tracking data for a specified period of time to help developers get an idea of how their orchestrations are being processed and how much coverage they are getting in their testing. In addition to simple coverage information the data presented helps to identify latency and code path exceptions by highlighting long running and error prone orchestration shapes which is key to effective performance testing.

The image below shows a sample of the coverage summary, at a glance you can see the percentage of overall coverage acheived upon an individual orchestration, if all of the shapes are touched as part of your orchestration then you will acheive 100% coverage, common reasons for not covering all shapes with an orchestration are not running tests that stress the error paths of your solution. This coverage summary also provides you metrics around execution duration of your orchestration easily identifying orchestrations that take a long time to execute.

The image below shows the orchestration coverage in detail, shapes highlighted in green have been executed as part of your last test pass and any shapes not executed will be highlighted in red. This enables you to easily understand what parts of your solution have been tested and which parts have not and adjust your testing as necessary.

As part of the profiler report you can see the "Top 10 Least Successsfull" shapes and the "Top 10 Longest Running" shapes, understanding where time is being spent inside your orchestration is key to effective performance testing and has been proven on a number of instances to pinpoint poor orchestration design and coding which when resolved has dramatically increased overall performance. The key is to understanding exactly where time is being spent and this tool offers a no-code view into your running system.

BizTalk Documenter

Creates compiled help files for a given BTS 2006 installation. This tool can be run on an ad-hoc basis using the UI or from the command line as a post build/deploy task to create a compiled help file describing a BTS 2006 installation. It will compile: BTS Host configuration, Send/Receive port configuration, Orchestration diagrams, Schema and Map content, Pipeline process flow, Adapter configuration, Rule engine vocabularies and policies, More… and publish them as compiled help files. Optionally you can embed custom HTML content and custom descriptions for all BTS artifacts to produce a more customized look and feel to the CHM output

This tool can be run on an ad-hoc basis using the UI or from the command line as a post build/deploy task to create a compiled help file describing a BTS 2K4 installation. It will compile:

• BTS Host configuration
• Send/Receive port configuration
• Orchestration diagrams complete with any custom code
• Schema and Map content
• Pipeline process flow
• Adapter configuration
• Rule engine vocabularies and policies
• More…

and publish them as compiled help files or Word 2003 XML. Optionally you can embed custom HTML content and custom descriptions for all BTS artifacts to produce a more customized look and feel to the CHM output.

The screenshow below shows some of the information collected about an Orchestration, you can see what variables, maps, messages, ports, correlation sets, etc. that are used by a given Orchestration.

The screenshot below shows further orchestration information which is collected, in this case we can see all of the inline code written by the developer for the orchestration, useful for code reviews and understanding a solution.

GenerateTypedBamApi

2007-06-06T03:47:00Z

Long overdue but I've finally got around to finishing up the latest release (1.1) of the GenerateTypedBamApi tool. You can find the new version here, if your not familar with the tool read on!

The GenerateTypedBamApi command line tool enables you to take a BAM Observation model represented as a Excel Spreadsheet and generate a set of strongly typed C# classes which you can then use to create and populate BAM Activities.

The native BAM API is loosely typed and therefore requires Activity Names and Activity Items to be supplied as string literals, this can be brittle especially as the observation model evolves over time and any typos, etc. will lead to runtime errors instead of compile time errors.

A strongly typed class is created for each Activity with properties for each Activity Item along with helper methods to write Activity Items to an Activity, Add References, Custom References (e.g. Message Bodies) and Continuation.

This tool uses a XSLT transform to turn the XML representing the BAM Observation Model into C# code.

For further information on how to use BAM and why you might want to use a BAM API approach over the in-bult graphical Tracking Profile Editor please Chapter 6 of my BizTalk Book: Professional BizTalk Server 2006: http://www.wiley.com/WileyCDA/WileyTitle/productCd-0470046422.html

Example

Consider the code whoen below which demonstates the creation of a BAM Activity (called Itinerary in this case), it populates a number of Activity Items, adds a reference to another Activity and finally adds a custom reference of a message body.

As you can see from the code many string literals representing the Activity Name and Activity Items are required throughout the code, any mistakes will cause runtime errors which can be frustrating especially when you have to undeploy and redeploy your BizTalk solution. Also, if you have a medium to large scale project then you will end up with many Activities and Activity Items which can require many lines of code to be manually created each time.

    string ItineraryActivityID = System.Guid.NewGuid().ToString();
    DirectEventStream des = new DirectEventStream("Integrated Security=SSPI;Data Source=.;Initial Catalog=BAMPrimaryImport", 1);
    des.BeginActivity("Itinerary", ItineraryActivityID);
 
    des.UpdateActivity("Itinerary", ItineraryActivityID, "Received", System.DateTime.Now,
        "Customer Name","Darren Jefford","County","Wiltshire","Total Itinerary Price",1285);
                
    des.AddReference("Itinerary", ItineraryActivityID, "Activity", "Flight", flightActivityID);
    des.AddReference("Itinerary", ItineraryActivityID, "MsgBody", "MessageBody", System.DateTime.Now.ToString(), myXmlMessageBody);
 
    des.EndActivity("Itinerary",ItineraryActivityID);

In contrast consider the code shown below which uses the Interary C# class created by this tool, each Activity Item can be set using class properties which are strongly typed and therefore checked at compile time instead of runtime. Simplified wrappers are provided around operations such as adding references and the ActivityID is stored internally once you've constructed the Itinerary class which saves you having to pass it each and every time. In short this code is far simpler and easier to maintain especially as Activities evolve during your development lifecycle, also for medium to large projects this code generation approach can save you having to write hundreds to thousands of lines of code!

    // Create a new Itinerary activity class, passing a GUID as the ActivityID
    DirectESApi.Itinerary itin = new DirectESApi.Itinerary( System.Guid.NewGuid().ToString() );
 
    // Begin the activity
    itin.BeginItineraryActivity();
 
    // Set activity items
    itin.Received = System.DateTime.Now;
    itin.CustomerName = "Darren Jefford";
    itin.County = "Wiltshire";
    itin.TotalItineraryPrice = 1285.00M;
 
    // Commit these changes to the database;
    itin.CommitItineraryActivity();
 
    // Add a link between this Itinerary activity and another flight activity that already exists
    itin.AddReferenceToAnotherActivity(DirectESApi.Activities.Flight, flightActivityID);
 
    // Add a message body to this activity (or any other data you require)
    itin.AddCustomReference("MsgBody", "MessageBody", System.DateTime.Now.ToString(), myXmlMessageBody);            
 
    // End the Activity
    itin.EndItineraryActivity();

TechEd 2007: Sessions and Book Signing

2007-06-03T15:51:00Z

I arrived in Orlando yesterday after a long old flight from London and should make the Party with Palermo tonight which looks like it'll be fun but as I'm presenting tomorrow it'll be a quiet night!

My two sessions are shown below, the second session being a very informal chalk/talk style session without any slides so if you want to know more about BizTalk Performance & Scalablity testing or have some questions, come along!

A book signing has been organised at the TechEd bookstore following my session on Monday, come along between 1815 and 1845 and Ewan and I will be signing copies!

My sessions this week are:

SOA306 - Building an Enterprise-Wide Instrumentation Solution Using the Microsoft BizTalk BAM Infrastructure
Monday 4th June: 1645 to 1800
Track(s): SOA and Web Services
Level: 300
Speaker(s): Darren Jefford

Business Activity Monitoring (BAM) is a powerful feature of BizTalk Server and is often marketed as allowing "the business" to understand what is happening within your BizTalk solution. BAM does this really well, but it can in fact be used in a variety of other ways which can deliver huge value to customers and address a number of issues they have with BizTalk based solutions and non-BizTalk based solutions.

In this session, we cover some of the fundamentals of BAM and detail how you can utilize BAM to collect a variety of information and produce a "tracking portal" which you can use to support your application, perform manual repair of messages, and generally observe your solution.

We also show how BAM is not just for BizTalk solutions and how it can be used to produce an enterprise-wide instrumentation solution that is highly scalable and flexible; we touch on the new Windows Workflow Foundation and Windows Communication Foundation (WCF) BAM Interceptor technology that enables data to be collected from Workflows and WCF services enabling a true end-to-end instrumentation solution.

SOA13-TLC - Microsoft BizTalk Performance Testing
Thursday 7th June: 1630 to 1745
Track(s): SOA and Web Services
Level: 400
Speaker(s): Darren Jefford

Testing is a critical area for any solution and BizTalk is no exception. In this session, we start by covering the foundations that you need to put into place before beginning any performance testing, covering the most common environment problems that cause problems during testing and highlighting a number of tools to make testing easier including BizUnit and LoadGen.

Then, we cover how you should monitor your solution during testing; highlighting the key performance counters that should be monitored and explaining how throttling works. We finish by covering a number of common symptoms that you're likely to experience during testing; we explain why these occur, how to spot them, and more importantly how to resolve them.

Professional BizTalk Server 2006: UK has stock!

2007-05-23T15:23:00Z

After what seems like a huge delay the usual UK retailers such as Amazon and Computer Manuals now have stock of the book! From what I've heard pre-orders have already started to drop through letter-boxes.

Enjoy, and please remember to put a review on Amazon if you get the chance!

For those of you going to TechEd in two weeks time, Ewan and I will be taking part in a book signing on Monday the 4th June between 6:15pm to 6:45pm at the TechEd book store following my breakout session.