Darren Jefford

New Blog!

Darren Jefford — Fri, 30 Dec 2011 20:26:11 GMT

I've decided to ressurect my blog but over on a new web-site that enables me to have more control over the look and feel. Please join me (if your still monitoring this blog) over at http://www.darrenj.co.uk

Software in the cloud: Cloud Workflow

Darren Jefford — Thu, 02 Oct 2008 12:31:30 GMT

As promised in my last posting let's take a look at the new "Workflow in the cloud" functionality offered in the July drop of the BizTalk Services

This cloud based workflow system leverages Windows Workflow and is hosted in our datacenter and extended as appropriate through "cloud-scale" runtime services and infrastructure. The idea being that you can run your workflows on our infrastructure which will scale to your throughput needs - subject to appropriate payments. These workflows will be durable and resilient to any failures - all of this will be transparent to you, simply deploy your workflows and away you go.

As it currently stands you only get one workflow type to use, a sequential workflow which is of course well suited to straight-forward predictable machine workflows. Within this workflow you get to use Activities but you can't however use the complete palette of WF activities that you get in the .NET framework.

Your restricted to a number of new cloud activities provided which are as follows: CloudHttpSend, CloudHttpReceive, CloudIfElse, CloudSequence, CloudServiceBusSend, CloudDelay and CloudWhile.

All of these do exactly what they say on the tin, the CloudHttpSend activity enables you to send an HTTP GET or POST to a remote URI, the CloudHttpReceive enables something to send data to a running workflow instance via an HTTP POST.

The CloudServiceBusSend activity may be a new concept to you as it refers to invoking a Service exposed via the Internet Service Bus using the relay service which I covered in this posting. This means you can have a WCF service hosted within your corporate network but still enable it to be invoked from parties outside of your corporate boundary provided they have the appropriate permissions. In the case of a cloud based workflow, it can invoke services hosted within a corporate boundary from a Microsoft data-center without having to punch new holes in your firewall but still maintain security.

So, let's take a look at how you can create your own workflow, deploy and execute it using the BizTalk Services release that you can all use. To start you'll need to create an account at http://biztalk.net and download/install the BizTalk Services SDK.

The first step is to create a new Sequential Workflow, all cloud workflows must be Sequential Workflows and XOML only. There is no way to have "code behind" files with these workflows, everything has to be expressed using a workflow, activities and as appropriate rules. Once this has been created you should consider deleting the automatically created .cs file to avoid you accidently adding code which can't be used.

Once you've done this you should see the new cloud based activities in your toolbox as shown below, if not you may need to right click the toolbox, choose "Choose Items.." and then ensure that the cloud prefixed workflow activities are selected. These new activities are held within the System.ServiceBus.Workflow assembly under the %PROGRAMFILES%\Microsoft BizTalk Services SDK\Assemblies directory.

For the purposes of this simple Hello World cloud workflow we'll just use one activity. The CloudHttpReceive activity will be used to demonstrate how we can communicate with a running workflow instance. You shoud have a workflow as shown below

The usual exclamation marks indicate that we have some configuration to do on this activity, for the cloudHttpReceive the only configuration we need to supply is a response that that will be returned as part of the remote HTTP post to our workflow, you can also configure the HTTP Status code to return as required. For the purposes of this workflow and the code you'll see in a bit, set the name of this activity to helloWorldHttpReceive and supply a response body of your choosing.

Now that our simple workflow is done let's deploy it to the Cloud Workflow infrastructure hosted by Microsoft, right now there isn't a Deploy addin to Visual Studio although there is an underlying Web Service. For now we need to perform a simple copy/paste deployment model.

Firstly we need to get the XOML representation of the workflow, a quick way of doing this is to right Click the workflow in Solution Explorer, Chose "Open With.." and then choose "XMl Editor", you can now copy the XOML representation into the clipboard.

We now need to deploy the workflow via the BizTalk Services website, browse to https://workflow.biztalk.net/ and supply your BizTalk Services credentials to view the workflow management portal as shown below.

Click "Manage Types" on the right hand side to manage the workflow types under your account, then click Add New to access the Workflow Deployment page which is shown below. For the purposes of this demo we'll call our Workflow "Hello World" and you can now paste the XOML representation into the supplied box.

Click Save changes and your cloud workflow is now ready to go! Note the "Rules" tab in the previous screenshot, this is where you can also provide any rules associated with your cloud workflow.

Now our workflow is created how can we go about kicking it off within the Microsoft datacenter? It's pretty straight forward once you know how to do it and the BizTalk Services SDK has a sample to help you out. Those of you familiar with BizTalk Orchestrations you'd expect that we could now perform a simple HTTP POST at a special URI and have a workflow instance created automagically? Not right now, I'm afraid - you have to manually crete an instance, then start it before you can submit an HTTP post, this as I understand it will be changed as the product teams progress.

The code to create, start and then HTTP POST to a workflow instance is shown below, it's all pretty straight forward and a lot of it is demonstrated in the SDK sample. The key thing in the code is how to target your HTTP POST of a running workflow instance. The URI is made up as follows:

http://workflow.biztalk.net/servicesHttp/YOURBIZTALKSERVICESUSERNAME/workflows/YOURWORKFLOWNAME/instances/YOURWORKFLOWINSTANCEID/HTTPRECEIVEACTIVITYNAME

And an example for my scenario here is as follows:

http://workflow.biztalk.net/servicesHttp/darrenj/workflows/HelloWorld/instances/83734f1c-d08f-4cc9-b396-b76ebdb60979/helloWorldHttpReceive

You'll need to add assembly references to your project for System.ServiceBus.dll, System.ServiceBus.Workflow.dll and System.ServiceModel.dll

[TestMethod]
public void TestMethod1()
{
    // Create a new Workflow Instance in the cloud, we then get the InstanceID back
   string instanceId = CreateWorkflowInstanceInTheCloud();

    // The Workflow Type
   string workflowTypeName = "HelloWorld";

    // The PlaceOrder Workflow starts with a cloudHttpReceive activity called "placeOrderHttpReceive"
    // We must target this directly when posting our Order information to the cloud workflow
   string httpActivityName = "helloWorldHttpReceive";

    // Build up the URL to directly target our newly created Workflow Instance and to post XML directly to the
    // waiting cloudHttpReceive activity

   Uri serviceUri = new Uri(
        string.Format(WorkflowClientConfig.HttpUri + "{1}/workflows/{2}/instances/{3}/{4}",
        WorkflowClientConfig.WorkflowHostName, _bizTalkServicesUsername, workflowTypeName, instanceId, httpActivityName));

    string httpPostBody = "<HelloWorldWorkflow><Message>Hello World</Message></HelloWorldWorkflow>";

    // Prepare to perform an HTTP POST to the waiting workflow
   HttpWebRequest request = (HttpWebRequest)WebRequest.Create(serviceUri);
    request.Method = "POST";
    request.ContentType = "text/xml";
    request.ContentLength = httpPostBody.Length;
    // Add the special header which contains our STS token retrieved from the cloud
   request.Headers.Add("X-MS-Identity-Token", GetTokenFromCloudSts());

    using(StreamWriter writer = new StreamWriter(request.GetRequestStream()))
    {
        // Write the OrderInformation XML representation into the stream
       writer.Write(httpPostBody);
       writer.Flush();

        // Wait for a response from the Cloud Workflow, the cloudHttpReceive activity is configured to respond with a fixed
        // Response if succesful;
       HttpWebResponse response = (HttpWebResponse)request.GetResponse();

        using(StreamReader reader = new StreamReader(response.GetResponseStream()))
        {
            string returnMessage = reader.ReadToEnd();

            Trace.WriteLine("Response message from the cloud is: " + returnMessage);

            // For some reason all responses from the cloudHttpReceive activity are wrapped into a <string> element
            Assert.IsTrue(returnMessage.Contains("Hello World Response"));
        }
    }
}

private string GetTokenFromCloudSts()
{
    // Using a tokenised URL let's build up a valid URL given the supplied BizTalk Services credentials
    // This URL when invoked will return an authentication token we can then use to perform administrative operations on
    // Our Workflow Configuration (Create, Start, etc.)
    string tokenUrl = string.Format(
            "https://{0}/issuetoken.aspx?u={1}&p={2}",
            WorkflowClientConfig.StsHostName,
            _bizTalkServicesUsername,
            _bizTalkServicesPassword);

    // Create an HttpWebRequest
    HttpWebRequest tokenRequest = (HttpWebRequest)WebRequest.Create(tokenUrl);
    Trace.WriteLine(tokenRequest.RequestUri);

    // No Request to send, just invoke the URL and get the authentication token
    HttpWebResponse tokenResponse = (HttpWebResponse)tokenRequest.GetResponse();

    // Get the token from the Response Stream
    byte[] tokenBody = new byte[500];
    int tokenBodyLength = tokenResponse.GetResponseStream().Read(tokenBody, 0, 500);

    // Convert to a UTF8 representation which we can then use in subsequent requests
    string authenticationToken = Encoding.UTF8.GetString(tokenBody, 0, tokenBodyLength);
    Trace.WriteLine(String.Format("Authentication Token: {0}", authenticationToken));

    return authenticationToken;
}

private string CreateWorkflowInstanceInTheCloud()
{
    // New up an instance of the provided System.ServiceBus.Workflow.WorkflowClient which provides a wrapper
    // around the cloud workflow SOAP Web Servics        
    WorkflowClient workflowClient = new WorkflowClient();

    // Provide BizTalk Services credentials
    workflowClient.CredentialType = TransportClientCredentialType.UserNamePassword;
    workflowClient.Credentials.UserName.UserName = _bizTalkServicesUsername;
    workflowClient.Credentials.UserName.Password = _bizTalkServicesPassword;

    workflowClient.Open();

    // Parameters can't be supplied in R12 of Cloud Services so we this empty
    // A shame as it would enable us to pass the OrderInformatoin directly
    Dictionary<string, object> parameters = new Dictionary<string, object>();

    // Create a workflow within the BizTalk Services username namespace 
    // The Workflow Type name is "HelloWorld"
    // No parameters in R12
    // Retrieve an authentication token from the BizTalk Services Cloud STS
    string instanceId = workflowClient.CreateWorkflowInstance(_bizTalkServicesUsername, "HelloWorld", parameters, GetTokenFromCloudSts());
    Trace.WriteLine("Workflow Instance Created:" + instanceId);

    // Once a workflow is created it must be explicitly started otherwise it won't active any activity shapes
    // In R12 there's no ability to create/start a workflow based on an external activity action such as an Http Send
    // The workflow instance was created within the BizTalk Services username namespace
    // The workflow Type name is "HelloWorld"
    // The instanceId was returned by the previous CreateWorkflowInstance call
    workflowClient.StartWorkflowInstance(_bizTalkServicesUsername, "HelloWorld", instanceId);
    Trace.WriteLine("Workflow Instance Started:" + instanceId);

    // Our work is done, close the WorkflowClient
    workflowClient.Close();

    // Return the InstanceId so we can identify the WorkflowInstance we've just created when we perform an HTTP post to it next
    return instanceId;
}

All pretty straight forward, and it's a bit more complex then it should be due to the requirement to create and start a workflow instance before being able to perform an HTTP post but this should come in time.

If you navigate back to the https://workflow.biztalk.net site then you can review running workflow instances and check they've completed successfully.

That's it for this post, this post has got rather large so I'll do another post on calling ServiceBus services via the cloudServiceBusSend activity and also detail some limitations in this release of Cloud Workflow (it's the first release for the team so don't expect everything to be there straight away!). I'll also cover a handy little ISBTraceTool that I put together to aid with visualizing cloud workflow activity.

More soon!

Dublin!

Darren Jefford — Thu, 02 Oct 2008 00:43:01 GMT

This announcement came sooner then I expected, I'd assumed PDC was going to be the first time Dublin was mentioned in public! The most detailed information before the PDC can be found here.

A number of internal people in the field (myself included) have been working with the product team for a number of years now to help shape some of these new technologies and ensure they will address real-world customer scenarios based on our customer experience. It's great to be able to start discussing what's coming down the line, although the real detail won't be available until the PDC.

So Dublin! Windows Server is our application server today and we're now expanding it's capabilities to deploy and manage .NET based applications (using WCF and WF) - no more roll your own!

If you appreciate the way BizTalk Server provides enterprise capabilities to host your Integration solutions today, you'll see a lot of similarities in how these hosting capabilities are being introduced in "Dublin" for WCF and WF. For those who might have shied away from WF because of having to roll their own infrastructure, this is great news!

Note that in the press-releases that BizTalk is formally being referred to as an integration server, e.g:

Q: Will "Dublin" work with BizTalk Server's enterprise connectivity services?
A: Yes. The integration server and application server workloads are distinct but complementary; customers want to be able to deploy them separately as needed to support their distinct requirements. For example, customers that don't need the rich LOB or B2B connectivity provided by an integration server, will deploy the Windows Server application server to host and manage middle-tier applications. Likewise, customers that need to connect heterogeneous systems across an enterprise, but don't need to develop and run of custom application logic, will deploy BizTalk Server. When customers need both capabilities, "Dublin" and BizTalk Server will work together nicely.

BizTalk is by no means "dead", in fact Microsoft committed to future versions including BizTalk Server 2009 recently for the integration server workload, BizTalk = Integration, Dublin = Application.

So if you want to expose [WF] workflows via [WCF] services but ensure performance and scalability (up to enterprise scale), you can now do this without having to write the code required to host these apps on Windows Server. Ensuring performance and scale of WCF services and WF is hard to do today, hence it's not done very often at least in my experience and sometimes causes a tendency to twist BizTalk into doing something it wasn't necessarily designed to do which causes problems of their own (coupling Web Sites/UI's directly to BizTalk for synchronous processing springs to mind).

We don't want customers in this situation to be forced into writing huge amounts of hard plumbing code to achieve this, we need a server product to do this for you, which is where Dublin comes in. Note some of the server features announced which will be familiar to BizTalk developers (content based routing, compensation, etc.).

If however you need to use the extensive adapter support, B2B, EDI, RFID, BAM, BizTalk Mapper style features then you'll still be wanting to use BizTalk. Both products will work together seamlessly through the WCF communication options so you can combine as appropriate.

Remember though that a number of BizTalk adapters have been re-written as WCF bindings and are available through the BizTalk Adapter Pack which offers some key LOB adapters. A new SQL adapter is in the works along with new Oracle adapters which you can find information on here. As these new adapters are exposed as WCF bindings you can leverage them with WCF and Dublin.

Dublin will also be the first and best consumer of the Oslo modeling platform, they'll be more detail on this at the PDC but trust me - this is going to blow your minds!

WCF and WF get a big makeover as already announced, we get new workflow types in WF and an extension library of Activities out of the box. If you want to call a SQL stored procedure, why write half a page of code when you can just configure a database activity? Combine that with the 10x improvement in performance and things are looking good! Imagine a world when a typical software solution can be implemented (modeled) exclusively through a workflow and out-of-the-box activities? ;-)

Notice also the subtle new feature in WF, "persistence control". Low Latency scenarios with BizTalk are achievable but has to be carefully designed, we don't currently have the ability in BizTalk to control when an Orchestration persists but imagine if we had this feature in WF - low-latency potentially becomes easier to achieve ;-)

That's enough for now, once more detail starts to emerge I'll post some more information and will also work internally on locking down some clear "where to use what" style scenarios for BizTalk and Dublin using some real-world customer scenarios.

Exciting times!

Professional BizTalk Server 2009?

Darren Jefford — Mon, 29 Sep 2008 13:34:00 GMT

I've been looking into the possibility of updating the highly successful Professional BizTalk Server 2006 book to take into account new features and capabilities in BizTalk Server 2006 R2 and the recently announced BizTalk Server 2009 product.

A fair bit has changed to warrant new chapters and updating of various chapters but of course the main body of content is still accurate and will be brought forward into any new edition.

The current book is of course still absolutely usable for both BizTalk Server 2006, R2 and even 2009 as the main engine, features and best practices are unchanged.

I've listed some of my thoughts around what needs to be covered to expand the reach of the book given the new version but would love to hear of topics and sections that you would like to see added if you've already read the current edition.

Please send me a mail with your thoughts via my blog site or add comments to this posting, I'd really appreciate your views on what we should consider.

I haven't got formal commitment from Wiley yet as I'm still at the Table of Contents stage, here are my outline thoughts as it currently stands.

Technology Primer - Update

· Cover WCF basics

· Cover the penalty of XmlDocument vs Serializable Classes vs XPath with real-examples of overhead

BizTalk Architecture - Update

· Emphasise further how important testing is, don’t expect performance and how critical the performance of supporting systems is. Detail techniques for isolating and testing performance of supporting systems

· Cover any changes with BTS2009

· Potentially position BTS2009 vs Oslo at the architectural level

Adapters - Update

· Include drill-down on WCF Adapter topics

· Overview

· Different WCF-* adapters

· Walkthrough

· Demonstrate how a custom binding can be developed using the WCF Adapter SDK (null adapter?)

· Cover the new WCF Adapter bindings (SQL, MQS, etc)

Business Activity Monitoring - Update

· Provide implementation of BAM Latency Timer pipeline component

· Provide more real-world and common questions/issues guidance

Business Rules Engine - Update

· Position BRE vs the WF rules engine and futures

RFID - New Chapter

· Drill-down chapter into the RFID features of BizTalk complete with real-world examples

Testing - Update

· Provide more real-world “case studies” of how it’s been done right and the benefits..

· New BizUnit features (Excel)

· Update with any new VS2008 (and maybe future release features)

Performance and Scalability - Update

· PAL for analysis of performance logs

· Discuss SAN technology improvements

· ‘n’ step plan for identifying where bottlenecks lie within a BizTalk rig

· Expand throttling “plain” English guide, step-by-step diagnosis guide

Low Latency - Update

· Cover new options such as NetTcp and NetNamedPipe WCF bindings

· Include real-world perf-test results, step by step guide

Administration - Update

· Virtualization (HyperV)

· Best Practice Analyser

· MSBuild

· SQL2008 differences

BizTalk Best Practices - Update

· Add further best practices, small and large – with evidence as to why

Cloud Services (BizTalk Services, SSDS, etc.) - New Chapter

· Overview

· Connectivity

· Identity

· Cloud Workflow

· SSDS

· Cloud Commerce scenario code and walkthrough

First look at Oslo futures - New Chapter

· More soon ;-)

Software in the cloud: The Relay Service

Darren Jefford — Wed, 24 Sep 2008 16:39:00 GMT

I’m now back from a trip to Seattle a few weeks back where I attended a huge week-long Microsoft internal technical training event. I presented three times during the week, two of the presentations were “Software in the cloud” sessions which received fantastic feedback.

The premise of these sessions was to paint a view of how software solutions may look in the future if they were to leverage cloud based services such as SQL Server Data Services (SSDS) and BizTalk Services (Cloud Workflow, Identity and Connectivity).

I presented a fictional but realistic architecture using today’s technologies and discussed the challenges, I then presented a forward view on how you might approach the same problem using cloud based technologies and how this solves a number of challenges today and opens up completely new opportunities which for the most part aren’t possible today.

I’m going to present this scenario and the resulting solution in a series of blog posts, this being the first.

Before we get started I want to introduce the BizTalk Services Relay which by itself is incredibly powerful but often overlooked. There is an SDK to download and play with via http://www.biztalk.net

So, if I have a Service offering today, say Credit Scoring functionality, it’s hard to enable business partners to invoke this service, especially if your organisation doesn’t host things out on the internet regularly.

You’ve got any number of options generally involving any combination of leased lines, VPNs, proxy configuration, etc. The key point here is that you must put in infrastructure and/or specific configuration for each business partner that wishes to leverage your service, and of course vice versa, each business partner will have to do the same to enable messages to pass out of their corporate boundary.

So, imagine a technology that enabled you to avoid all of this infrastructure/configuration whilst still maintaining security and integrity of messages passing between the organisations?

This is where the Relay comes into play; the first step is for your WCF service to register it’s endpoint with the cloud based relay service, a valid BizTalk Services account will be required to register an endpoint under the users namespace. This step is shown below as step number 1.

The endpoint address will use a specific “service bus” prefix, e.g: sb://connect.biztalk.net/services/darrenj/OrderService/

Once an endpoint has been registered with the relay service it must be kept alive otherwise the socket could of course timeout. This is done through a series of “ping” messages automatically passed under the covers between the relay service and the endpoint.

Once an endpoint is registered a client can then invoke the service, within Visual Studio you can simply type in the sb:// prefixed address via the usual Add Service Reference dialog and the proxy will be automatically created and configured – neat!

The proxy can then use the contract as usual, once invoked the message will be passed to the endpoint registered in the cloud (shown as number 3 in the diagram above) which will in-turn pass the message on to the service (shown as number 4 in the diagram above), the reverse then happens with the response (shown as number 4 in the diagram above).

In essence this is a straight forward message relay pattern with all messages passing via the relay, enabling two parties to exchange messages where they otherwise wouldn’t have been able to.

This works fine, but it would be nicer (and probably quicker) if we could enable both parties to communicate directly but the firewalls won’t allow this. However the relay service knows quite a bit about the two parties.

When the service registered itself with the relay, the relay can see which dynamic port number the hosting organisation’s firewall is using for this communication session. When a client communicates with the relay, the relay can again see the dynamic port number being used.

By sharing the port numbers with each party the relay can effectively step out of the way and enable both parties to communicate with each other after this initial port handshaking, this is depicted below and is explained in better detail in this great article

This process is called NAT traversal and is used by many things today including MSN Messenger, Groove, Skype, etc. Some firewalls may not allow NAT traversal at which point the relay will fall back to the original relay pattern. As long as both parties can see the internet and therefore the relay service then we’re in business!

This is a big step forward from today where you have to provision and configure network infrastructure such as Proxies, Firewalls, Private Network Links, VPNs, etc. whenever you wish to expose invoke services between organisations.

With this relay service you can break down your corporate barriers to communicate across organisations and through the relay you can also utilise non-durable publish/subscribe messaging effectively publishing business “events” which others can subscribe to.

So how about security? Firewalls are there for a reason and you typically aren’t allowed to punch holes through them. Access to any services exposed through the relay is restricted through the BizTalk Services provided implementation of a Security Token Service (STS) and messages can be secured as you require using encryption of the payload and digital signatures.

So, it must be hard to host a service through the relay? Well not really, it’s just a bit of WCF configuration on your service an example of which is shown below:

<system.serviceModel>

<binding name="default" allowBrowsing="true" sendTimeout="01:00:00"

connectionMode="RelayedDuplexSession">

</binding>

</relayBinding>

</bindings>

<endpoint

address=""

binding="relayBinding"

contract="PaymentService.IPayment"

behaviorConfiguration="relayBehavior"

bindingConfiguration="default"/>

</clientCredentials>

</transportClientEndpointBehavior>

</behavior>

</endpointBehaviors>

</behaviors>

</system.serviceModel>

You don’t need to include the custom endpointBehavior in all cases, if you don’t then CardSpace will pop up to authenticate you and validate that you have permission to register a service at the specified endpoint – not ideal for non-interactive services J

Hopefully that’s given you a good flavour of what the relay service can do, it’s a fantastic piece of technology and opens up the opportunity to break down the enterprise SOA barriers whereby your services can only be consumed or exposed within your corporate boundaries.

To my mind, this opens up some great new opportunities for solutions moving forward, as we’ll see in my next few blog posts J

Cloud Services

Darren Jefford — Thu, 17 Jul 2008 01:03:00 GMT

CSD have just announced Release 12 (R12) of Codename BizTalk Services, Clemens has a good low-down on what’s new here. Remember BizTalk Services != BizTalk. BizTalk isn’t used in the cloud or the client – it’s just a branding thing.

The Connectivity service (relay) is an amazing piece of technology breaking down corporate boundaries and enabling true communication between organisations and people despite all of the firewalls and proxies put in place. It’s been available for a while but I’ll do a blog post on my take soon in case you haven’t seen it in action.

The big news in this release is the addition of cloud based Workflows, we are now hosting Windows Workflow in our datacenter which can run your own Sequential Workflows. You can’t use the full toolbox of usual WF activities though but a cut-down selection that the team have supplied.

These workflows as Clemens discusses on his blog are aimed towards the “service orchestration”, so a business process that you have can be modelled using a workflow and invoke any number of Internet Service Bus (ISB) services and also perform HTTP requests, workflows can also be communicated with using HTTP.

The current list of cloud activities are as follows

· CloudHttpSend

· CloudHttpReceive

· CloudIfElse

· CloudSequence

· CloudServiceBusSend

· CloudDelay

· CloudWhile

Only XOML workflows are allowed (so no code behinds here!) and if you need to do any coding you must push this into a Service that your workflow can invoke using the CloudServiceBusSend or Http activities.

You must copy/paste the XOML from Visual Studio into a HTTP form available when you sign into www.biztalk.net, clunky but effective for now!

Workflows can’t be activated through say an HTTP post but instead an instance has to be first created and then started before any activities will be executed, so if you had a workflow with a CloudHttpReceive activity at the top you would have to follow those previous steps before you can send an HTTP post to it.

There is a handy WorkflowClient class provided that means you can automate the creation and starting of a workflow instance, I’ve got a unit test that does this before communicating with the created workflow instance.

I’ll post a little sample shortly to get you going straight away but in the meantime check out the samples provided in the BizTalk Services SDK.

Long time, no blog!

Darren Jefford — Thu, 17 Jul 2008 00:50:00 GMT

Yes, It’s been quite a while since my last blog post! A lot has been happening internally including a job move for me to the Solution Architecture team in MCS as a Solution Architect from the fantastic Application Development Consulting (ADC) team.

I still retain my strong technical focus you’ll be pleased to know and I don’t think I’ll be getting away from BizTalk anytime soon J

I’ve been doing a lot of work in the “fluffy cloud” space in recent months and will be presenting a rather complex session at our internal technical event later this month.

This work has included pulling together a fictional scenario which demonstrates stitching together SQL Server Data Services (SSDS), Cloud Services (Connectivity, Identify and Workflow) and good old BizTalk to address challenges with “today’s” technologies and enable new possibilities.

It’s been an interesting experience, not everything can be done right now as both SSDS and the Cloud Services are still in the CTP phase and missing certain features but the future looks very interesting.

More on this over the next few days and weeks plus after my presentation and demo is over I’ll post the code and more details on my fictional scenario so you can take a look and see what I've been tinkering with.

My other pet project has been a BizUnit DSL tool to enable graphical visualization and editing of BizUnit tests, again more soon hopefully

64bit support for the BizTalk Orchestration Profiler and BizTalk Documenter

Darren Jefford — Wed, 28 Nov 2007 14:36:00 GMT

A number of you have contacted me in recent months to report that the BizTalk Orchestration Profiler and BizTalk Documenter didn't run on 64bit machines, this seems to have been down to compiler settings which I've now resolved.

I've uploaded a planned release for both tools to codeplex - if you've got a 64bit machine I'd appreciate it if you could install and test the tool to see if it works on your servers? Please let me know either way - I'll then make the releases as the default released versions once I've got validation.

You can find the releases at the following URLs:

http://www.codeplex.com/BizTalkDocumenter/Release/ProjectReleases.aspx?ReleaseId=8689

http://www.codeplex.com/BiztalkOrcProfiler/Release/ProjectReleases.aspx?ReleaseId=8688

I've done some tidying up across the entire codebase to make it easier to support and also reworked and re-released the Word output option for the BizTalk Documenter thus allowing Word documents to be generated as well as CHM files. Let me know if the Word output doesn't work as expected for you when compared to CHM output.

Please keep any feature suggestions coming...

How everyone should test their BizTalk based solutions: Presentation

Darren Jefford — Thu, 01 Nov 2007 21:06:00 GMT

I've managed to find somewhere to store the presentation deck I delivered yesteday and you can now download it from here. I've also added the slides I had to cut from the presentation to enable it to fit within the SOA & BPM conference breakout length - these cover the throttling semantics at a high level and detail some common symptoms you might see during performance testing and what you should look at to resolve them.

There's of course more detail in the book! Enjoy.

Codename “Oslo”

Darren Jefford — Thu, 01 Nov 2007 02:43:00 GMT

Microsoft has just made the first public announcement of “Oslo” at the SOA and BPM conference, at a high level “Oslo” is an overarching initiative across multiple products and Microsoft divisions, in fact the first release as announced today will be made up of BizTalk Server “6”, BizTalk Services “1”, Visual Studio “10”, System Center “5” and .NET Framework “4.0”. Therefore Oslo != Just BizTalk.

Official press release information can be found here and here along with a good overview by Directions on Microsoft which you can find here, but here’s my take for what’s it worth! J

Oslo is driven out of Connected Systems Division (CSD) which is the almagamtion of the BizTalk, WCF, WF, ASMX, MSMQ, .NET Remoting, etc. teams and has around 1000 people, Oslo as highlighted above is a huge program of work within CSD and across multiple divisions within Microsoft (AD, System Center, Developer Division, etc.). It's a huge investment.

Two key themes for Oslo were annouced during the keynote, Models (Making models a mainstream part of development) and Services (Extending services from the client to the cloud).

Those of you familiar with BizTalk Orchestration and Windows Workflow or even “Whitehorse” style designers such as those created using the DSL Tools or the Class Designer in VS are well versed with how a modelling tool works effectively enabling you to view and modify an underlying “thing” with an abstracted graphical language.

There has been lots of modelling attempts across the years across the industry; two things stand out for me as to why. Firstly the model is typically a representation at a point in time, you draw a nice pretty picture either for reference purposes or to then generate some physical representation like code.

Sometimes you get round-tripping which enables the model to be up-to-date but this becomes increasingly hard in most scenarios to the point of not being possible so invariably the model becomes out of date with the actual reality and therefore its value is minimal.

Secondly you have lots of “models” within a typical project, perhaps you have a business process modelled using BizTalk Orchestration, the physical datacenter modelled using the VS Team Architect tools and operational management within System Center. Nothing is shared between these models and therefore everyone has their own version of reality – often out of date and there’s no knowledge transfer between the various software development lifecycle roles.

In short, the modelling vision is a great thing but has a number of serious limitations today which limits its values to small isolated parts of your solution, imagine though having one model to describe your solution that could be used across your entire solution development lifecycle (Business Analyst, Developer, Tester, Operations, etc.).

This model of course needs to be rendered into different views for each of these stakeholders, a business analyst has a different “view” requirement to a developer for instance.

All of this is fine but don’t we end up with a stale description of a solution? Not in Oslo as the model is executed; therefore the model has to be up-to-date as its being used to execute the solution.

These models then have to “live somewhere”, firstly to make them broadly discoverable by all the stakeholders involved in the development process and secondly to enable them to be executed by some form of host. This is where the Metadata repository comes in which leverages SQL Server.

The viewing/editing experience will be provided through a new tool offering an “office like experience”, this will be able to interrogate the repository and retrieve/persist models. Precise technical detail of how all this hangs together isn’t available right now but more information will be available publically in due course.

So Modelling is a big focus for Oslo, basically taking the modelling concept but making it mainstream and used across the entire development lifecycle and thus connecting up all roles.

The next big focus for Oslo is around services and messaging; BizTalk today provides an incredibly powerful and flexible messaging platform through Adapters, Pipelines and the MessageBox. We have publish and subscribe, correlation, transformation all in the box.

Through the use of adapters we can easily integrate with a variety of technologies provided by different vendors running on different platforms but this typically speaking is all about integration/messaging within your corporate boundaries.

You can of course implement B2B scenarios where you can do cross business integration but any communication between B2B partners typically requires specific and dedicated communication links such as a leased line, VPN, SSL, etc. This is expensive and normally only implemented by the larger organisations.

Oslo is set to address this and more through the use of what Microsoft is calling a “Internet Service Bus”, imagine being able to connect two organisations together without requiring anything more than a internet connection, no need for leased lines, NAT traversal?

This in part is offered today through the BizTalk Labs preview of the BizTalk Connectivity Services, this offers the concept of a relay which can patch two applications together even if they’re behind their own respective firewalls and would normally have no way to directly communicate with each other.

The relay offers NAT traversal technologies and once it’s patched you together it can then completely step out of the communication path enabling you to communicate together. Now this might not see that useful on the surface but it opens up a whole new breed of solutions for the small-medium market but of course any size of user or customer.

In short the relay enables communicate between users and applications even though there is no direct communication link between the two – of course this NAT traversal solution has been down before in silos by software such as Groove and MSN Messenger. The key here is that the Connectivity Services are doing all the hard NAT stuff for you and the service is hosted by Microsoft in the cloud.

A Security Token Service (STS) is also supplied through the BizTalk Labs preview which offers an identity provider to enable you to offload identity and authentication functionality. Moving forward there is a commitment to provide workflow hosting.

All of these services are cloud hosted meaning that a Microsoft datacenter somewhere provides the hardware, resilience, software, etc. to expose these services, you don’t need to provision datacenter space, hardware, etc. but instead rely on Microsoft of another third party. How this will be charged for hasn’t been agreed at this time.

So combining models and these cloud based services is interesting as suddenly you can develop your solution using Oslo tools and then make a deployment-time decision to host within your datacenter or to host within the cloud. This is an interesting choice for organisations that currently maintain large datacenters and have to worry about monitoring, resilience and the like. It also opens up completely new possibilites about how users/services can be contacted regardless of where they are, as long as they have a internet connection and can "see" the cloud they can communicate regardless of what network infrastructure is in the way.

So, all of that aside – your probably thinking about BizTalk today? Should you continue to use it, will any new investment in BizTalk Orchestrations, Rules Engine, etc. be redundant in a few years time? There’s also a slightly scary statement in the Directions on Microsoft report that advises that you may wish to limit new development using BizTalk Server.

I don’t speak for the product team, but can say that enabling you to run BizTalk Server 2006 artefacts within the Oslo release will absolutely be supported – Microsoft can’t turn away from the significant investment that customers have made today and aren’t planning to.

So, in short carry on using Biz Talk. Oslo is a way away yet and there’s an incredible amount you can do with our technology stack today (BizTalk Server 2006 R2, Office 2007 Sharepoint, WCF, WF....)

That’s it for now, will provide some more thoughts and info as time goes on.

How everyone should test their BizTalk based solutions presentation links

Darren Jefford — Thu, 01 Nov 2007 02:32:00 GMT

Thanks to everyone who came to my presentation today, I was humbled by the huge turnout! The presentation deck will be broadly shared to attendees which contains hyperlinks to all the tools showcased but in the meantime the links are below if you want to get started right away. I'll look to share the entire deck via the blog shortly if I'm allowed!

A new release of LoadGen has very recently been made available which I didn't mention today, this adds WCF support for injecting load amonst other things.

PerfMon Counter spreadsheet
http://media.wiley.com/product_ancillary/22/04700464/DOWNLOAD/Chapter9_Code.zip

Orchestration Profiler
http://www.codeplex.com/biztalkorcprofiler

BizTalk Documenter
http://www.codeplex.com/biztalkdocumenter

LoadGen 2007 (now includes WCF, Sharepoint 2007 support amongst others)

http://www.microsoft.com/downloads/info.aspx?na=22&p=1&SrcDisplayLang=en&SrcCategoryId=&SrcFamilyId=&u=%2fdownloads%2fdetails.aspx%3fFamilyID%3dc8af583f-7044-48db-b7b9-969072df1689%26DisplayLang%3den

Old version of LoadGen 2004 (recommend you use the 2007 release moving forward)
http://go.microsoft.com/fwlink/?LinkId=59841

BizUnit
http://www.codeplex.com/bizunit

Microsoft SOA & BPM Conference

Darren Jefford — Mon, 29 Oct 2007 19:51:00 GMT

I've just arrived in Seattle for the Microsoft SOA & BPM conference that kicks off on Tuesday 30th October in Seattle, I'm presenting a best-practice session on Wednesday afternoon titled "How everyone should test their BizTalk Server based solutions", it's a great deck with some compelling demos that went down very well at a internal technical conference earlier this year. If we can get BizTalk solutions tested properly - the world will be a much better place!

More as the week goes on, should be some interesting presentations....... In the meantime Sam Gentile has a great running commentary of the week.

Orchestration Profiler for BizTalk Server 2006 and BizTalk Documenter

Darren Jefford — Thu, 05 Jul 2007 14:53:00 GMT

Hi,

I've recently taken ownership of the source for the BizTalk Orchestration Profiler and BizTalk Documenter tools that Jason Birth originally developed, the documenter has worked fine with BizTalk Server 2006 but the Profiler was broken in a number of places. I have just finished a new build of the Profiler which now works with 2006 and fixed a few other niggles, both tools are now available and I'm open to feature suggestions for both tools moving forward.

If you haven't seen or used the tools before then do take a look - they are invaluable in many circumstances and I for one use them one way or another on every BizTalk engagement, the Profiler is here: http://www.codeplex.com/BizTalkOrcProfiler and the Documenter is here: http://www.codeplex.com/BizTalkDocumenter

Information on both tools is copy/pasted below from the respective CodePlex sites, enjoy!

BizTalk Orchestration Profiler

Creates CHM report files illustrating the level of coverage for specified BizTalk orchestrations. This tool can be run to gain a consolidated view of orchestration tracking data for a specified period of time to help developers get an idea of how their orchestrations are being processed and how much coverage they are getting in their testing. In addition to simple coverage information the data presented helps to identify latency and code path exceptions by highlighting long running and error prone orchestration shapes which is key to effective performance testing.

The image below shows a sample of the coverage summary, at a glance you can see the percentage of overall coverage acheived upon an individual orchestration, if all of the shapes are touched as part of your orchestration then you will acheive 100% coverage, common reasons for not covering all shapes with an orchestration are not running tests that stress the error paths of your solution. This coverage summary also provides you metrics around execution duration of your orchestration easily identifying orchestrations that take a long time to execute.

The image below shows the orchestration coverage in detail, shapes highlighted in green have been executed as part of your last test pass and any shapes not executed will be highlighted in red. This enables you to easily understand what parts of your solution have been tested and which parts have not and adjust your testing as necessary.

As part of the profiler report you can see the "Top 10 Least Successsfull" shapes and the "Top 10 Longest Running" shapes, understanding where time is being spent inside your orchestration is key to effective performance testing and has been proven on a number of instances to pinpoint poor orchestration design and coding which when resolved has dramatically increased overall performance. The key is to understanding exactly where time is being spent and this tool offers a no-code view into your running system.

BizTalk Documenter

Creates compiled help files for a given BTS 2006 installation. This tool can be run on an ad-hoc basis using the UI or from the command line as a post build/deploy task to create a compiled help file describing a BTS 2006 installation. It will compile: BTS Host configuration, Send/Receive port configuration, Orchestration diagrams, Schema and Map content, Pipeline process flow, Adapter configuration, Rule engine vocabularies and policies, More… and publish them as compiled help files. Optionally you can embed custom HTML content and custom descriptions for all BTS artifacts to produce a more customized look and feel to the CHM output

This tool can be run on an ad-hoc basis using the UI or from the command line as a post build/deploy task to create a compiled help file describing a BTS 2K4 installation. It will compile:

• BTS Host configuration
• Send/Receive port configuration
• Orchestration diagrams complete with any custom code
• Schema and Map content
• Pipeline process flow
• Adapter configuration
• Rule engine vocabularies and policies
• More…

and publish them as compiled help files or Word 2003 XML. Optionally you can embed custom HTML content and custom descriptions for all BTS artifacts to produce a more customized look and feel to the CHM output.

The screenshow below shows some of the information collected about an Orchestration, you can see what variables, maps, messages, ports, correlation sets, etc. that are used by a given Orchestration.

The screenshot below shows further orchestration information which is collected, in this case we can see all of the inline code written by the developer for the orchestration, useful for code reviews and understanding a solution.

GenerateTypedBamApi

Darren Jefford — Wed, 06 Jun 2007 03:47:00 GMT

Long overdue but I've finally got around to finishing up the latest release (1.1) of the GenerateTypedBamApi tool. You can find the new version here, if your not familar with the tool read on!

The GenerateTypedBamApi command line tool enables you to take a BAM Observation model represented as a Excel Spreadsheet and generate a set of strongly typed C# classes which you can then use to create and populate BAM Activities.

The native BAM API is loosely typed and therefore requires Activity Names and Activity Items to be supplied as string literals, this can be brittle especially as the observation model evolves over time and any typos, etc. will lead to runtime errors instead of compile time errors.

A strongly typed class is created for each Activity with properties for each Activity Item along with helper methods to write Activity Items to an Activity, Add References, Custom References (e.g. Message Bodies) and Continuation.

This tool uses a XSLT transform to turn the XML representing the BAM Observation Model into C# code.

For further information on how to use BAM and why you might want to use a BAM API approach over the in-bult graphical Tracking Profile Editor please Chapter 6 of my BizTalk Book: Professional BizTalk Server 2006: http://www.wiley.com/WileyCDA/WileyTitle/productCd-0470046422.html

Example

Consider the code whoen below which demonstates the creation of a BAM Activity (called Itinerary in this case), it populates a number of Activity Items, adds a reference to another Activity and finally adds a custom reference of a message body.

As you can see from the code many string literals representing the Activity Name and Activity Items are required throughout the code, any mistakes will cause runtime errors which can be frustrating especially when you have to undeploy and redeploy your BizTalk solution. Also, if you have a medium to large scale project then you will end up with many Activities and Activity Items which can require many lines of code to be manually created each time.

    string ItineraryActivityID = System.Guid.NewGuid().ToString();
    DirectEventStream des = new DirectEventStream("Integrated Security=SSPI;Data Source=.;Initial Catalog=BAMPrimaryImport", 1);
    des.BeginActivity("Itinerary", ItineraryActivityID);
 
    des.UpdateActivity("Itinerary", ItineraryActivityID, "Received", System.DateTime.Now,
        "Customer Name","Darren Jefford","County","Wiltshire","Total Itinerary Price",1285);
                
    des.AddReference("Itinerary", ItineraryActivityID, "Activity", "Flight", flightActivityID);
    des.AddReference("Itinerary", ItineraryActivityID, "MsgBody", "MessageBody", System.DateTime.Now.ToString(), myXmlMessageBody);
 
    des.EndActivity("Itinerary",ItineraryActivityID);

In contrast consider the code shown below which uses the Interary C# class created by this tool, each Activity Item can be set using class properties which are strongly typed and therefore checked at compile time instead of runtime. Simplified wrappers are provided around operations such as adding references and the ActivityID is stored internally once you've constructed the Itinerary class which saves you having to pass it each and every time. In short this code is far simpler and easier to maintain especially as Activities evolve during your development lifecycle, also for medium to large projects this code generation approach can save you having to write hundreds to thousands of lines of code!

    // Create a new Itinerary activity class, passing a GUID as the ActivityID
    DirectESApi.Itinerary itin = new DirectESApi.Itinerary( System.Guid.NewGuid().ToString() );
 
    // Begin the activity
    itin.BeginItineraryActivity();
 
    // Set activity items
    itin.Received = System.DateTime.Now;
    itin.CustomerName = "Darren Jefford";
    itin.County = "Wiltshire";
    itin.TotalItineraryPrice = 1285.00M;
 
    // Commit these changes to the database;
    itin.CommitItineraryActivity();
 
    // Add a link between this Itinerary activity and another flight activity that already exists
    itin.AddReferenceToAnotherActivity(DirectESApi.Activities.Flight, flightActivityID);
 
    // Add a message body to this activity (or any other data you require)
    itin.AddCustomReference("MsgBody", "MessageBody", System.DateTime.Now.ToString(), myXmlMessageBody);            
 
    // End the Activity
    itin.EndItineraryActivity();

TechEd 2007: Sessions and Book Signing

Darren Jefford — Sun, 03 Jun 2007 15:51:00 GMT

I arrived in Orlando yesterday after a long old flight from London and should make the Party with Palermo tonight which looks like it'll be fun but as I'm presenting tomorrow it'll be a quiet night!

My two sessions are shown below, the second session being a very informal chalk/talk style session without any slides so if you want to know more about BizTalk Performance & Scalablity testing or have some questions, come along!

A book signing has been organised at the TechEd bookstore following my session on Monday, come along between 1815 and 1845 and Ewan and I will be signing copies!

My sessions this week are:

SOA306 - Building an Enterprise-Wide Instrumentation Solution Using the Microsoft BizTalk BAM Infrastructure
Monday 4th June: 1645 to 1800
Track(s): SOA and Web Services
Level: 300
Speaker(s): Darren Jefford

Business Activity Monitoring (BAM) is a powerful feature of BizTalk Server and is often marketed as allowing "the business" to understand what is happening within your BizTalk solution. BAM does this really well, but it can in fact be used in a variety of other ways which can deliver huge value to customers and address a number of issues they have with BizTalk based solutions and non-BizTalk based solutions.

In this session, we cover some of the fundamentals of BAM and detail how you can utilize BAM to collect a variety of information and produce a "tracking portal" which you can use to support your application, perform manual repair of messages, and generally observe your solution.

We also show how BAM is not just for BizTalk solutions and how it can be used to produce an enterprise-wide instrumentation solution that is highly scalable and flexible; we touch on the new Windows Workflow Foundation and Windows Communication Foundation (WCF) BAM Interceptor technology that enables data to be collected from Workflows and WCF services enabling a true end-to-end instrumentation solution.

SOA13-TLC - Microsoft BizTalk Performance Testing
Thursday 7th June: 1630 to 1745
Track(s): SOA and Web Services
Level: 400
Speaker(s): Darren Jefford

Testing is a critical area for any solution and BizTalk is no exception. In this session, we start by covering the foundations that you need to put into place before beginning any performance testing, covering the most common environment problems that cause problems during testing and highlighting a number of tools to make testing easier including BizUnit and LoadGen.

Then, we cover how you should monitor your solution during testing; highlighting the key performance counters that should be monitored and explaining how throttling works. We finish by covering a number of common symptoms that you're likely to experience during testing; we explain why these occur, how to spot them, and more importantly how to resolve them.

Professional BizTalk Server 2006: UK has stock!

Darren Jefford — Wed, 23 May 2007 15:23:00 GMT

After what seems like a huge delay the usual UK retailers such as Amazon and Computer Manuals now have stock of the book! From what I've heard pre-orders have already started to drop through letter-boxes.

Enjoy, and please remember to put a review on Amazon if you get the chance!

For those of you going to TechEd in two weeks time, Ewan and I will be taking part in a book signing on Monday the 4th June between 6:15pm to 6:45pm at the TechEd book store following my breakout session.

Professional BizTalk Server 2006: In Stock (In the US)

Darren Jefford — Wed, 02 May 2007 12:13:00 GMT

Another quick update, it seems that stores in the US now have stock and are shipping. Both Amazon.com and BarnesAndNoble are reporting stock! The UK should follow very shortly, 7 days seemed to be the difference with the Team System book

BAM Best Practice: Starter for 10..

Darren Jefford — Mon, 30 Apr 2007 12:11:00 GMT

I knocked out a brain-dump of some BAM best practices this morning for a colleague, it seems like a good thing to share more broadly. It's a starter for 10 and I'll refine it over time I'm sure, the book covers all of these things in far more detail as you'd expect :)

• Avoid continuation where you can as it incurs performance overhead, it’s not the end of the world if you were to use continuation but just remember the overhead. The rule of thumb seems to be that up to 2 continuations is OK any more and you should create new activities and link them together using references.

• You must always use continuation if you are writing data to an activity over time and using an eventstream other than the DirectEventStream.

• If you are using continuation you must always call EndActivity passing the original ActivityID once you have enabled continuation, otherwise it can lead to orphaned activities that don’t move from Active to Completed. In short EndActivity must be called for every ActivityID used throughout the lifetime of an Activity.

                es.BeginActivity(“MyActivity”, “ActivityID_1” );
                ...
                es.UpdateActivity(“MyActivity”,”ActivityID_1”,”Name”,”Darren”);
                es.EnableContinuation(“MyActivity”,”ActivityID_1”,”ActivityID_2”);
                es.EndActivity(“MyActivity”,”ActivityID_1”);

                ....
                es.UpdateActivity(“MyActivity”,”ActivityID_2”,”Age”,”99”);
                es.EndActivity(“MyActivity”,”ActivityID_2”);

• Run SQL queries against the SQL Views not the tables otherwise you may not retrieve all of the data due to table partioning occuring under the covers through the use of the Data Maintenance (DM) job

• Schedule the DM_<ActivityName> SSIS job to ensure your Activity tables are swapped out for empty ones to maintain insertion speed

• Use OrchestrationEventStream/MessagingEventStream wherever possible as this will keep the Bam data transactionally consistent with the execution of the orchestration or message in that if the orchestration rolls back the BAM data will be as well. You also benefit from being able to piggy back the orchestration persistence database roundtrip or interchange commit.

• Ensure the tracking host is running if your using a EventStream other then DirectEventStream, the host responsible for tracking can be configured through the BizTalk Administration Console. This defaults to BizTalkServerApplication so ensure this host is started otherwise data will not appear in the BAMPrimaryImport database.

• If your using an EventStream other than DirectEventStream then BAM data is copied out of the BizTalkMessageBox using TDDS, you can see the data pending via the Tracking Data Size perfmon counter, although if you have global tracking enabled this will also include tracking data.

• BAM can store up to 1MB of data through the AddReference method on the eventstream, a great well-pefoming solution for message-bodies which enables the storage of message bodies alongside the data that describes how it was processed.

Presenting at TechEd

Darren Jefford — Thu, 26 Apr 2007 16:13:00 GMT

I'm pleased to announce that I'll be presenting two sessions at TechEd which is being held at Orlando between Monday 4th June and Friday 8th June! The first is a breakout session which covers BAM and demonstrates some real-world examples of how we've put BAM to great use and how it has a place in almost every BizTalk solution, the second is a chalk-talk covering BizTalk performance testing.

Ewan Fairweather (author of the Administration chapter in my book) will also be presenting a breakout session titled "SOA310 - Best Practices and Customer Evidence on Enterprise Ready BizTalk Implementations"

The session abstracts are shown below for reference, and it looks like a book signing will be organised during the week for those of you attending that want to get a signed copy! See you there!

SOA306 - Building an Enterprise-Wide Instrumentation Solution Using the Microsoft BizTalk BAM Infrastructure
Track(s): SOA and Web Services
Level: 300
Speaker(s): Darren Jefford

SOA13-TLC - Microsoft BizTalk Performance Testing
Track(s): SOA and Web Services
Level: 400
Speaker(s): Darren Jefford

Professional BizTalk Server 2006: Shipping!

Darren Jefford — Tue, 24 Apr 2007 16:10:00 GMT

I've just had an update from Wiley and the book arrived into their warehouse last Friday (20th April) and it started physically shipping to stores yesterday (Monday 23rd April) so if you've pre-ordered expect it to arrive real soon! I hope you enjoy it!

Woo Hoo!!

Watch out: DebugView (OutputDebugString) & Performance

Darren Jefford — Tue, 03 Apr 2007 23:43:00 GMT

For developer level tracing the tracing classes held within the System.Diagnostics namespace are often used along with the Windows Sysinternals DebugView tool. Tracing events are written to OutputDebugString via the DefaultTraceListener trace listener.

This listener writes output to OutputDebugString (which DebugView can monitor), and a managed debugger (if one is attached). In addition the DefaultTraceListener trace listener is responsible for popping up and dealing with the managed assert message box. You can disable the "default" DefaultTraceListener in your application’s config file as required.

This approach to tracing works extremely well and is something I use myself a lot, but you should be aware of the associated performance overhead. DebugView effectively acts as a debugger and hooks the OUTPUT_DEBUG_STRING_EVENT event, as a result your application (a BizTalk process perhaps) will experience a performance impact as the application threads will be suspended whilst the debug information is output, effectively serializing your application. Therefore if you have many Orchestrations in-flight (or multiple threads if your not using BizTalk) they will all be side-affected if DebugView is started and performance will therefore be affected.

It's worth noting that if the DebugView tool is not started (and therefore there are no subscribers to the information) the performance impact is negligible and shouldn't in my experience be something to worry about, therefore do not be afraid to output information that you feel will be useful.

Either way you should ensure that your tracing code is implemented such that it can be disabled to reduce the performance overhead, typically you will need the ability to enable it at run-time. When carrying out performance testing you will most likely want to disable tracing, although gaining an understanding of the overhead is also valuable.

In my view the performance hit associated with software tracing is outweighed by the ability to diagnose run-time issues in a non-intrusive manner. Most Microsoft products, including BizTalk Server, rely heavily on software tracing which the product group leverage to diagnose customer issues off-line. Just be aware that running DebugView during performance testing or on a live server is very likely to significantly impact performance! I'm not saying not to use it, Please Do! It makes everyone's life easier but just be aware of the performance impact when your running those tests and in live.

Roll on the mighty ETW TraceListener planned for Visual Studio “Orcas”, kernel model ultra-fast tracing from managed code! ETW has been available for a while but has been pretty hard to leverage from managed code. It’s worth nothing that BizTalk itself uses ETW heavily to provide extremely rich internal “developer level” tracing which we can use to diagnose problems at customer sites.

Thanks to Jon Fancey for highlighting this during the review stage of the book, he experienced a solution being seriously impacted when DebugView was left running on a live server, and to Matt Ellis (BCL) for adding some detail around the DefaultTraceListener.

BAM: PivotTables not working on the BAM Portal

Darren Jefford — Tue, 03 Apr 2007 23:20:00 GMT

I’ve used the same base VPC for my various BizTalk Server 2006 images over the last few years, this has worked fine apart from a few annoying bugs that of course plagued every single instance of a VPC using it, of course I never really spent the time to iron them out so suffer from them every now and again :-(

All are most certainly down to me not having exactly the right bits installed or not having the configuration quite right. One annoying problem was the Pivot Table view on the BAM Portal refusing to work by coming up with the following unhelpful error:

“The query could not be processed: Cannot connect to the server 'DARRENJBTS06'. The server is either not started or too busy.”

Over the years I’ve tried just about everything to fix it: Configuration, Security, Service Packs,etc. but have never managed to solve it until now - after some digging internally it seems that my base VPC was missing Version 9.0 of the Analysis Services OLE DB Provider, installing this resolved the issue straight away, the update is listed on this page along with some other updates.

Hope it helps!

Professional BizTalk Server 2006: Release Date

Darren Jefford — Thu, 29 Mar 2007 16:24:00 GMT

Hi,

I'm reliably informed by Wiley that the book should be in-stock at their warehouse on the 23rd April 2007 and should then ship to distributors/stores in the US followed by international outlets subject to delivery times. Enjoy!

Woo Hoo!

Professional BizTalk Server 2006: Chapter List

Darren Jefford — Mon, 26 Mar 2007 15:51:00 GMT

Following on from my previous post, included below is the final chapter list and a brief overview of the content covered in each chapter.

Chapter 1: Technical Primer—This chapter primes the reader on some key technologies that each developer should have a basic understanding of to aid with development and debugging. This chapter covers XML schemas, namespaces, XPath, and serializable classes.
Chapter 2: BizTalk Architecture—This chapter is the only chapter that really deals with explaining the basic BizTalk principles. It positions the value that BizTalk brings to solutions and compares this to writing a custom solution without BizTalk. It then runs through all the key architectural pieces of BizTalk and explains what they do and how they work.
Chapter 3: Adapters—This chapter covers key adapter concepts—such as ports, tracking, hosting, and context properties—and then drills into each adapter supplied with BizTalk to explain what the adapter does, which context properties it promotes, and, where appropriate and relevant, a walkthrough to show how to use the adapter. The chapter also details the new R2 adapters at a high-level and in time for publication we'll add downloadable content to demonstrate how to write an adapter using the new WCF Adapter framework shipped with BizTalk Server 2006 R2 (not available at the time of writing).
Chapter 4: Pipelines—This chapter covers the core principles of pipelines and details how they work. The chapter then highlights how pipeline components should be developed and provides a worked example.
This chapter also covers the Messaging Engine architecture and how it works, understanding this is key to being able to exploit the architecture to the fullest.
Chapter 5: Orchestrations—This chapter introduces BizTalk orchestrations and the environment within which they execute. We discuss key areas such as persistence points and logical ports before covering how orchestrations can be developed and which features are available.
Chapter 6: Business Activity Monitoring—This chapter introduces the Business Activity Monitoring (BAM) technology and details how BAM can be used to instrument your BizTalk solution and your entire enterprise. The chapter then covers how a fictional scenario can be instrumented and how you can make use of the information collected by BAM.
Chapter 7: Rules Engine—This chapter discusses the key principles that underpin the Rules Engine, and then covers all the concepts to enable effective use of the Rules Engine within your solution.
Chapter 8: Testing—This chapter starts by describing the types of testing you must apply to your BizTalk solution, including unit testing, functional testing, integration testing, and performance testing. The chapter goes on to discuss how BizUnit can be used to automate testing, how you can perform code coverage of your BizTalk solution, how LoadGen can be used to generate load, and how you can profile your solution using the Visual Studio profiler.
Chapter 9: Performance and Scalability—This chapter covers all the things you must have in place before beginning any performance testing. Forgetting these often leads to bad performance results. The chapter then explains how to monitor BizTalk, SQL, and IIS, explaining all the relevant performance counters and what they actually mean. We also provide a complete reference on what BizTalk throttling is and how it works, including an explanation of all the various throttling states. The final section of the chapter discusses a number of common symptoms. These are problems that customers run into regularly and include high CPU usage and sudden processing drop off. The chapter explains the common reasons and things to check to identify the underlying problem.
Chapter 10: Low Latency—This chapter covers what low latency means to BizTalk solutions and how you can measure it. We then drill into a variety of techniques that you can employ to reduce latency for your solution.
Chapter 11: Administration—This chapter describes everything required to administer BizTalk effectively. It's a great resource for administrators as well as developers. The chapter covers the key administration tools and tasks that you need to undertake, and highlights areas that are often overlooked, such as SSO maintenance, suspended instances, subscriptions, etc. The chapter also covers in great detail how the SQL Server used by BizTalk needs to be looked after, including backup, log shipping and how you can monitor the performance of the SQL Server. The chapter also covers MOM and how it can be used in conjunction with the BizTalk Management Pack to greatly simplify management.
Chapter 12: End-to-End Scenarios—This chapter highlights all the End-to-End Scenarios that were shipped as part of BizTalk 2006. These are often overlooked but provide full working solutions of real-world customer scenarios that have been fully tested and demonstrate best practice. Each scenario is supplied with complete source code and provides implementations that you can use within your own solution. We cover each scenario and highlight key deliverables in each scenario such as the code required to use the SSO store for configuration data and an adapter to enable messages to be sent for manual repair.
Chapter 13: BizTalk Best Practices—This chapter discusses a number of best practices and techniques that have been used in a number of projects, including handling large messages, storing configuration data, subscribing to failure messages, and instrumenting your solution.
Chapter 14: Windows Workflow and BizTalk—This chapter positions the key concepts of Windows Workflow and discusses how it compares with BizTalk Server. We then demonstrate how the BAM Interceptor for Windows Workflow works and position how BizTalk, WF, and SharePoint can be used together to enable compelling solutions.

Professional BizTalk Server 2006: Off to the printers

Darren Jefford — Mon, 26 Mar 2007 15:05:00 GMT

A year on, ~224,000 words and over 600 pages later, and Professional BizTalk Server 2006 has finally gone to the printers! This book has been a mammoth undertaking for all involved, but having just reviewed the entire book end-to-end I’m extremely proud of what we’ve achieved.

The final cover is shown below with all of the authors photos!

I'm also pleased to announce that Patric McElroy who is a Product Unit Manager (PUM) for the BizTalk Server Team, kindly wrote a great foreword for the book and we included the following fantastic quote on the cover:

“This book is an indispensable resource for solution architects and developers who want to dig deeper into BizTalk. It is valuable material that has been written by insiders with deep experience in architecting and delivering solutions for our most demanding enterprise customers.”
—Patric McElroy, Product Unit Manager, BizTalk Server, Microsoft Corporation

A big thanks to Yossi Dahan and Jon Fancey who both reviewed the entire book and provided some great comments and suggestions, we also included a very kind comment from Yossi on the back cover:

“This book provides a fantastic insight into the inner workings of BizTalk, making it an invaluable resource for all those things you would otherwise never know. As you learn the ‘tricks of the trade’ from these experienced engineers, you quickly realize why everyone who has anything to do with BizTalk simply MUST read this book.”
—Yossi Dahan, Managing Consultant, Sabra Ltd.

The final back cover text is shown below:

This book provides insight into how industry experts have successfully architected, developed, and maintained Microsoft BizTalk Server in mission-critical environments. Authored by highly-regarded consultants with a wealth of knowledge on designing, building, and operating enterprise applications using BizTalk, this comprehensive guide gives you the techniques and best practices you’ll need to develop effective projects.

Darren Jefford is a Principal Consultant with the Microsoft UK Application Development Consulting (ADC) team who has extensive real-world experience with BizTalk Server and the broader Microsoft platform. Kevin B. Smith formerly worked as a Technical Lead Software Design Engineer for the BizTalk Server Product Team and helped ship three versions of BizTalk Server. Ewan Fairweather works as a Premier Field Engineer for Microsoft, providing onsite support to enterprise customers, including maintaining and optimizing their BizTalk Server solutions.

Achitects, developers, testers, and administrators will achieve instant success when they apply the deep technical information covered in this book. They will better appreciate the internal workings of BizTalk Server and will understand detailed solutions for challenges often experienced with BizTalk-based systems.

My initial plans were to co-author the book with Kevin Smith but he (sensibly) was reticent due to the sheer amount of work required. I therefore started the book as sole author but as the writing got underway I began to realise my limitations, BizTalk is a huge product and it’s impossible to know everything about everything, especially in enough depth for a book. I finally persuaded (nagged to death) Kevin to author 3 chapters on topics that he knew extremely well, especially as he was part of the BizTalk product team and responsible for the design and implementation of the messaging engine.

As the writing continued the Administration chapter loomed into view and being a developer at heart – Administration is a very weak point! I managed to talk an (admittedly tipsy) Ewan Fairweather into writing the chapter at the Microsoft UK summer party, he pulled out all the stops against tight deadlines to write some amazing content on Administration which given his background (supporting customers real-world BizTalk deployments) will be of great value to customers and I must admit to learning an embarrassing amount whilst reviewing his chapter.

So, your all probably wondering – why on earth did I decide to invest so much personal time writing a book on BizTalk? As ever it’s a long story, but the following fragment from the book introduction explains it nicely:

The compelling drive for me to create this book resulted from two main factors. First, after working with these customers I had a clear view of what worked, what didn’t, and the things to highlight early on to avoid problems later in the development life cycle of a BizTalk solution.

The next compelling drive was that such real-world, best-practice advice was tied up inside a small number of people’s heads. This clearly doesn’t scale to help the broader base of BizTalk customers, and it’s often frustrating (to both sides) to be called on site to see the same issues crop up again and again.

These problems, in my experience, are not the fault of the product or the customer per se but instead represent the learning curve required for enterprise software development. It’s a hard problem space, and tiny mistakes or oversights have big consequences. This book is positioned to help address this. It’s a fusion of how the product works under the covers and cutting-edge best practices designed to enable you to make best use of the product within your solution.

The decision to write a book was a tortuous one. I loved the idea of writing a book, but I was given a number of reality checks by colleagues with regard to the effort required — something that (with a young family) I wasn’t sure I could commit to, and I decided to shelve the project.

The final straw, however, came in late in 2005. While onsite with a customer, I was asked the following question: “Why does it take Darren Jefford or Kevin Smith to come in to help us understand these problems and highlight the types of things we should be aware of? This type of information isn’t in the documentation….”

The book is now off being printed and should be available in early May I’m reliably informed, based on my experience with the Team System book it will be available in the US first, followed by the UK a few weeks later so you should expect delivery by the end of May.

Ewan and I will be presenting at TechEd US (Orlando) so look out for us there if your attending, I believe Wiley will probably be setting a book signing opportunity up if your interested, once the session titles are formalised I'll post them here.

Phew! Now back to reality and spare time! I hope you enjoy the book and the blog should hopefully get back to normal and I can post some of the interesting WCF bits and pieces I've been up to.