Connected Systems in the Great White North : BizTalk 2006

A Web based version of the BizTalk Rules Composer

pkelcey — Sun, 11 May 2008 06:55:00 GMT

As part of a proof-of-concept I was working on last year, we needed to show that the BizTalk Business Rules could be created and managed via a Web Application instead of through the standard Business Rules Composer tool that we provide out of the box. The client had certain requirements that meant that using the standard Composer just wasn't viable. Therefore, I replicated a portion of the Composer as a Web based application. I thought I'd post some images from the app and see if this is something people are interested in. If you are, I'll work to clean up the code and post it to http://www.codeplex.com/. Let me know by posting a comment here.

There were two tools in the. One let you view and edit vocabularies. The other let you view/author/edit/delete/publish/deploy policies and rules.

Image 1: The home page, gives the user the ability to access the 2 tools contained in the app.

Image 2: The Policy and Rule Explorer This screen lets a user select a specific Policy to review. Once they select a Policy, they are shown the entire version history for it. For each version, then can load up a list of all the rules contained within that version. At this point, they have the option to publish, deploy or copy that particular version. They also have the ability to select each rule and view its Conditions and actions.

Image 3: Adding a New Rule

Image 4: The Vocabulary Explorer. Lets a user view the definitions contained within all the Vocabularies and lets them alter the values of existing definitions.

Cheers and keep on BizTalking...

Peter

Microsoft ESB Guidance: Error - Host restarting every minute

pkelcey — Fri, 01 Feb 2008 09:33:00 GMT

We'll I just spent about three hours tonight fighting with this issue, so I thought I'd post about it to hopefully save someone else from having to go through this.

I was trying to run a couple of samples from the ESB and I noticed that whenever I used an example that had to use an orchestration, I received the following error in my event log

Event Type: Warning
Event Source: BizTalk Server 2006
Event Category: BizTalk Server 2006
Event ID: 5410
Date: 12/4/2007
Time: 4:08:39 PM
User: N/A
Computer: EDIBTS2006
Description:
An error occurred that requires the BizTalk service to terminate. The most common causes are the following:
1) An unexpected out of memory error.
OR
2) An inability to connect or a loss of connectivity to one of the BizTalk databases.
The service will shutdown and auto-restart in 1 minute. If the problematic database remains unavailable, this cycle will repeat.

Error message: Exception has been thrown by the target of an invocation.
Error source: mscorlib

My host instance would then shut down and try to restart itself every minute. Each retry would fail with this same error. I also ended up with a running service instance in the admin console that I couldn’t terminate.

Each time I sent another message, I would have to manually clean these left over instances from the DB before I could restart the host instance successfully. I finally noticed that this error was only occurring whenever a component in my “In-proc” host instance tried to run. Base on this I tracked the issue down the btsntsvc.exe.config file that we were suppose to edit during install. I had forgotten to remove the “[path]” place holder that is in the <ConfigurationFile> tag. We were suppose to replace this with the actual path to the Microsoft.Practices.ESB.PipelineComponents.config file. (I’ve did not have this step in my install checklists, so I have since added it). I realized that the AppDomains were not loading correctly and this caused my errors in every orchestration in that tried to run in that AppDomain.

This is what I had left in my config file:

<ConfigurationFile>
[path]\Microsoft.Practices.ESB.PipelineComponents.config
</ConfigurationFile>

As I said, I had forgotten to replace the [path] place holder. Anyways, I put in the proper path and I expected this to correct the situation, but I kept getting the same annoying error. After another hour of frustrated debugging, I realized that I had entered the config data like this:

C:\Projects\Microsoft.Practices.ESB\source\code\core\config\Microsoft.Practices.ESB.PipelineComponents.config

</ConfigurationFile>

You’ll notice that I had put line breaks before and after the path and filename. It was these dam line break that were causing the problem. I took them out and and used this:

<ConfigurationFile>C:\Projects\Microsoft.Practices.ESB\source\code\core\config\Microsoft.Practices.ESB.PipelineComponents.config</ConfigurationFile>

Everything worked fine and the issues are gone.

I’ve seen a number of people asking about this on codeplex and other locations, so double check your settings and avoid the headaches.

Cheers and keep on BizTalking...

Peter

ESB Exception Management fails with Canadian Regional Settings

pkelcey — Thu, 24 Jan 2008 22:28:00 GMT

I spent an hour or so this morning working with a college to debug a strange error he was getting when he tried to use on of the Exception Management Framework samples that are in the ESB guidance for BizTalk. He was intentionally creating exceptions and he expected to see them routed into the ESB exception database. However, every time the ALL.Exception SQL Send port tried to route the message to the SQL Server, he got the following error.

The adapter failed to transmit message going to send port "ALL.Exceptions" with URL "SQL://VBROKERSERVER7/EsbExceptionDb/". It will be retransmitted after the retry interval specified for this Send Port. Details:"HRESULT="0x80040e07" Description="Error converting data type nvarchar to datetime."

Now we both expected that he had made a mistake during the installation and configuration phase, but after we spent a fair bit of time comparing his installation to my installation (which worked by the way!) we realized that we had identical configurations. We finally realized that the only difference between the environments was that I had configured my Windows server to work with a regional setting of "English (United States)" while he was using "English (Canada)". The problem was occurring because the two countries store the day/month data in reversed order. Once we changed his system so that the user account that BizTalk ran under used "English (United States)" his system was fine.

I've since seen someone with a near similar problem using the Dutch regional settings. He found a code level work around that can be used to fix the problem and allow any country's setting to be used. Details are at http://www.codeplex.com/esb/workitem/view.aspx?workitemid=4430

Cheers and keep on BizTalk

Using the ESB Guidance - Installation Process and Checklists

pkelcey — Wed, 23 Jan 2008 07:16:00 GMT

Many people are eager to get their hands on the new Microsoft ESB Guidance for BizTalk and to start exploring all of the great stuff it contains. However, we’re seeing that a lot of people are struggling with the installation process. Therefore, I thought I’d put together some documentation on the process I followed and I also figured that it would be useful to create some simple checklists that outline all of the tasks, actions and software you need to complete.

Based on my experiences (and backed up by my college Jim Bowyer’s experiences) most of your time will be spent installing the prerequisites that the ESB guidance requires. The guidance does include some fairly comprehensive help files that do outline all of the service packs, hot fixes etc that you need, however I’ve found that most people rush through this section a bit and miss one of two of the required components or required tasks. I myself spent several hours trying to debug the sample applications only to realize that I had missed one key hot fix and one key step in the configuration process. My strongest recommendation is to really take your time when setting up your environment and make sure you’ve got everything install and configured properly before you try and install the ESB guidance. A little care taken at early will save you massive amount of time later when you’re trying to debug malfunctioning components and sample applications.

After running through this process a couple of times, I’d estimate that I could do a complete installation in a little less than ½ a day. It took me almost two days the first time I did this due to fact that I rushed through the process and missing a couple of steps. The ½ day estimate is also based on the fact that I have a base Virtual PC image that I can start from. Without this image (I.e. if you’re building a machine from the O/S up), you’ll obviously need more time (perhaps even up to 2 days). Finally, please note documentation is not my day job; these are merely supplementary notes to help folks get started with the ESB Guidance. I hope you find this valuable and I welcome your feedback.

I've attached a PDF document containing the process overview and checklists.

Update: For those of you that are reading this through an RSS feed or from a syndicated blog site, you might not be able to see the attachment. Here's a link you can use to download the file. http://www.peterkelcey.com/downloads/ESBInstallProcess_V1.pdf

Cheers and keep on BizTalking..

Awesome new BizTalk Database Optimization Whitepaper

pkelcey — Fri, 07 Sep 2007 04:48:00 GMT

One of the most tricky aspects of developing BizTalk applications can be the performance tuning and optimization phase. A great new whitepaper has just been released to MSDN that has extremely deep and thorough coverage on how to configure, install and tune the a BizTalk database tier. This is a must have paper for anyone dealing with high load, high performance BizTalk applications

http://msdn2.microsoft.com/en-us/library/bb743398.aspx

Great work done by John B. Brockmeyer and team.

Cheers and keep on BizTalking

Using the ESB Guidance: Dynamic Transformation and Routing

pkelcey — Sat, 20 Jan 2007 10:42:00 GMT

In my last entry, I showed how developers could use our new ESB guidance to submit generic messages into BizTalk and have that message routed to a Service Endpoint based on a UDDI entry. Today, I thought I'd expand on that example and add in another common scenario that occurs within an ESB architecture, namely "Dynamic Transformations". (If you haven't seen the last post, its probably best to check it out first as I'll be building on the concepts introduced there)

The Goal

Submit a generic message to the ESB and have it dynamically transformed and then have the new message dynamically routed to a Service Endpoint that is defined in a UDDI entry. The client application needs to be able to specific the transformation map to use as well as the name of the Service.

The Solution

The first thing I did was to build out a test client that I could use to submit message to the ESB. I needed to be able to specify the body of my message, the map I want to use for the transformation and the name of the Service that I want the message submitted to.

In my last post I spoke about the ESB On Ramp Web Service and show how you could submit generic XML message to it. I also outlined about how to submit message processing instructions to the ESB by using SOAP headers attached to that SOAP message. To achieve dynamic transformation and routing, we'll again need to use the this On Ramp and we'll again attach message processing instructions via the SOAP Headers. Therefore, the first thing I did after creating my test client was to add a Web Reference to the ESB On Ramp Web Service.

Next, I created a function that would handle all interaction with the OnRamp Web Service. To achieve both dynamic transformation followed by dynamic routing, I needed to use four of the SOAP headers.

ProcessingInstruction - (Allows me to specify the first processing task for the ESB to perform)
Itinerary - (I'll talk more about this in a minute)
MapType - (Allows me to specify the fully qualified name of the map I want BizTalk to use in the transformation)
EndpoingUddiLabel - (Allows me to specify the name of the Service that I want message routed to)

private string Submit_Transform_Route(string strService, string strMessage, string strMap)

{

// Get an instance of the On Ramp Service

ESB.OnRamp ramp = new ESB.OnRamp();

// Assign security credentials

ramp.Credentials = new System.Net.NetworkCredential("username", "password");

// Assign values to the headers we need to perform the UDDI based routing

ramp.EsbSoapHeadersValue = new ESB.EsbSoapHeaders();

//Tell the ESB that the first task is to "Transform" the message

ramp.EsbSoapHeadersValue.ProcessingInstruction = "TRANSFORM";

//Let the ESB know what all of the steps I want performed are

ramp.EsbSoapHeadersValue.Itinerary = "TRANSFORM,ROUTE";

//Tell the ESB which Map to use in the transform

ramp.EsbSoapHeadersValue.MapType = strMap;

ramp.EsbSoapHeadersValue.EndpointUddiLabel = strService;

// Get the message to send

XmlDocument doc = new XmlDocument();

doc.LoadXml(strMessage);

try

{

//Send the message to the OnRamp and return the result

ramp.Receive(doc);

return "Message Submitted Successfully.";

}

catch (System.Exception ex)

{

return "Error: " + ex.Message;

}

You'll notice that its almost identical to the function I developed for my last post. In that function I had to set three of the EsbSoapHeaders to achieve UDDI based routing for my message. I had to set the ProcessingInstruction = "ROUTE", the Itineray to "ROUTE" and the EndPointUddiLabel to the name of the Service I wanted the message routed to.

For this example, I actually wanted the ESB to perform multiple and sequential tasks on my message. (i.e. transform it first and then route it). Since the "ProcessingInstruction" header only allows me to specify a single task for the ESB to perform, I needed to find some header that would let me submit a list of instructions. This is where the "Itinerary" header comes into play. It is this header that allows me to pass in a comma delimited list of tasks which the ESB will perform one after the other.

Once I had this function built, all I had to do was call it from the onClick where I passed in the Map name and the Service name.

private void button1_Click(object sender, EventArgs e)

{

Cursor.Current = Cursors.WaitCursor;

btnSubmit.Enabled = false;

string result = Submit_Transform_Route(txtServiceName.Text, txtOriginalMessage.Text, txtMapName.Text);

btnSubmit.Enabled = true;

Cursor.Current = Cursors.Default;

MessageBox.Show(result, "Result", MessageBoxButtons.OK, MessageBoxIcon.Information, MessageBoxDefaultButton.Button1, MessageBoxOptions.DefaultDesktopOnly);

}

Its important to understand that you can only use a Map that has already been deployed into your BizTalk server. So you do need to pre-develop the Map before you can use it. For my example here, I just used one of the prebuilt samples that comes with the SB guidance. Also, you have to provide the fully qualified name of the map.

How it Works

The way this scenario works is very similar to the Dynamic Routing scenario I explained in my last post. Our message is submitted to the OnRamp Web Service, it promotes all of the SOAP headers that we provided up into the standard BizTalk Message Context. Within BizTalk, we then have our "Agents" (i.e. Orchestrations) that are setup to subscribe to our message. Each Agent subscribes to a message based on the "ProcessingInstruction" property. In my example, I set the ProcessingInstruction equal to " TRANSFORM". This resulted in my message being picked up by the DynamicTransformation.odx orchestration. (Found in the Microsoft.BizTalk.ESB.Agents.Transform project). It is this orchestration that will perform the transformation for us.

This transformation orchestration does a couple of things for us.

It checks to see if we have specified a map for us. If we haven't it attempts to determine the map name based on information sent in the SOAP headers.
Once a map name is determined, the transformation is performed for us.
Advance the itinerary for us.

"Advance the itinerary"??? What's that mean? We'll lets take a look ...

Itineraries

Things to understand:

Message processing tasks are performed by "Agents" which are really just BizTalk Orchestrations
Agents subscribe to messages based on the "ProcessingInstruction" property. I.e. The Transform agent only picks up messages with a ProcessingInstruction set to "TRANSFORM". The delivery agent only to messages with "ROUTE"
Itineraries are comma delimited list of processing instructions that will eventually be moved up into the ProcessingInstruction property as each step is completed.

After an agent picks up a message from the messageBox and completes its core task, it is required to examine the current message itinerary. If there are additional steps that need to be completed, the agent is responsible for extracting the next step out of the itinerary and moving it up into the ProcessingInstruction property. The message is then published into the messageBox where another agent can potentially pick it up based on the new processing instruction. It is this mechanism that allows us to "chain" together our agents and complete multiple tasks on our message as it moves through the ESB.

Lets take a look at a portion of the Transform agent orchestration.

In this image, we can see the 2 key tasks that the agent is performing. In the "MessageAssignment" step, the actual transformation is performed. The code that performs the dynamic transform is:

tMapType = System.Type.GetType(FullyQualifiedMapType);

transform (OutboundMessage) = tMapType(InboundMessage);

The second key box is the "Advance Itinerary" one. It contains the following code:

OutboundMessage(*) = InboundMessage(*);

itineraryStep = Microsoft.BizTalk.ESB.Helpers.ItineraryHelper.Advance(OutboundMessage(Microsoft.BizTalk.ESB.Itinerary));

OutboundMessage(Microsoft.BizTalk.ESB.Itinerary) = itineraryStep.Itinerary;

OutboundMessage(Microsoft.BizTalk.ESB.ProcessingInstruction) = itineraryStep.NextStep;

As you can see, this code advances the itinerary forward one step by calling the ItineraryHelper class. The ProcessingInstruction property is then updated to contain this next step. For our example, the "TRANSFORM" instruction I manually assigned to the ProcessingInstruction is replaced with the second Itinerary task "ROUTE". Once this code runs, a new message is published back to the messageBox where our Routing agent picks it up and perform the dynamic routing. (If you want details on how that agent will run, check out the previous post) This is one way in which Dynamic Transformation and Dynamic Routing can be achieved using the ESB guidance.

Extending Beyond this Example:

The ESB guidance currently ships with two default agents, the Transform agent and the Delivery agent. It's important to understand that you can extend your ESB beyond this basic functionality by creating you own Agent orchestrations that subscribe to your own custom ProcessingInstructions. The guidance contains a set of guidelines on how to develop your own agents so that you can develop your own custom and potentially complex ESB tasks.

Now, if you are interested in leveraging the ESB guidance for your organization, we have currently released it to a core set of Microsoft Partners. You can find a list of these partners at http://www.microsoft.com/biztalk/solutions/soa/esbpartners.mspx

Using the Microsoft ESB Guidance: UDDI

pkelcey — Tue, 02 Jan 2007 22:18:00 GMT

Some of you may have heard by now that we are in the process of releasing the Microsoft Enterprise Service Bus (ESB) guidance. This guidance contains pre-built components, use cases, documentation and sales material that our Business Process Integration partners can use to rapidly deploy ESBs on top of BizTalk, Windows, SharePoint and SQL Servers. Over the holidays I spent some time digging into the code and thought I'd share a bit of what I've been seen.

I thought that I'd first talk about the standard ESB concept of "EndPoint Resolution" and how our guidance allows you to achieve this. Endpoint resolution is core requirement in any ESB as a Service may not always reside at the same address or endpoint. Therefore we need to be able to locate our services at runtime (wherever they may be). The Microsoft ESB guidance comes prebuilt with this capability, so I thought I post a small "How-To" blurb about how you can do it.

The Task

The basic task I thought I'd tackle is to submit a generic message from a basic client application into the ESB and have it routed it to a Service based on a entry in a local UDDI server. The following image show the Services that exist in my UDDI server.

The service that I'll wanted to submit the message to is "ENDPOINT: PostOrderMessage". You can see from the screen image that this Service has been setup with a File Folder as its endpoint. "C:\projects\Microsoft BizTalk ESB\Tests\Common\Filedrop\Output\Order %MessageID%.xml" If the solution works properly, I should be able to find my messages sitting in this folder.

ESB Concept: The "On Ramp"

As you dig into the guidance you'll quickly learn that one of the key components within it is the "On Ramp". The On Ramp acts as a central entry point into the ESB engine and it allows client applications to submit generic messages that contain processing, routing and transformation instructions. The On Ramp is a key concept to understand in order use the ESB guidance and if you're planning on using it, you'll need to have a good understanding it.

To keep things simple, the On Ramp is a Web Service that applications can use to submit XML based messages into BizTalk. On top of this, the On Ramp allow us to specify a number of processing, routing and transformation instructions that the BizTalk will follow when processing the message. To send these processing instructions, we use specific SOAP headers that are attached to our message when we submit to the On Ramp . If you've got the ESB guidance already, you can view all of the possible headers by opening the "EsbEnvGeneric.xsd" property schema file in the Microsoft.BizTalk.Esb.Schema project. This XSD file is a standard BizTalk property schema and denotes the properties that our SOAP headers will eventually be promoted into. For those of you that don't have the guidance, here a quick list of the parameters you can use when submitting your messages.

ProcessingInstruction
Itinerary
MapRulesPolicy
MapSelAssemblyName
MapSelMethodCall
MapSelTypeName
MapType
MapUddiLabel
MapXPath
EndpointAddress
EndpointConfigurationString
EndpointConfigurationRulesPolicy
EndpointConfigurationUddiLabel
EndpointConfigurationXPath
EndpointDeliveryAgent
EndpointMessagePattern
EndpointRulesPolicy
EndpointSelectAssembly
EndpointSelectMethod
EndpointSelectType
EndpointUddiLabel
EndpointXpath

For most projects, you'll ending have to work with most these headers in order to achieve full ESB capabilities, but for our simple task we'll just need to use three:

EndPointUddiLabel - We use this to tell the ESB the name of the Service to lookup in the UDDI
ProcessingInstruction - We use this to tell the ESB that the message needs to be routed.
Itinerary - Where we'll tell the ESB that this message needs to following a routing itinerary

Building the Solution

The first thing I did was to deploy and configure the ESB guidance as per the installation instructions it ships with. I won't cover those steps here, so I'll just jump to what I did once that process was complete.

The first thing I did was to build out a client app that I could use to submit my messages to the ESB. I used a very basic Windows Forms application as my client. All I needed was a form that would let me to define my message and specify the name of the Service I wanted to route the message to.

Once I had my basic form built out, the next step I needed to take was to add a web reference to the ESB On Ramp.

Once I'd added the web reference, I had everything I needed to submit messages into the ESB. For my project, I created a function to handle all interaction with the On Ramp web service. It creates my message, assigns the three SOAP headers I need and submits the message it to the On Ramp Web Service. Here's the code:

/// <summary>

/// This function prepares a message and submits it to the ESB on ramp

/// </summary>

/// <param name="strMessageBody">The core content of the message</param>

/// <param name="strServiceName">The name of the service in the UDDI that we want to send the message to</param>

private string SubmitMessage(string strMessageBody, string strServiceName)

{

// Get an instance of the On Ramp Service

ESB.OnRamp ramp = new ESB.OnRamp();

// Assign security credentials

ramp.Credentials = new System.Net.NetworkCredential("username", "password");

// Assign values to the headers we need to perform the UDDI based routing

ramp.EsbSoapHeadersValue = new ESB.EsbSoapHeaders();

ramp.EsbSoapHeadersValue.EndpointUddiLabel = strServiceName;

ramp.EsbSoapHeadersValue.ProcessingInstruction = "ROUTE";

ramp.EsbSoapHeadersValue.Itinerary = "ROUTE";

// Get the message to send

XmlDocument doc = new XmlDocument();

doc.LoadXml(strMessageBody);

try

{

//Send the message and return the result

ramp.Receive(doc);

return "Message Submitted Successfully.";

}

catch (System.Exception ex)

{

return "Error: " + ex.Message;

}

As you can see, the ESB class that was created when we added our Web Reference, contains a 2 classes. The "ESB.OnRamp" class which is our proxy to the actual Web Service and the "ESB.EsbSoapHeadersValue" class. This second class contains all of the properties that relate to SOAP headers that the ESB On Ramp will use.

I setup the OnClick event of my Submit button to call this function and I passed in the values of my two textboxes. As easy as that, I had built an application that was able to send a message into my ESB architecture. Once I'd done that, the ESB engine took care of everything else. It only took me ~10 minutes to build out my app and start routing my messages to Services. Its really is pretty simple and easy to get going with this. I've included a screen cam video at the end of this entry that shows me running the solution. (Its set for 1024x768 resolution)

Now that you've seen how to route message using UDDI, some of you might wonder what exactly is going on under the BizTalk covers. So lets take a look.

The Inner Workings

The first thing to look at is obviously the On Ramp. The On Ramp is essentially a Web Service and a BizTalk receive port. The receive location within the port uses the SOAP adapter to interact with the Web Service and also uses a custom pipeline "ReceiveForESBGeneric".

The SOAP adapter is configured to use the same Web Service that we made a web reference to in our client application. The custom pipeline is responsible for promoting the SOAP headers into BizTalk message properties so that other elements within BizTalk can access our instructions. Once the receive location processes the incoming message, the message (along with its promoted properties) is published into the standard BizTalk MessageBox and the On Ramp is done its work.

At this point, another BizTalk artifact takes over, the "Generic.odx" orchestration. This orchestration comes as part of the guidance and can be found int the Microsoft.BizTalk.ESB.Agents.Delivery project. The messages received from our On Ramp receive port are sent to this orchestration using BizTalk's standard publish/subscribe mechanism. This orchestration handles all incoming messages that need to be routed.

The orchestration has a direct bound receive port based on specific filter expressions. The receive shape that has a filter setup to receive all messages that have their "ProcessingInstruction" property set to "ROUTE". Hence why I had to set that that SOAP header to use a value of "ROUTE" in my client application.

Once receiving the message, the orchestration checks to see if an proper endpoint has already been assigned to this message. If it hasn't, then the orchestration will begin the process of resolving it. This is done in the "Call Resolution Class" expression box in the "Resolve EndPoint" scope. In this expression box, a call is made to a function call "Microsoft.BizTalk.ESB.Helpers.Resolver.Resolve". This function is a part of a custom library that ships with the ESB guidance and it handles the lower level steps required to resolve the endpoint.

In our example, we specified a UDDI service name in the EndPointUddiLabelField SOAP header. Because we did this, the Resolve function will attempt to resolve the endpoint by making a call to the UDDI server. If it finds the requested service in the directory, then the endpoint is returned (as a string) to the orchestration.

This endpoint is then used to configure the "GenericDeliveryAgentSend" dynamic port . The configuration of the port is done in the "Set Endpoint" expression shape.

Once configured, the port is able to transmits our message to the Service endpoint.

As you can see, the orchestration also contains number of exception handling steps which all tie into the central exception management framework that the guidance contains. The orchestration also contains a number of "Tracing" steps which allow for easier debugging of the process.

More than just UDDI and more than just Routing

I've quickly shown how to submit a message to the ESB engine that ends up being routed based on a UDDI entry. However I must add that you aren't limited to UDDI lookups for your address resolution. Using the SOAP headers, you can tell the ESB engine to use the Business Rules Engine or you can pass in the End Point directly with your message and there are also options for using XPath. The code contained in the guidance is incredibly flexible with regards to how you want to handle your endpoint resolution.

Also, message Routing is just one feature of the guidance. It also provides some great capabilities around dynamic message transformation, centralized exception management for services, linked itineraries and more. There are more than a few projects I've worked on where I wish I had this functionality prebuilt for me!

Cheers and Keep on BizTalking...

Installing, setting up and using the BizTalk SAP adapter

pkelcey — Tue, 28 Nov 2006 17:38:00 GMT

Wow, its been a busy couple of months here.

I was looking forward to getting to the Microsoft SOA conference in Seattle back at the start of October, but I ended up being pulled back to Toronto for a pressing BizTalk opportunity we had going. That opportunity pretty much took all of my time for the first couple of weeks of October. The team here was pretty busy running the Canadian BizTalk Partner Roadshow (outlined in my last post). I had a chance to meet some great partners during the roadshow, but I'll save that discussion for a future post.

For now, I wanted to some of the work we had to do for the that opportunity that kept me from getting to the Microsoft SOA conference. One of the things that we had to show was how the BizTalk SAP adapter could be installed, configured and used. Since the process can be a bit confusing and tricky, I thought I'd post the steps I followed.

Step 1 - The obvious first step is to install and configure BizTalk 2006. I won't go into the details of this as there are already a bunch of detailed documents out there that outline this process. Once this is done, you'll likely wonder where the SAP adapter is. It isn't installed as part of the core BizTalk Server installation process. The adapter comes in a separate installation package and you'll need to make sure you have this before you proceed further.

Step 2 - The SAP.NET Connector is a prerequisite for the BizTalk SAP adapter and must be installed before you try installing the adapter. This is the trickiest step for most people and is often the most frustrating part of the process. The frustration comes from trying to figure out what version of the connector you need to install for your BTS 2006 environment. There are multiple versions of the connector and if you don't have the correct one, then nothing is going to work for you. One of the most common errors people hit when installing connector is that the it has a dependancy on Visual Studio 2003. Since BizTalk 2006 is dependant on Visual Studio 2005, the installation fails and you're stuck. Beyond this specific error, there are a number of other errors that you can run into if you aren't using the correct version. The key to getting past these issue is to install the SAP.NET Connector Runtime version 1.0.3. The runtime version does not have the VS 2003 dependancy and you'll be able to install it in your BizTalk 2006 environment. A key note, the Connector is not provided by Microsoft and you will have to obtain this from SAP.

Step 3 -Before you get too far into this process, you'll want to contact your SAP administrator to get your hands on the connection/username/password information. You'll obviously need this information when we set about configuring the adapter. From my experience, getting connection information and username/password info can often take a few days. Therefore, I recommend asking for this as soon as you find out you'll need to use the adapter.

In all honesty, getting your hands on the connector and getting the proper connection information will take the majority of your time (It did for me). Once you're past these steps, the remaining steps can be completed quite quickly. In fact, I've attached a short AVI file that shows the remaining steps. Feel free to download it and use it however you need (training, client demos etc) as it really shows how simple it is to configure and use the adapter. The entire movie is only 4 minutes long and its about 1.5 megs zipped.

Step 5 - You'll need to install the SAP adapter. It's a pretty simple install and when its done, you'll be able to see the new SAP adapter using the Biztalk Admistration Console. (This is shown in the AVI)

Step 6 - Once the adapter is installed, we'll need to configure it and setup a send port that we can use to connect to the live environment and generate our message schemas. (Shown in the AVI) Note, that we want to configure the adapter before we do anything else in Visual Studio.

Step 7 - Now that the adapter is installed, we can use it to generate our schemas. To generate the SAP schemas, right click on your project, select "Add->Add Generated Item". From the pop-up window, select the "Add Adapter Metadata" option and the Adapter Metadata wizard will launch. One the wizard's first screen, select the SAP option. You'll need to provide the name of the SQL Server and the database where your SAP Adapter is configured. Once you have entered this info, the adapter you already created will show in the third drop down. You'll then be presented with a search screen that will allow you to search against the IDOCs and RFC functions in your SAP environment. (Note, you can use the * wildcard when searching). Once you select the IDOC or RFC you want to generate the schema for, select the next button and the schemas will be generated for you.

Setp 8 - You're done, go ahead and develop the rest of the BizTalk project as you normally would.

The AVI file will show as a link at the end of this post. I filmed it at 1024x768 so set your resolution to this before running.

Cheers and keep on BizTalking.

Peter

The parallel shape DOES NOT give you concurrent parallel processing - a common mistake

pkelcey — Tue, 22 Aug 2006 20:45:00 GMT

After running across this topic several times (and in several different places) over the past few weeks, I thought it would be worth writing up a quick blurb about it. This topic I've run into revolves around the parallel shape in BizTalk orchestrations and whether or not it actually provides you with parallel processing.

The answer is NO!

It's a common (and understandable) misconception that the parallel shape in BizTalk orchestrations is there to allow you to setup and run concurrent processes. After all, one would think that that if we have a parallel shape, then we'd be able to setup parallel processing. I've seen several clients and partners try and use this shape to increase performance by processing large messages using parallel activities. This is NOT what the shape is intended for as it does not result in your activities being performed in parallel. Activities that you put in parallel branches will still execute one at a time, however they will not have to execute in any specific order. I.e. a process involving steps A, B, C could execute as BCA or CAB or ABC.

Now it is understandable if this seems odd. It would seem obvious that a parallel shape should result in parallel processing. The trick here is to understand that what the parallel shape is referring to is a parallel activity in a business process and not a parallel activity in your computer program. The parallel shape isn't designed to instruct the system to run multiple threads, it is designed to allow multiple business activities to occur independently of each other without one having to wait for the other to complete.

A great example of this would be something like an online insurance quotation website. This site would likely want to use BizTalk to implement a scatter-gather pattern. Following this pattern, our online web application would submit a single message to BizTalk. This message would contain all of the information required to generate a quote for our end user. A BTS orchestration would be required to broadcast this message out to multiple external insurance companies and then wait for a message to come back from each one. (Example shown below).

(Note: I've borrowed this picture from www.enterpriseintegrationpatterns.com a great site)

Now because our orchestration will be waiting for external systems to respond, we won't have any control over the order in which each of these vendors will respond. You wouldn't want to create an orchestration that laid our your three receive shapes in sequence. Instead we'll use a parallel shape and put each of our receive functions in a separate branch. This will allow the orchestration to receive the return messages in whatever order they occur. We will still be receiving the messages one at a time, but we will be able to receive them in any order.

So keep this example in mind when using the parallel shape. You can't think of it as a low level instruction to BizTalk to launch two processing threads. You need to be thinking at a higher level in the processes stack and remember that the parallel shape is referring to the activities happening with the business process.

It is an important concept to understand as it can have performance impacts. In my next post, I'll look at a scenario where improperly using the parallel shape will actual decrease your performance instead of increasing it!

Cheers and keep on BizTalk.

Peter

BizTalk and Database Mirror: Supported or Not?

pkelcey — Fri, 14 Jul 2006 23:08:00 GMT

Lots of BizTalk developers and architects have been looking at the SQL Server 2005 database mirror option and are wondering if they can use this to implement a low cost disaster recovery solution for their BTS environments. It certainly is a tempting solution and it sure seems to be viable from technical point of view. Unfortunetly, our product team has stated that this is NOT a supported configuration. The reason for this is that there are key issues surrounding DTC and SQL Server before this configuraion will be viable in production systems. We've seen some clients try and implement this on their own and they have encountered issues around BizTalk continuing to look for the primary server after the failover has occured.

Personally, I'm hoping we might see these issues worked out in time for the R2 release, however that is just my own personal hope and can't be taken as an official timeline in any way.

Cheers and keep on BizTalking

New Interesting BizTalk Adapters from our Partners

pkelcey — Mon, 26 Jun 2006 20:12:00 GMT

I just wanted to point out a couple of interesting adapters that our partners are providing for BizTalk 2006.

Zouak consulting is providing a beta version of an adapter for integrating with RIM's Blackberry. You can find info about it at.

http://www.zouak.com/Products.aspx

Another partner, TwoConnect, has put together a full feature adapter for Salesforce.com's CRM solution. You can find info for it at.

http://www.twoconnect.com/pages/product_solutions/salesforce.aspx

Now I was fortunate enough to be able to talk with someone in our services division that has already used the salesforce adapter. He was in a great position to speak about the value of the adapter as he had also used the standard salesforce.com web services. He felt that using the adapter saved him about 2-3 weeks of development time over using the base web services interface. Now 2-3 weeks is a fair bit of time, so it seems like TwoConnect is providing a fairly valuable and affordable adapter here.

Cheers and keep on BizTalking

BizTalk and WinFX (oh, I mean .NET 3.0!)

pkelcey — Sun, 18 Jun 2006 23:04:00 GMT

There certainly been a lot going on around here. We've renamed WinFX to .NET 3.0, we've announced the R2 version of BizTalk 2006 and we've had one of our founders announce plans to transition out of his fulltime role. I had planned to drill down a bit into the first two items. With regards to the last item, I'll simply wish Bill all the luck in the world with a very noble cause.

So, WinFX is from now on going to be known as .NET 3.0. There's been a small firestorm of debate around this to a certain extent and I can understand why. However, I really believe that it's just a name and our time will be better spent in by trying to understand who the technologies contained within .NET 3.0 are going to be able to help development teams businesses.

So with that in mind, what is .NET 3.0 going to mean to BizTalk and BizTalk developers? Well, this question allows me to segway nicely into the recent release about BizTalk 2006 R2. The major changes that you're going to be seeing with R2 (slated for the first half of 2007!) are:

New capabilities for addressing concerns in the manufacturing and retail verticals. I'll talk about this in a later post.
Close alignment with Office 2007 and .NET 3.0. Let’s take a closer look at this one.

You'll need to be aware of three key areas of BizTalk that are going to be affected by this "alignment"

The thing is that R2 is going to be shipping with a Windows Communication Foundation (WCF) adapter included out of the box. This will allow BizTalk to connect to any WCF enabled application.

The second thing you should be aware of is that we're going to be releasing an extension to the WCF called the WinFX Adapter Framework (AF). This is a new framework that will allow for easier and simpler developer of WCF based adapters. You're probably asking, "What's the different between this and the BizTalk Adapter Framework". We'll, adapters built using the BAF can only be used by BizTalk whereas adapters built using AF can be used by any Windows application. Your adapters are no longer going to be limited to a single product. Some of you might now be asking, "If BizTalk has it's own adapter framework, how will it be able to use these new AF based adapters?" We'll that new WCF adapter that I spoke of in the last paragraph is going to let you tie these new adapters into your BizTalk project. Please note, these new AF adapters aren't there to replace BizTalk, they will only provide simple point to point connectivity. You're still going to need BizTalk for handling complex scenarios like process orchestration, long running transactions, business activity monitoring, business rules deployment etc...

The third thing that you should be aware of is Windows Workflow Foundation (WF) and its impact on BizTalk. WF is our new technology for creating and running any and all types of workflow. Human level workflow, document-centric workflow, user interface page flow and even workflow across a Service Oriented Architecture can all be developed using WF. The key thing for you to understand is that WF is certainly not here to kill or replace BizTalk! In fact, it will allow BizTalk developers to create even richer applications as they will be able to integrate their enterprise level orchestrations with workflow contained in other Microsoft products such as Office 2007 and SharePoint. BizTalk will still be the tool for implementing workflows and orchestrations across applications while WF will be for creating workflows within applications. You might be wondering if BizTalk is going to use WF to power its orchestrations. For the short term, no, BizTalk 2006 R2 is going to continue to use the architecture unveiled with BizTalk 2004. The next version (not R2, the next full release) will introduce WF as its workflow technology.

I know there are going to be an insane number of questions out there from developers, architects and business decision makers. We'll do our best to answer them and I'm planning on addressing as many as I can here. We’ll likely be seeing the usual questions about whether or not WCF + WF can be used to replace BizTalk, but we’ll deal with those as they come. The key things to focus on right now is that .NET 3.0 is going to be delivery new features and capabilities for BizTalk developers. Lets dive into these and make sure we can get the most from them.

Cheers for now

Peter

Consolidate to a single BizTalk database? Yes you can!

pkelcey — Mon, 01 May 2006 23:15:00 GMT

I ran into a interesting and undocumented feature of BizTalk 2006 a couple weeks ago that I thought I'd share. A colleague of mine, Jim Bowyer, had found out that with BizTalk 2006, you can consolidate most of your BizTalk database into a single one.

For those of you that aren't completely familiar with the internals of BizTalk, the following SQL Server databases are used to power a BizTalk environment (Note: some may not appear in your environment if you chose not to install all available BizTalk features)

MessageBox Databases
SSO Credential Database
BizTalk Analysis Database
BizTalk Configuration Database
Human Workflow Services Admin Database
BizTalk Rules Engine Database
BizTalk Base EDI Database
BizTalk Tracking Database
Notification Service
Windows SharePoint Services Configuration
Windows SharePoint Services Content
BAM Star Schema
BAM Primary Import
BAM Archive
BAM Analysis (OLAP)
BAM Notification

(For a more complete outline of these DB's (and what they are used for) check out this page )

Now that's quite a number of databases to maintain and manage, so it's natural that smaller organizations might be slightly intimidated about having to install, manage and support so many. I found out last week that a new feature has been introduced to BTS 2006 that will allow smaller customers to consolidate nearly all of these databases into single one. (The exceptions being the BAM Notification and EDI DBs).

Now since this feature was introduced post Beta 2, the documentation surrounding it is pretty weak. Additionally, the guidance around this concept is still almost non-existent. So here's what I've been able to find out from our product team.

Reasons for:

You are looking for a simplified install
You are looking for simplified management post install
You are looking for new variations of disaster recovery beyond the standard ones recommended with a multi-DB installation (i.e. you might be able to use database mirroring instead of having to rely on log shipping. Note: our product team has fully tested this and therefore doesn't support it yet)

Reason against:

You are really going to limit your ability to scale out your DB tier. Under a normal install, you would be able to place different DBs on different machines. You lose that ability under this type of set
The testing that has been done for this type of install is admittedly very small. I haven't seen any real numbers around potential performance impacts that this might have.
The lack of documentation, testing and formal support for this feature

It should be noted that this feature is included for small BTS installations and shouldn't be considered for larger enterprises.

If you do decide that you do want to go ahead with this, here's how:

During the configuration stage of your install, you can specify that you would like to install all features into the same database. (All except for the EDI and BAMAlertsNSMain databases - as stated above). When you fire up the BizTalk Server Configuration tool, and you're at the stage where you can select the features you want to install. This is where you can specific the consolidation. You'll need to specify the same database name for each feature (except EDI and BAM Alerts). After you do this, remember to adjust the connection settings when you use the BizTalk Server 2006 Administration MMC Snap-in. By default, it will want to connect to a database called BizTalkMgmtDb. If you consolidate your databases into a one new database then you need to specify that new one in the MMC. You do this by right-clicking the BizTalk Server 2006 Administration node and selecting “Connect To Existing Group“. This gives you the option to select a SQL Server and Database that the management db lives on.

Hope that helps and keep on BizTalking...