Connected Systems in the Great White North

ESB Toolkit How To Video #4: Dynamic Itinerary Resolution

Update: After I posted this originally, I was notified that my video did not contain any audio! Therefore, it was probably difficult to understand what was going on :) I've recreated the video and replaced the old one so that the video and audio are both working. Cheers

Welcome to #4 in my series of ESB Toolkit How To Videos. Each of these videos has built up from the one before it, so if you have seen the previous videos, I encourage you to do so. In video #3 (found here), I showed you how to use the toolkits new "Itinerary Resolution" feature.  In that video, I showed you how to use the Static Itinerary Resolver within a BizTalk receive port to load up an itinerary from the itinerary database and attach it to an incoming message. To do this, we added a resolver configuration string to the pipeline such as "ITINERARY:\\name=DemoService;" (where DemoService was the itinerary that we wanted to load) One shortcoming of that solution is that had multiple itineraries you would need a different receive port for each one.  In a real world solution, you might want to assign an itinerary to a message based on the message's schema, or some piece of data found within the message or based on who sent it and you would want to do all of this with as few receive ports as possible.  Therefore, the ESB really needs to be able to dynamically and intelligently figure out what itinerary an incoming message needed with us having to hard code the name.

Fortunately, the ESB Toolkit provides us a flexible and intelligent way to resolve itineraries and this is the Business Rules Engine resolver. This allows us to the use the BizTalk Rules Engine to make a decision about what itinerary should be used. We can create rules to look at Context properties associate with the message (such as the message type, the sender etc) or we can look at the content within the message to make our decision on which itinerary to use. In this video, I'll show you how to create an BRE policy to resolve itineraries and I'll show you how to setup your BizTalk receive ports to use this policy. I'll show you how to implement context and content based itinerary resolution.

Here's the link to the video

Cheers and keep on BizTalking...

Peter

ESB Toolkit How To Video #3: Itinerary Resolution in the Bus

Hi folks

Here's video #3 in my "ESB Toolkit How To" series.  In the last video (found here) we built out a complex itinerary that implemented dynamic mapping and dynamic routing. We also saw how to use the resolver mechanism to retrieve service information from a UDDI version 3.0 server. At the end of that video we used the Itinerary Test Client (which ships with the toolkit) to test our itinerary.  When I show this demo in person, I'm always asked the same question. "Does the client always have to submit the itinerary along with the message when they call the ESB?"  In version 1.0  of the ESB Toolkit, the was "Yes". The ESB did require the client to provide the itinerary along with the message. This was of course an issue for many of the architects I spoke to.  Most people wanted the ESB to be responsible for figuring out what itinerary was needed for an incoming message. Ideally, the client should just have to submit their message into the ESB and the bus would figure everything else out for them.  Requiring the client to create, manage and submit an itinerary is just not a solid architectural pattern.

Fortunately, in the ESB Toolkit 2.0, this is no longer a problem. We now have a great new feature called Itinerary Resolution that allows the ESB to figure out what itinerary a message should have.  Additionally, version 2.0 of the toolkit, gives us the new "ESB Itinerary Database". This DB gives us a central location to store our itineraries in and the Itinerary Resolution component is actually able to dynamically retrieve messages from this central database.  These two new features allow us to implement a solution where a client no longer has to have any knowledge of itineraries. Now, they simply have to pass in their message to the ESB where it will dynamically figure out what itinerary is needed and it will load that itinerary from the database.

In this video, I'll show you how to implement this feature. To demo this feature, I use an InfoPath form which sends data to a generic OnRamp which uses the Itinerary Resolver. I'll walk you through the process of storing your itinerary into the database, creating the new generic OnRamp that will use the itinerary resolution feature and I'll show you how to configure the Itinerary resolver to retrieve the itinerary from the database.

Here's the link to the video.

As always, I try to show as much detail as possible in these videos. If you feel that I have skimmed over something too quickly, or if some element is unclear, please let me know and I'll post a follow up video.

Cheers and keep on BizTalking...

Peter

ESB Toolkit - Life After Codeplex

So the ESB Toolkit is no longer going to be release through codeplex. So where do you register bugs and discuss issues? We have created a couple of new sites for this

You can register bugs on the Connect site at https://connect.microsoft.com/site/sitehome.aspx?SiteID=886&wa=wsignin1.0

There is also a new dedicate forum available  at http://social.msdn.microsoft.com/Forums/en-US/biztalkesb/threads these are great for posting issues since this is a monitored forum.

Cheers and keep on BizTalking...

Peter

ESB Toolkit How To Video #2: Service Composition

Hi everyone, it was quite a long time since I posted the first "How-to" video (found here), but I have finally completed the second one. In this demo, I show you how to do a number of new things

1. to create an itinerary that compose three services together into a single composite service
2. How to setup your BizTalk environment to support this type of itinerary. Including how to create the send ports required for a composite service itinerary
3. How to retrieve information from a UDDI v3 server from within your itinerary
4. How to test resolvers inside visual studio using the resolver web service
5. How to implement dynamic mapping and dynamic routing inside an itinerary

Based on feedback from my last video I understood that the AVI format I used sucked and that you didn't want to have to download multiple ZIP archives just to watch a short video. Therefore, I've found a better recording tool that creates WMV files. Therefore this video is only 13 megs in size.  Hopefully that is more convenient for everyone.

If you happen to hear a baby crying at the end, I apologize, that is my new daughter trying to make her first appearance on my blog :)

This video covers a lot of ground and I show a lot of things. So if I have gone too fast over something, or some element isn't clear, let me know and I'll post a follow up video.

Here's the link to the video

Cheers and keep on BizTalking...

Peter

ESB Toolkit 2.0 - Finally Here!

All the other BizTalk bloggers have probably already beaten me to this, but I still thought I'd mention that the ESB Toolkit 2.0 has finally been released in full. You can access it at http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=bc86cf1e-ef29-4b19-95f7-388f64555090

Cheers and keep on BizTalking

Peter

Disabling Itinerary Encryption in the ESB Toolkit 2.0

Those of you that worked with the CTP of the ESB Toolkit 2.0 will notice a new feature in the final release, namely "Itinerary Encryption". The Itinerary Design now allows you to use a certificate to encrypt your itineraries before you export them out of Visual Studio. This is a key new piece of functionality since your itineraries may potentially contain sensitive configuration information or sensitive processes that you do not want to leave exposed as open text XML.

In the properties window for the Itinerary Designer you can see a new property called "Encryption Certificate". You can use this property to select a certificate from a certificate store.

Now as important as this option is, what I'm going to write about is how to disable this.  On my dev machine, I did not have any valid certificates installed, so I wasn't able to select one to use or encryption. This prevented me from validating or exporting my itinerary since the validation tool kept throwing an error. Since this was only a dev machine, I didn't care about the security of these itineraries, so I really wanted to disable this feature so that I could keep working. Fortunetly, there is a simple and easy way to do this.

If you have installed the ESBT to the default location, you should be able to find a file called "ruleset.config" in the "C:\Program Files\Microsoft BizTalk ESB Toolkit 2.0\Tools\Itinerary Designer" folder. This file contains a list of validation rules the the Itinerary Designer uses when validating or exporting your itinerary.  If you open this file in Visual Studio, you will find a node called <property name="EncryptionCertificate">. Inside this node, you will see there are two rules that define how the validation of certificates should be handled. The first rule is the one the designer uses by default and it says that an error should be thrown if you do not have a certificate assigned. I commented out this rule and when I ran the validation routine again, I only received a warning message about the lack of a cert. I was then able to export my itinerary.  Here's what the modified file looked like for my system.

<property name="EncryptionCertificate">
<!--<validator type="Microsoft.Practices.Modeling.Validation.X509CertificateContainerValidator, Microsoft.Practices.Modeling.Validation"
messageTemplate="A X509 Certificate is required in the model property '{0}' to encrypt any sensitive property in the designer."
name="EncryptingCertificate validator"/>-->
<!-- Warning message when not enforcing encryption -->
<validator type="Microsoft.Practices.Modeling.Validation.X509CertificateContainerValidator, Microsoft.Practices.Modeling.Validation"
messageTemplate="Some data may not be secured because no X509 Certificate was specified in the model property '{0}'."
tag="Warning"        
name="EncryptingCertificate (warning) validator"/>
</property>

Cheers and keep on BizTalking

Peter

Goodbye ESBG? Hello ESBT???

Yes indeed, the ESB Guidance is going to experience a name change once version 2.0 is released later this month.  The ESB Toolkit 2.0 (set to release in mid June) offers so much more than just guidance that we needed to make the name more clear. I want to point out that this is more than just a name change or marketing scheme. Version 2.0 of the Toolkit 2.0 contains a ton of new enhancements over version 1.0 and since these enhancements are going to be key for most BizTalk shops, we needed to take some dramatic steps to mature the Toolkit from its initial “open source”/”best effort support” roots and make sure that is is enterprise ready and fully supported. We also realized the components in the toolkit offer a ton of value right out of the box and they no longer really act just as “guidance”. these are ready to use, valuable components for organizations building out their SOA infrastructure.

A couple of key points about ESBT 2.0:

1. From now on, the toolkit will be available via MSDN and not via Codeplex
2. We’re not going to be making the source code available like we did in version 1.0. This is tied to our planned changes around the toolkit’s support model.
3. As a first step to increasing support, we are launching a public forum on MSDN to provide stronger community support for the toolkit Microsoft employees will be monitoring this forum.
4. Beyond this, additional support measures are going to be phased in over the next several months. Eventually, you should be able to receive full support for the components in the toolkit via standard Microsoft support channels such as Premier.
5. Dramatically simplified install process (the old install guidelines I posted last year and now thankfully useless). I’ve managed to complete my latest install in less than 20 minutes.

Also, kudo’s need to go out to Dmitri Ossipov and his team who have done a great job of developing this. Most BizTalk shops will find that they will want to leverage some part of the toolkit for their future projects.

I’ve fallen drastically behind on my plans to release a number of “how to” videos for the ESB Toolkit 2.0. However, with the release coming soon, I' hope to change that quickly. Check back soon for some new “how to's” on two way synchronous routing, service composition (or service chaining), building your own ESB ready orchestrations etc.

Cheers and keep on BizTalking…

Peter

BizTalk 2009 Book and ESB Guidance

Some of you might be familiar with George Dunphy's "Pro BizTalk 2006" book that was released a couple of years ago through APress. Well, he's getting set to release a new one for BizTalk 2009 and I was very excited when he asked if I would be a contributing author. I've been given the task of writing a chapter about the ESB Guidance 2.0 for BizTalk 2009. There are a number of very smart and experienced guys contributing to the book, so if you're looking for a good BizTalk 2009 book, I'd recommend this one... (and of course I'm not biased ;) )

http://www.amazon.com/Pro-BizTalk-2009-George-Dunphy/dp/1430219815/ref=sr_1_1?ie=UTF8&s=books&qid=1239727570&sr=8-1

Cheers and keep on BizTalking

My SOA Conference Session on ESB is online

I had a chance last month to speak at the Microsoft SOA Conference about the latest version of the ESB Guidance for BizTalk Server (version 2.0).  The recordings from that conference have now been posted online.

You can find my ESB session at http://soaconference2009.spaces.live.com/blog/cns!FA6FC7F5DB1C07!136.entry

For a list of all the session recordings, check out

http://soaconference2009.spaces.live.com/default.aspx?wa=wsignin1.0&sa=351061685

Cheers and keep on BizTalking...

Peter

ESB Guidance 2.0 Screen Videos: Part 1 - Basic Routing

So I've been speak a lot about version 2.0 of the ESB Guidance and I thought I'd better get some of that content up to the blog. I'll be posting a series of demos that I've built that show how to use the new features and tools in V2.  In order to keep the blog entries shorter and easier to get out quickly, I'm just going filming me doing the demo and then I'll post the AVI file for you to watch. (i.e. pictures and videos being worth a thousand words or so) That saves me from having to take screen shots and to try and verbally explain what I did.

In part one, I'll be showing how to implement a basic routing scenario. I have a basic web service and test client setup. I want the ESB to route a message from the test client to the web service. I also want the ESB to determine where the web service's endpoint is by calling out to a UDDI server.

What I'll be showing is how this can be implemented using the basic routing capabilities of the ESB 2.0. I'll take you through the process of setting this up from start to finish.  We setup the BizTalk components, define the Itinerary (which defines the process flow within the ESB) and we'll test the end to end process. If any of those terms were new for you (i.e. itineraries, biztalk, uddi etc), then I would recommend you take a bit of time and review the basic concepts of the ESB guidance.

Those of you that worked with the ESB guidance 1.0 will already know how to implement the scenario that I just outlined. However one of the great new features of v2.0 is the the new visual itinerary design tool. It is an incredibly welcomed addition to the guidance as we no longer need to write the raw XML that itineraries are made of. 

Due to the size of the file, I've had to zip it and split it into 7 files. Be sure to download all of them before and then extra them into the AVI.

File 1

File 2

File 3

File 4

File 5

File 6

File 7

Hopefully this helps those of you who are new to the guidance and helps you to get using it more quickly.

Cheers and Keep on BizTalking...

Peter

Busy travelling, busy speaking…

So, I’ve been booked into the Microsoft 2009 SOA conference in Redmond at the end of this month to speak about the new version of the ESB Guidance (V2.0). For those of you who are familiar with version 1.0 of the guidance, I strongly recommend that you check out v2 (http://www.codeplex.com/esb). It has a number of great improvements that really help to "polish” up the concepts that were introduced in the initial release.  We now a visual Itinerary designer right inside VS 2008 (no more having to manually code up raw XML itineraries yeah!!!)  We’ve also got a new Itinerary database where we can store, version and retrieve our itineraries.  We also got a a new pipeline component called the ItineraryForwarder that allows us to chain together service calls without the need for an orchestrations (i.e. Call Service A and send the response to Service B and then send its response to service C).  We use to require a BizTalk orchestration for this, but now this kind of simple chaining can be done entirely using ports and pipelines.  Also, the team has done a nice job of cleaning up the install by giving us a Configuration application instead of requiring us to manually modify a bunch of different install scripts.

After I’m done at the SOA conference, I’ll have just enough time to fly home say a quick hello to my wife and bed before turning around and heading right back to Seattle to speak at our own internal TechReady conference about the same thing.  It appears like I’ll be making a big push early on in the year for that “elite” status on Air Canada… :)

Content from the Canadian Connected Systems Conference

We've been to Vancouver and Toronto so far and lots of people have been asking about the presentations that we delivered at the Connected Systems Conference.  We're uploading them now to a skydrive site and you should be able to access them at

http://cid-ece4536d84e10155.skydrive.live.com/browse.aspx/Connected%20Systems%20Conference%202008/Final%20Slides 

The site should be open for anyone to download the presentations and you won't need a password or ID.  It may take us a day to get all of the presenters to upload their decks, so if you don't see the one you wanted, give it a day and then check back.

Cheers and keep on BizTalking

Peter

The Canadian Connected Systems Conference 2008

a.       Vancouver – 30 October 2008

b.      Toronto – 4^th November 2008

c.       Montréal – 6^th November  2008

Overview
In a service-oriented world, effective business processes unite people and systems. Find out how BizTalk® Server builds on the Business Process Management and SOA capabilities in prior releases to help your organization extend core process management technologies even further. 

We will share the latest on Microsoft’s SOA offering - as well as details of the strategies and technologies that Microsoft is delivering today, over the next year, and into the future.  At this event, you will learn:

·         BizTalk today and through FY’09

·         Microsoft’s Roadmap for the future

·         Microsoft’s “Real world” SOA vision,
positioning and messaging

·         Service Virtualization & ESB guidance 2.0

·         RFID & mobility of BizTalk

These events will provide further insight for customers and technology partners into best practices for building SOA and BPM solutions, guidance for advancing SOA, and using the latest Microsoft-based technologies for connecting people, processes and information.

Agenda

Target Audience
This event will be of value to Technical Decision Makers, Developers and Architects interested in BizTalk, enterprise integration, and SOA solutions.

To Register
Space is limited so register today to ensure your attendance at this event.  Click the city name below to register:

Vancouver:   October 30, 2008     (Thurs)
Toronto:       November4, 2008     (Tues)
Montreal:      November 6, 2008    (Thursday)

How to implement retries on BizTalk ESB Off-Ramps

I'm sitting in the Vancouver airport reflecting on a very successful and interesting 2 day deep dive training course that we ran for our west coast partners. This was the first time that we ran the course over two days, and it allowed us to get a lot deeper into the code than we normally would in a one day session. Because of this, we received a lot of great questions from the developers who attended. While I'm waiting for my flight, I thought I'd blog about a couple of them.

Those of you familiar with the concept of Off-Ramps in the Microsoft ESB Guidance for BizTalk server will know that they are implemented using Dynamic BizTalk send ports. Dynamic ports have the ability to configure themselves at runtime based on contextual data stored within a message. The message itself is able to tell the dynamic port how to function, what transport protocol to use, what endpoint to point towards etc.   The ESB guidance contains an adapter provider mechanism and a resolver mechanism which allows you to store this configuration data in external data stores like a UDDI server, SQL Server, a rules engine etc.  The ESB resolver mechanism looks up this information from this remote data store and feeds it into an adapter provider. The adapter provider then converts this data into a format that a BizTalk dynamic port can use to configure itself.

BizTalk also has the concept of static ports which do not load configuration data at runtime but instead require an administrator to hard code configuration data right into the port. Static ports are not as flexible as dynamic ports and are therefore not as effective when implementing an ESB pattern. However, they do have a number of interesting features that aren't seen in dynamic ports. One of these features is the ability to define a retry interval and a set number of retries.  If the port fails to transmit a message successfully the first time, it will shut down and then retry the message after a set period of time. The port will continue to try and resend the message until it reaches a maximum number of retries. A BizTalk administrator can set the length of the retry interval and the number of retries for a static port directly in the BizTalk administration console.  However, when you create a dynamic port in the admin console, these configuration options do not show up. The question that one attendee asked was

"Since the ESB guidance is heavily dependant on dynamic ports , is it unable to support retries"?

Fortunately the answer is definite "NO". The ESB guidance with its adapter providers is able to support retries. To do this, you need to tweak the adapter providers classes to tell them to set this property for us.

The adapter providers effectively work like a dynamic runtime level BizTalk administrator, they provide a complete set of configuration information to the dynamic port.  The adapter providers pull this information from the resolver mechanism which is able to connect to the UDDI server, database server etc. In the basic guidance, the adapter providers pull down a small subset of configuration information from the resolver mechanism. We can modify this to pull additional information such as retry settings.

I've outlined the steps below that you need to follow in order to implement retry functionality. For this walk-through, I'm going to use the UDDI server to store our information. I'm also going to be working with the WCF-basicHTTP adapter provider that allows us to implement SOAP 1.1 messaging

1)  The first step is to modify the service entry in the UDDI server so that it contains the retry settings.  Open up the service entry in the ESB that you want to implement retries for. Add a new binding to this service call called "RetryCount" and give it a value of "5" (i.e. the adapter will retry 5 times). Create a second new binding called "RetryInterval" and set this equal to "1" (this means retry every 1 minute).

2) We don't need to modify the resolver mechanism as it is already built to automatically pull down any and all information from the UDDI registry. As long as we put the data in there, it will pull it down and given it to the adapter provider, which we'll do next.

3) Open up the Microsoft.Practices.ESB.sln solution file and open the ESB.Adapter.WCF-basicHTTP project. Inside this project, open the AdapterProvider.cs class.  If you scroll down to lines 82 and 83, you'll see where the adapter provider takes the settings retrieved from the resolver and attaches them to the outgoing message as adapter properties.  You will need to add the following 2 lines in the section.

pipelineContext.Write(BTS.RetryCount,"http://schemas.microsoft.com/BizTalk/2003"  , ResolverDictionary["Resolver.RetryCount"] );
pipelineContext.Write(BTS.RetryInterval, "http://schemas.microsoft.com/BizTalk/2003", ResolverDictionary["Resolver.RetryInterval"] );

This code will retrieve the settings we made in UDDI and will attach them to the outgoing message context. The dynamic adapter will then use these settings when it attempts to send the message.

4) Build and deploy the new version of the adapter provider project.

5) Restart the BizTalk host instance and the IIS service instance

That's it, you've implemented retry capabilities in ESB guidance for the WCF-basicHTTP OffRamp.

Cheers and keep on BizTalking

A Web based version of the BizTalk Rules Composer

As part of a proof-of-concept I was working on last year, we needed to show that the BizTalk Business Rules could be created and managed via a Web Application instead of through the standard Business Rules Composer tool that we provide out of the box. The client had certain requirements that meant that using the standard Composer just wasn't viable. Therefore, I replicated a portion of the Composer as a Web based application. I thought I'd post some images from the app and see if this is something people are interested in. If you are, I'll work to clean up the code and post it to http://www.codeplex.com/. Let me know by posting a comment here.

 There were two tools in the. One let you view and edit vocabularies. The other let you view/author/edit/delete/publish/deploy policies and rules.

Image 1: The home page, gives the user the ability to access the 2 tools contained in the app.

Image 2: The Policy and Rule Explorer This screen lets a user select a specific Policy to review. Once they select a Policy, they are shown the entire version history for it. For each version, then can load up a list of all the rules contained within that version. At this point, they have the option to publish, deploy or copy that particular version. They also have the ability to select each rule and view its Conditions and actions.

Image 3: Adding a New Rule

Image 4: The Vocabulary Explorer. Lets a user view the definitions contained within all the Vocabularies and lets them alter the values of existing definitions.

Cheers and keep on BizTalking...

Peter