Connected Systems in the Great White North

PDC & Source Code for ESB Toolkit How To Videos

2009-10-28T17:18:00Z

Since I started releasing the ESB Toolkit how-to videos, I’ve had several people ask me for the source code for all of the components I demo’d. Well, it took me way to long to get them packaged up, but here they are finally. I’ve packaged up everything that I used during the demos and it should be fairly straight forward for you to install it. There is an installation guide (Word 2007 format) contained in the attached ZIP archive that walks you through the install process.

The reason that I finally got around to getting this done is that my examples are now going to be used for the hands-on-labs at PDC 2009. If you want to get some hands on experience with the ESB Toolkit, drop by the Hands-On-Labs area of PDC and you will be able to work through the same projects that I have demo’d in the ESB Toolkit How-To series.

Cheers and keep on BizTalking…

Peter

SCOM Management Pack for BizTalk 2009 is Available.

2009-09-01T17:24:42Z

To all you folks who have been waiting for the new Management Pack for BizTalk 2009 to be released, it is finally here. You can download it at

http://www.microsoft.com/downloads/details.aspx?FamilyId=389FCB89-F4CF-46D7-BC6E-57830D234F91&displaylang=en&displaylang=en

Cheers and keep on BizTalking…

Peter

“Pro BizTalk 2009” Book is Finally Available

2009-08-25T17:04:39Z

Yeah! Pro BizTalk 2009 is now published and available. It was an interesting project to be involved with (even in just a small way). You can find it on Amazon now at http://www.amazon.ca/Pro-BizTalk-2009-George-Dunphy/dp/1430219815/ref=sr_1_2?ie=UTF8&s=books&qid=1251208723&sr=8-2

I wanted to thank Richard Seroter who acted as the technical reviewer for my chapter. I also wanted to point out his great book “SOA Patterns with BizTalk 2009” (in case anyone hasn’t already seen it).

Cheers and keep on BizTalking…

Peter

ESB Toolkit How To Video #7: A SharePoint Adapter Provider

2009-08-05T18:39:31Z

Welcome to #7 in my series of ESB Toolkit How To Videos. If you have not seen the previous videos, I encourage you to do so. The previous ones can be found here

1) Basic Itinerary Routing and UDDI Integration

2) Composite Itinerary and Dynamic Mapping

3) Itinerary Resolution in the Bus

4) Dynamic Itinerary Resolution in the Bus

5) Including Custom Orchestrations in the Itinerary Designer

6) Performance Metrics using Built in BAM

One very common thing that people will want the ESB to do is to route message to SharePoint where they can potentially kick off human based workflow. However, the base ESB Toolkit does not have an adapter provider for Windows SharePoint Services. BizTalk does have an "Adapter" for WSS, however the ESB toolkit does not have an "Adapter provider" for WSS. The Adapter Providers in the ESBT bridge between the new ESBT code and the traditional BizTalk components.

Now fortunately, the ESBT can be extended very quickly and we can add in our own adapter providers without the need for very much code at all. This past week, I was working with the guys over at QuickLearn to build out a WSS adapter provider that they planned to use in an upcoming BizTalk/ESBT demo. In this video, I'll show you how we created the adapter provider (using only a few lines of code) and how we registered it so that the design time tools and the runtime engine could use it. Now, while creating the adapter doesn't require much effort, I will admit that I had a hell of a time trying to get it registered properly and all of the associated configuration files aligned correctly. The ESBT relies heavily on reflection to load components at runtime, so if you don't name your component properly or register it correctly in the config files, then the ESBT engine won't be able to load the provider properly and you'll have a fun time trying to debug just exactly what you did wrong.

In order to make it easier for you, I've included my code and configuration files for download so that you don't have to rebuild everything that I show in the video.

I need to credit Nick Hauenstein over at QuickLearn (http://www.quicklearn.com/) for doing most of the initial leg work on building this out.

You can access the video here

You can download the project here

Cheers and keep on BizTalking

Peter

ESBT Toolkit How To Video #6: Performance Metrics using Built in BAM

2009-07-24T04:32:00Z

Welcome to #6 in my series of ESB Toolkit How To Videos. If you have not seen the previous videos, I encourage you to do so. The previous ones can be found here

1) Basic Itinerary Routing and UDDI Integration

2) Composite Itinerary and Dynamic Mapping

3) Itinerary Resolution in the Bus

4) Dynamic Itinerary Resolution in the Bus

5) Including Custom Orchestrations in the Itinerary Designer

There is a great (but sadly under appreciated and under discussed) feature in the ESB Toolkit that I want to focus on today. Back at last years SOA conference, I spent some time talking with Dmitri Ossipov (who lead the development of the ESBT) and he showed me some of the new features he was adding into the toolkit. The one that caught my eye immediately was the built in BAM tracking. The ESB Toolkit has the built in ability to automatically log tracking and performance information for your itineraries as they are processed by the ESB components. If you have enabled "Tracking" in your itineraries, then the ESBT automatically logs information to a BAM table in SQL Server every time any of the itineraries services execute. This data includes data about which service executed, which itinerary and version it was part of, what time it executed, what specific instance it was part of as well as the status of the overall itinerary. This data is a gold mine for those of you looking to track the overall performance of your ESB and related services. It can be used to view all kinds of invaluable information about which itineraries are being used, how well they are performing, which ones are too slow, which ones are failing, which individual services are under performing etc.

Now while the ESBT will automatically capture all of this information you, it doesn't come with any kind of user friendly mechanism for viewing, analyzing or interacting with this data. In this video, I'll show you how to easily setup a number of BAM Views that will allow you to quickly assess performance stats about your itineraries. I build these views out in Excel and that allows me to use standard pivot tables and charts to view the data and I slice it up. I have also created views that allow me to tracking the performance of specific itineraries across different versions and across time. This is great info to have if you want to understand if recent changes to an itinerary have resulted in better or worse performance than you previously had.

The following image shows one of the views that I setup to show me what the average, worst and best performance times were for my itineraries. This is pretty useful information to have as I can quickly identify performance issues in my itineraries. You can easily see that the "ComposedDemo" takes quite slow when compared to the "DemoService" itinerary. I can even drill down in the specifics of a single itinerary and find out if a specific ESB service is causing the overall poor performance.

Now, unlike my previous videos, where I just showed you how to build out a specific solution, I'm going to actually provide you with the components I've built. I've attached the Excel file that contains my BAM Views and you should be able to easily deploy these into your system and access the charts and tables that I show in the video. You will just need to have the ESBT installed as well as the BizTalk BAM components installed and configured.

In order to avoid make an hour long video, I do not go very deep into what BAM is or how you work it. If you are not familiar with BizTalk's BAM feature, there may be some elements of this video that are not 100% clear. However, if you have a basic understanding of what BAM is, then I hope this video will offer some great value.

The video can be accessed here:

The BAM Excel Project can be accessed here:

Cheers and keep on BizTalking

Peter

ESB Toolkit How To Video #5: Including Custom Orchestrations in Itineraries

2009-07-17T22:35:05Z

Update: Sorry folks, the first link I posted for the video was incorrect. I have updated it now:

Welcome to #5 in my series of ESB Toolkit How To Videos. If you have not seen the previous videos, I encourage you to do so. The previous ones can be found here

1) Basic Itinerary Routing and UDDI Integration

2) Composite Itinerary and Dynamic Mapping

3) Itinerary Resolution in the Bus

4) Dynamic Itinerary Resolution in the Bus

One of the great things about the ESB Toolkit is that it can be extended in a large number of ways. One of the common ways that developers will want to extend it, is to create custom BizTalk orchestrations that can be included within an ESB itinerary. With the default install of the Toolkit, you have access to two orchestration. The generic routing orch and the generic mapping orch. In this video, I'll show you how to create a new orchestration and then register it with the ESB so that you can access it from the itinerary designer and create an itinerary that will route messages to it.

The video can be accessed here

Cheers and keep on BizTalking...

Peter

ESB Toolkit How To Video #4: Dynamic Itinerary Resolution

2009-06-17T18:37:39Z

Update: After I posted this originally, I was notified that my video did not contain any audio! Therefore, it was probably difficult to understand what was going on :) I've recreated the video and replaced the old one so that the video and audio are both working. Cheers

Welcome to #4 in my series of ESB Toolkit How To Videos. Each of these videos has built up from the one before it, so if you have seen the previous videos, I encourage you to do so. In video #3 (found here), I showed you how to use the toolkits new "Itinerary Resolution" feature. In that video, I showed you how to use the Static Itinerary Resolver within a BizTalk receive port to load up an itinerary from the itinerary database and attach it to an incoming message. To do this, we added a resolver configuration string to the pipeline such as "ITINERARY:\\name=DemoService;" (where DemoService was the itinerary that we wanted to load) One shortcoming of that solution is that had multiple itineraries you would need a different receive port for each one. In a real world solution, you might want to assign an itinerary to a message based on the message's schema, or some piece of data found within the message or based on who sent it and you would want to do all of this with as few receive ports as possible. Therefore, the ESB really needs to be able to dynamically and intelligently figure out what itinerary an incoming message needed with us having to hard code the name.

Fortunately, the ESB Toolkit provides us a flexible and intelligent way to resolve itineraries and this is the Business Rules Engine resolver. This allows us to the use the BizTalk Rules Engine to make a decision about what itinerary should be used. We can create rules to look at Context properties associate with the message (such as the message type, the sender etc) or we can look at the content within the message to make our decision on which itinerary to use. In this video, I'll show you how to create an BRE policy to resolve itineraries and I'll show you how to setup your BizTalk receive ports to use this policy. I'll show you how to implement context and content based itinerary resolution.

Here's the link to the video

Cheers and keep on BizTalking...

Peter

ESB Toolkit How To Video #3: Itinerary Resolution in the Bus

2009-06-15T17:57:08Z

Hi folks

Here's video #3 in my "ESB Toolkit How To" series. In the last video (found here) we built out a complex itinerary that implemented dynamic mapping and dynamic routing. We also saw how to use the resolver mechanism to retrieve service information from a UDDI version 3.0 server. At the end of that video we used the Itinerary Test Client (which ships with the toolkit) to test our itinerary. When I show this demo in person, I'm always asked the same question. "Does the client always have to submit the itinerary along with the message when they call the ESB?" In version 1.0 of the ESB Toolkit, the was "Yes". The ESB did require the client to provide the itinerary along with the message. This was of course an issue for many of the architects I spoke to. Most people wanted the ESB to be responsible for figuring out what itinerary was needed for an incoming message. Ideally, the client should just have to submit their message into the ESB and the bus would figure everything else out for them. Requiring the client to create, manage and submit an itinerary is just not a solid architectural pattern.

Fortunately, in the ESB Toolkit 2.0, this is no longer a problem. We now have a great new feature called Itinerary Resolution that allows the ESB to figure out what itinerary a message should have. Additionally, version 2.0 of the toolkit, gives us the new "ESB Itinerary Database". This DB gives us a central location to store our itineraries in and the Itinerary Resolution component is actually able to dynamically retrieve messages from this central database. These two new features allow us to implement a solution where a client no longer has to have any knowledge of itineraries. Now, they simply have to pass in their message to the ESB where it will dynamically figure out what itinerary is needed and it will load that itinerary from the database.

In this video, I'll show you how to implement this feature. To demo this feature, I use an InfoPath form which sends data to a generic OnRamp which uses the Itinerary Resolver. I'll walk you through the process of storing your itinerary into the database, creating the new generic OnRamp that will use the itinerary resolution feature and I'll show you how to configure the Itinerary resolver to retrieve the itinerary from the database.

Here's the link to the video.

As always, I try to show as much detail as possible in these videos. If you feel that I have skimmed over something too quickly, or if some element is unclear, please let me know and I'll post a follow up video.

Cheers and keep on BizTalking...

Peter

ESB Toolkit - Life After Codeplex

2009-06-12T23:14:56Z

So the ESB Toolkit is no longer going to be release through codeplex. So where do you register bugs and discuss issues? We have created a couple of new sites for this

You can register bugs on the Connect site at https://connect.microsoft.com/site/sitehome.aspx?SiteID=886&wa=wsignin1.0

There is also a new dedicate forum available at http://social.msdn.microsoft.com/Forums/en-US/biztalkesb/threads these are great for posting issues since this is a monitored forum.

Cheers and keep on BizTalking...

Peter

ESB Toolkit How To Video #2: Service Composition

2009-06-12T22:57:00Z

Hi everyone, it was quite a long time since I posted the first "How-to" video (found here), but I have finally completed the second one. In this demo, I show you how to do a number of new things

to create an itinerary that compose three services together into a single composite service
How to setup your BizTalk environment to support this type of itinerary. Including how to create the send ports required for a composite service itinerary
How to retrieve information from a UDDI v3 server from within your itinerary
How to test resolvers inside visual studio using the resolver web service
How to implement dynamic mapping and dynamic routing inside an itinerary

Based on feedback from my last video I understood that the AVI format I used sucked and that you didn't want to have to download multiple ZIP archives just to watch a short video. Therefore, I've found a better recording tool that creates WMV files. Therefore this video is only 13 megs in size. Hopefully that is more convenient for everyone.

If you happen to hear a baby crying at the end, I apologize, that is my new daughter trying to make her first appearance on my blog :)

This video covers a lot of ground and I show a lot of things. So if I have gone too fast over something, or some element isn't clear, let me know and I'll post a follow up video.

Here's the link to the video

Cheers and keep on BizTalking...

Peter

ESB Toolkit 2.0 - Finally Here!

2009-06-09T19:21:14Z

All the other BizTalk bloggers have probably already beaten me to this, but I still thought I'd mention that the ESB Toolkit 2.0 has finally been released in full. You can access it at http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=bc86cf1e-ef29-4b19-95f7-388f64555090

Cheers and keep on BizTalking

Peter

Disabling Itinerary Encryption in the ESB Toolkit 2.0

2009-06-08T23:14:00Z

Those of you that worked with the CTP of the ESB Toolkit 2.0 will notice a new feature in the final release, namely "Itinerary Encryption". The Itinerary Design now allows you to use a certificate to encrypt your itineraries before you export them out of Visual Studio. This is a key new piece of functionality since your itineraries may potentially contain sensitive configuration information or sensitive processes that you do not want to leave exposed as open text XML.

In the properties window for the Itinerary Designer you can see a new property called "Encryption Certificate". You can use this property to select a certificate from a certificate store.

Now as important as this option is, what I'm going to write about is how to disable this. On my dev machine, I did not have any valid certificates installed, so I wasn't able to select one to use or encryption. This prevented me from validating or exporting my itinerary since the validation tool kept throwing an error. Since this was only a dev machine, I didn't care about the security of these itineraries, so I really wanted to disable this feature so that I could keep working. Fortunetly, there is a simple and easy way to do this.

If you have installed the ESBT to the default location, you should be able to find a file called "ruleset.config" in the "C:\Program Files\Microsoft BizTalk ESB Toolkit 2.0\Tools\Itinerary Designer" folder. This file contains a list of validation rules the the Itinerary Designer uses when validating or exporting your itinerary. If you open this file in Visual Studio, you will find a node called <property name="EncryptionCertificate">. Inside this node, you will see there are two rules that define how the validation of certificates should be handled. The first rule is the one the designer uses by default and it says that an error should be thrown if you do not have a certificate assigned. I commented out this rule and when I ran the validation routine again, I only received a warning message about the lack of a cert. I was then able to export my itinerary. Here's what the modified file looked like for my system.

Cheers and keep on BizTalking

Peter

Goodbye ESBG? Hello ESBT???

2009-06-05T17:55:57Z

Yes indeed, the ESB Guidance is going to experience a name change once version 2.0 is released later this month. The ESB Toolkit 2.0 (set to release in mid June) offers so much more than just guidance that we needed to make the name more clear. I want to point out that this is more than just a name change or marketing scheme. Version 2.0 of the Toolkit 2.0 contains a ton of new enhancements over version 1.0 and since these enhancements are going to be key for most BizTalk shops, we needed to take some dramatic steps to mature the Toolkit from its initial “open source”/”best effort support” roots and make sure that is is enterprise ready and fully supported. We also realized the components in the toolkit offer a ton of value right out of the box and they no longer really act just as “guidance”. these are ready to use, valuable components for organizations building out their SOA infrastructure.

A couple of key points about ESBT 2.0:

From now on, the toolkit will be available via MSDN and not via Codeplex
We’re not going to be making the source code available like we did in version 1.0. This is tied to our planned changes around the toolkit’s support model.
As a first step to increasing support, we are launching a public forum on MSDN to provide stronger community support for the toolkit Microsoft employees will be monitoring this forum.
Beyond this, additional support measures are going to be phased in over the next several months. Eventually, you should be able to receive full support for the components in the toolkit via standard Microsoft support channels such as Premier.
Dramatically simplified install process (the old install guidelines I posted last year and now thankfully useless). I’ve managed to complete my latest install in less than 20 minutes.

Also, kudo’s need to go out to Dmitri Ossipov and his team who have done a great job of developing this. Most BizTalk shops will find that they will want to leverage some part of the toolkit for their future projects.

I’ve fallen drastically behind on my plans to release a number of “how to” videos for the ESB Toolkit 2.0. However, with the release coming soon, I' hope to change that quickly. Check back soon for some new “how to's” on two way synchronous routing, service composition (or service chaining), building your own ESB ready orchestrations etc.

Cheers and keep on BizTalking…

Peter

BizTalk 2009 Book and ESB Guidance

2009-04-14T20:06:22Z

Some of you might be familiar with George Dunphy's "Pro BizTalk 2006" book that was released a couple of years ago through APress. Well, he's getting set to release a new one for BizTalk 2009 and I was very excited when he asked if I would be a contributing author. I've been given the task of writing a chapter about the ESB Guidance 2.0 for BizTalk 2009. There are a number of very smart and experienced guys contributing to the book, so if you're looking for a good BizTalk 2009 book, I'd recommend this one... (and of course I'm not biased ;) )

http://www.amazon.com/Pro-BizTalk-2009-George-Dunphy/dp/1430219815/ref=sr_1_1?ie=UTF8&s=books&qid=1239727570&sr=8-1

Cheers and keep on BizTalking

My SOA Conference Session on ESB is online

2009-02-28T03:33:17Z

I had a chance last month to speak at the Microsoft SOA Conference about the latest version of the ESB Guidance for BizTalk Server (version 2.0). The recordings from that conference have now been posted online.

You can find my ESB session at http://soaconference2009.spaces.live.com/blog/cns!FA6FC7F5DB1C07!136.entry

For a list of all the session recordings, check out

http://soaconference2009.spaces.live.com/default.aspx?wa=wsignin1.0&sa=351061685

Cheers and keep on BizTalking...

Peter