David Hurtado's Integration Traces

Introduction to xRM. One Platform. Many Applications

dhtoran — Thu, 16 Jul 2009 12:14:59 GMT

Although xRM is not directly related to me, my job and my area of expertise, I really like this video and the idea behind it.

Enjoy!

Bumptop and User Experience: Creating new problems and solving them

dhtoran — Mon, 15 Jun 2009 17:27:47 GMT

This post has nothing to do with BizTalk or EAI/SOA/etc, but it’s interesting, so here it goes!

I’ve a thread with some colleagues about User Experience (something I see myself as an enthusiast). Somebody shared a video about Bumptop.

As the YouTube description states, “BumpTop is a fun, intuitive 3D desktop that keeps you organized and makes you more productive. Like a real desk, but better”.

Well, my point of view is quite different. The above picture shows all but organization. It’s a complete mess.

In my opinion, Bumptop is a huge effort to recreate the realworld™ desktop inside a Windows desktop. It’s a huge effort, and a huge step backwards: it has all lots of problems inherited from realworld™. In fact, in the picture abobe shows nothing but bad practices in tidiness. If only your mom saw this…

Lets take a feature, taken from the video, that will make your mom happier: it allows the user to tidy things by applying a mouse gesture over an untidy area, so everything inside the gesture get automatically organized in a small and nice pile.

Well, basically, this feature solves a problem that was created by Bumptop, which sounds funny to me! You can’t have the initial mess without Bumptop.

Even more funny is a Media Love comment found in Bumptop home page. It states:

“Bumptop has become an indispensable tool for my company. Between stock imagery, fonts, plug-ins, or .pdfs, I am downloading anywhere between 12-25 files a day onto my desktop […]”

¿downloading 12-25 files a day onto the desktop? Well, I really recommend this person to go and attend to some kind of training such as “using a personal computer for dummies” or similar. I really can’t imagine a worse and less productive use of the desktop as a “download store”.

Here goes the video:

Also fun that some interesting features of Bumptop, shown in the video, has nothing to do with 3D or realworld™ metaphors, they are just some very clever use of drag and drop and file handlers.

Book review: SOA Patterns with BizTalk Server 2009 (II) – WCF Primer

dhtoran — Tue, 09 Jun 2009 15:39:32 GMT

This is a part of small posts reviewing the book ‘SOA Patterns with BizTalk Server 2009’, by Richard Seroter.

Chapter II – an WCF Primer

The second chapter has been a little surprise for me: it is a WCF primer and has nothing to do with BizTalk Server. Why?

There are two reasons for writing a WCF primer when dealing with BizTalk Server: the first one is that WCF is not a “technology replacement for ASMX web services”, as a lot people see it. WCF is a communications subsystem. A full communications subsystem that most of future .NET developments will be based upon.

The second reason is that the new BizTalk WCF Adapter + replaces most of the old adapters found in common scenarios. WCF plays a very important role in BizTalk Server.

So, back to the Seroter’s book, how’s the WCF primer like?

It’s basically a very deep and detailed document about WCF. The chapter succeed in isolate the concepts of contracts, service implementations, bindings and service hosting. It also includes a detailed end-to-end sample including all the concepts. I’ll definitely recommend this chapter to anyone interested in learning WCF.

ESB Toolkit 2.0 released

dhtoran — Tue, 09 Jun 2009 09:30:12 GMT

Good news that the BizTalk ESB Toolkit 2.0 (previously known as ESB Guidance 2.0) has been released for download.

The download is no longer published on CodePlex. Now you can download it from the BizTalk Developer Center. Changes in the name and the download links have a good reason: now it’s supported by Microsoft.

Even if you are new to BizTalk or have no experience with Enterprise Service Bus pattern, it really worth to have a look at it.

I’m gonna copy/paste some features from the ESB site:

· Endpoint run-time discovery and virtualization

· Loosely coupled service composition

· Dynamic message transformation and translation

· Dynamic routing

· Centralized exception management

· Quality of service

· Protocol transformation

· Extensibility

Book review: SOA Patterns with BizTalk Server 2009 (I) - Refresher

dhtoran — Mon, 01 Jun 2009 11:45:46 GMT

This is a part of small posts reviewing the book ‘SOA Patterns with BizTalk Server 2009’, by Richard Seroter. I think I’ll be posting chapter reviews and thoughts as I read them.

Preface and Chapter I – BizTalk Server refresher

The chapter one, as one can imagine, is more or less a refresher about what BizTalk Server is and what can it perform. I was not going to post about it since it seems to be very basic: it just creates the context for the rest of the book.

But as I was reading it yesterday and I was thinking an issue I had with one of my customers last week (talking about what BTS is and how to can it be used in different projects), I realized that introductions are more important when dealing with BizTalk Server. Why?

Basically because BizTalk Server is a thing that is difficult to explain. What is BizTalk Server? Is it a product or a platform? Is it an EAI, SOA or BPM? Or all of them? Is it development environment?

The first chapter changes my traditional order about explaining what BizTalk Server is. It starts explaining what problems does it solve first (EAI, B2B, BPM), and then goes forward explaining the BizTalk Server architecture. I think it’s a good idea starting about what can it do prior to what is it.

At this point, I’ll extract a bright point in the definition from architecture section:

[…] BizTalk Server at its core is an event-processing engine, based on a conventional publish-subscribe pattern […]

That’s the point. At its core, it’s all about publish and subscribe events. It cannot be stated clearer.

Cite: SOA, build or buy

dhtoran — Fri, 22 May 2009 15:25:22 GMT

Repeat with me: SOA is something you do, not something you buy

- David Linthicum

This is a nice cite I’ve read at the preface of the book I was talking about yesterday.

Yes, it seems a cite said by Captain Obvious, but it’s still amazing how many vendors try to sell their products as “SOA-out-of-the-box”.

It’s a pity. With so much hype, many customers have a fuzzy idea of what SOA is, and some of them are convinced to buy it.

But they can’t. Because SOA is something you do, not something you buy.

New book and free chapter: SOA Patterns with BizTalk Server 2009

dhtoran — Thu, 21 May 2009 18:17:40 GMT

I’ve just received for review from Packt Publishing the new book ‘SOA Patterns with BizTalk Server’ by Richard Seroter.

There’s also a free chapter from the book, ‘New SOA Capabilities in BizTalk Server 2009: WCF SQL Adapter’ downloadable as .pdf from the Packt book page.

Although I usually take this kind of books as reference guides more that a reading from start-to-end, I think I’m gonna read this one deeply and paying as much attention as I can, since I’m involved in a project that uses almost every single concept along the 12 chapters.

So the full review will arrive. Meanwhile, take a look at the worthy chapter index.

Also (this has nothing to do with the book content), I’m a nature photograph enthusiast, and the book has a nice macro photo in the portrait. Here it is:

Connecting systems – From Object Oriented to ESB for dummies

dhtoran — Thu, 21 May 2009 14:41:21 GMT

I’m working with a customer in a BizTalk Server 2009 + ESB solution for eAdmin. It’s a really nice project and I’m doing lot of different tasks here.

One of these tasks involve to design and explain the whole solution to many different people at the customers’. I’ve found that some dudes that are new to service oriented architectures find it difficult to understand the concepts behind ESB. Well, maybe it’s me that I don’t explain well…

Anyways, I’m doing a group of diagrams to explain the different ways and the evolution to connect different applications. Starting from object oriented and ending with an ESB. I know that OO does not map to “connect different applications”, but it serves as the starting point, the beginning of all.

Here are the Visio diagrams. They need a deep explanation, I know, but I’m posting here just the pictures (click to enlarge)

1. The basic OO world with two applications, the client and the “server” (as it serves the logic):

2. The distributed object oriented, that is just OO + web services. I’m really fascinated as nowadays still a lot of companies still sell this as “service oriented”:

3. Service oriented. Adding the concept of contract and service. Still have dependency on the server, aka the server must implement a service.

4. Service Oriented with a broker. It’s an EAI broker because I’m assuming a scenario where the server app cannot expose it’s logic as a service.

5. Enterprise Servive Bus. Extends the broker model and to a fussy model. There’s not a lineal / hardwired process to communicate the client with the server, but a set of rules and capabilities that have the intelligence to dynamically perform the communication flow. Ideally, we completely forget about the EAI stuff and we only speak services (so server app exposes a service again).

I’m playing with the idea of making a video, where the model starts at 1. and goes evolving to 5. with animations, and text and explanations fade in and out. I’ll finish this if time permits… I’m doing it with PowerPoint, which is not definitely the “uber animation creation tool”, but it’s extremely easy work with :-)

New BizTalk HotRod Magazine – Issue 6 Q2 2009

dhtoran — Wed, 22 Apr 2009 10:12:16 GMT

There’s a new issue published of the nice BizTalk HotRod Magazine. It contains some very nice articles written by clever people. And then there’s an article written by me :-)

I’m gonna copy/paste the index from Thiago’s blog, Connected Thoughts:

Writing Great BizTalk Applications by Rajinder Singh

BizTalk Rule Engine, a practical application by Alexander Krotov

Development Challenges with XML over AS2 by Suresh Kumar R & Sijo Jose T

Batching Outbound Messages: Management of Multiple Batch Criteria per Party by Bhola Meena

Hierarchical naming convention for BizTalk Messaging artifacts by David Hurtado Toran

To Be, Or Logical Not To Be: If-Then-Else in BizTalk Maps by Chris Romp

Enhancing the BizTalk Mapper: How to Wrap XSLT in Custom Functoids by Richard T. Broida

Muenchian Grouping and Sorting in XSLT by Chris Romp

Eliminate BizTalk Admin Problems with Terminator by Larry Meadows

Static Code Analysis for BizTalk Using BizTalkCop – Avoiding Common Pitfalls by Benny Mathew

you can get the issue in the BizTalk HotRod page. Here’s the direct link to the pdf.

Workflow still matters

dhtoran — Tue, 26 Feb 2008 15:07:00 GMT

As we said yesterday, workflow still matters, and I'm catching this stuff again in the Realworld(tm), trying to clear concepts and how-tos in these (not so) new componentes of .NET. Workflow Foundation is one of these, and I'll try to talk also about WCF, WPF/Silverlight, etc. from here and beyond.

So I'll start with one simple but interesting question. Let's assume you need to develop some business logic in .NET...

...is there any reason for not using Workflow Foundation?

Note that I'm not asking about doing all-or-nothing using WF. It's just a matter of searching for a reason for not using it as a part of the solution.

I'll try to look for the anwser for myself but, anyways, comments/discussions are welcome!

Workflow matters

dhtoran — Thu, 21 Jul 2005 10:51:00 GMT

David Chappell blogs about it. From my point of view, technologies such BTS Orchestration Engine are very interesting since they allows to:

Separate control flow from code that perform actions.
Draw control flow, instead of write it. It's more important for things not easy to write in a conventional programming language (ie: parallel execution, long running transactions with state management, etc)

BizUnit 2.0 released

dhtoran — Mon, 18 Jul 2005 09:24:00 GMT

Finally, BizUnit 2.0 is here, one of my favorite BizTalk projects must-haves. These guys work hard, in the very right direction...

I case you're not aware of what BizUnit is, it is a test framework fo BizTalk. Read their blog for more info.

Easy Promotion of Context Properties

dhtoran — Thu, 07 Jul 2005 11:43:00 GMT

Abstract: I’ve been dealing with context properties, and I’ve discovered a couple of very useful tricks for promoting and routing.

some definitions:
As you’ll probably know, there are two types of properties in the context of a message: properties based on field values (MessageDataPropertyBase) and properties not based on fields (MessageContextPropertyBased). I’ll call the first ones Message Properties and the other ones Context Properties.

Message Properties based on message fields are automatically written and promoted into message context by BizTalk.
Context Properties not based on message fields can be assigned inside an orchestration using the expression Message(Property) = “value”;
One of the most interesting stuff about Context Properties is that are not tied to a concrete schema, so they can be used regardless the message type. A sample of a system context property is FILE.ReceivedFilename.

the problem of promoting context properties:
When you create a context property, not based on a message field, and you assign a value inside an orchestration, it is written, but not promoted. What does it means? it means that you cannot route the message based on this property.
Some days ago I had the situation where needed exactly this: content based routing based on a custom context property that had a value calculated inside an orchestration.

A common solution is to create a custom pipeline component that writes and promotes properties. Jon Flanders has created a good generic component to do this: the ContextAdder Pipeline Component.

the trick:
In my case, the pipeline solution is not an option, since I’m routing between orchestrations via Direct Port Binding. Also, coding a custom component to promote a property seems to complex for me…
Somebody told me a good trick to promote context properties inside an orchestration, easy and direct:
Create a CorrelationSet based on the property.
Even if you are not going to use it, when you initialize a CorrelationSet, the Orchestration engine makes the promotion of the properties involved, since correlation is just an special kind of routing.

So now I have some dummy CorrelationSets, that I call <Property>Promote_CorrelationSet

nice and easy! :-)

Fun with BAM and orchestration execution time

dhtoran — Tue, 28 Jun 2005 09:42:00 GMT

Abstract: I’ve using BAM to measure execution time in some orchestrations. I cannot use HAT because I needed to do some calculations to rest waiting time of the external processes. It’s extremely easy. Things get funny when we needed to get more that one row per orchestration instance (because of loops which are there…)

Today I’ve doing this BAM at a customer’s. I’m going back home in a high velocity train, but it’s a long trip, so I think this post is going to get long… I’ll divide it in 2 parts:

Part 1- The easy sample
The scenario is quite simple: some orchestrations provide a synchronous web service façade for a legacy system. The logic for integration with lecagy includes files interchange, sockets comminications and some transformations from/to Xml and HL7 flat files.
What we want: stress the solution and get detailed information about orchestration execution time. This execution measured time should not include web service communications or file interchange with the legacy system, because the intention is to measure BizTalk performance, regardless of network bandwidth or external dependencies.

Here is the sample:

In the simplified sample (actual processes are a little more complex :-), we must measure time between start (Receive 1) and Send 1, and time between Receive 2 and last send back (Send 2). Time between Send 1 and Receive 2 must be taken away.

All this information is in tracking database, and can be queried at a basic level with HAT. Probably it’s not very complex to query the tracking database to get these values. Since each shape has a start and end time, it’s just a question of making the right queries.
Anyways, as I have only one working day to prepare the environment for the tests, I don’t have time to study the tracking DB internals. I’ve done the easy way: BAM milestones. It’s easy because there’s no need to know about BizTalk internals (tracking db design) and flexible because BAM definitions can be designed and deployed independently to the orchestration development.

So, I’ll show how to do it in just 10 minutes (for the above sample):

1- Create BAM definition with the BAM Structures Mega Designer (aka the little Excel sheet :-). These definitions include 4 Activity Items, all of them as milestones. I’ll not use duration view items to catch intervals, instead I’ll do a Transact/SQL DATEDIFF later.
Time estimated: 3 minutes

2- Export the BAM definition to an Xml file, and deploy it in the server with BAM Management Utility (bm.exe).
Use the Tracking Profile Editor to assign each activity item to orchestration shapes.
Don’t forget to assign an Activity Id, tipically an Id field takes from the message schema.
Deploy the profiles to the server.
Time estimated: 3 minutes

3- Only have 4 minutes left, so I’ll assume there is some test automation :-).
Launch the tests (with NUnit in my case).
Open SQL Server Query Analyzer, connect to the DB server, switch to the BAMPrimaryImport database and open the corresponding BAM view. There you can see the milestones catched.
Now you can calculate execution times with basic date adds and rests. It you want to do it with SQL DATEDIFFs, execute something like this:
SELECT [fields],
DATEDIFF(ms, [Milestone1], [Milestone2]) As Segment1,
DATEDIFF(ms, [Milestone3], [Milestone4]) As Segment2,
(Segment1+Segment2) As TotalTime
Time estimated: 5 minutes (ouch, 1 minute late!)

well, actually the last part is not true… you cannot use Segment1 and Segment2 aliases in the last line, so you’ll need to repeat the DATEDIFF statements. I've included in this way to improve readability.
Also, another nice way to do it is query the standard views with Excel and play with columns and formulas.

Now we are ready to launch stress tests and query orchestration execution times easily. The stress tests will be launched via Application Center Test, so we can control concurrent users, test timing, etc.

Part 2- I have a Loop and BAM Activity items cannot be assigned
The second orchestration has been more tricky. It’s a 1-to-many façade, so it has a loop to send many messages to the legacy systems. Something like:

The problem: Activity items cannot be assigned to shapes inside the loop. Why? because each activity item is a row, and the Activity Id is taken from a message field that is unique for the orchestration.
The solution: use the API to add new BAM rows for each loop iteration. The Activity Id of each child-row will be the parent Activity Id plus the counter of the loop, so they can be related easily in a query.
The BAM API is very straightforward. One object, three methods: The EventStream object is used to create a new activity (BeginActivity method), update rows (UpdateActivity) of close the activity (EndActivity).
You can use a DirectEventStream object that writes the data in the BAMPrimaryImport database directly. This can impact the performance because it’s synchronous.
You can either use the BufferedEventStream object that use a asynchronous store-and-forward. It does not guarantee that the event will be inserted in BAM immediately, but it does not affect caller performance.
Here is the tricky thing: There is a property called ConnectionString that points to the database. If you use the DirectEventStream, ConnectionString points to BAMPrimaryImport database. But if you use BufferedEventStream, its ConnectionString points to the MessageBox database, because it’s used for store-and-forward.

My code between the orchestration and a helper component does more or less the following:
For each loop iteration:
- Begin an new Activity, using [ActivityId].[LoopCounter]
- Update with a milestone before sending to legacy
- Update with a milestone after receiving from legacy
- Close the Activity

So, for each orchestration instance, I have in the BAM database:
- One row with
    - Orchestration Activity Id
    - Milestones for BeginOrch and EndOrch
- Many “child” rows (the loop) with
    - Orchestration Activity Id + Counter
    - Milestones for BeginSend and EndReceive

The query to aggregate everything (courtesy of eXtreme.net, because I don’t have quick Transact/SQL skills), return a single row with the calculated fields for TotalTime and TotalTimeWithoutLegacyStuff. It uses DATEDIFF for the differences, and SUBSTRING to relate different parent and child Activity IDs

Calling A Receive Pipeline Inside an Orchestration in BizTalk 2006

dhtoran — Fri, 17 Jun 2005 02:35:00 GMT

Impresive! from Steve W Thomas:

For starter, why would you want to call a Receive Pipeline from within an Orchestration? I had to struggle for a bit to come up with a good reason… I can find it useful in debatching Scenarios that require mapping prior to debatching or for debatching into smaller batches using a map. I could also find it useful when working with flat file.

[...]

Super Cool: Supports receiving multiple messages returned from the pipeline and can use enumeration to process each message.