David Hurtado's Integration Traces

Workflow still matters

As we said yesterday, workflow still matters, and I'm catching this stuff again in the Realworld(tm), trying to clear concepts and how-tos in these (not so) new componentes of .NET. Workflow Foundation is one of these, and I'll try to talk also about WCF, WPF/Silverlight, etc. from here and beyond.

So I'll start with one simple but interesting question. Let's assume you need to develop some business logic in .NET...

...is there any reason for not using Workflow Foundation?

Note that I'm not asking about doing all-or-nothing using WF. It's just a matter of searching for a reason for not using it as a part of the solution.

I'll try to look for the anwser for myself but, anyways, comments/discussions are welcome!

Workflow matters

David Chappell blogs about it. From my point of view, technologies such BTS Orchestration Engine are very interesting since they allows to:

• Separate control flow from code that perform actions.
• Draw control flow, instead of write it. It's more important for things not easy to write in a conventional programming language (ie: parallel execution, long running transactions with state management, etc)

BizUnit 2.0 released

Finally, BizUnit 2.0 is here, one of my favorite BizTalk projects must-haves. These guys work hard, in the very right direction...

I case you're not aware of what BizUnit is, it is a test framework fo BizTalk. Read their blog for more info.

Easy Promotion of Context Properties

Abstract: I’ve been dealing with context properties, and I’ve discovered a couple of very useful tricks for promoting and routing.

some definitions:
As you’ll probably know, there are two types of properties in the context of a message: properties based on field values (MessageDataPropertyBase) and properties not based on fields (MessageContextPropertyBased). I’ll call the first ones Message Properties and the other ones Context Properties.

Message Properties based on message fields are automatically written and promoted into message context by BizTalk.
Context Properties not based on message fields can be assigned inside an orchestration using the expression Message(Property) = “value”;
One of the most interesting stuff about Context Properties is that are not tied to a concrete schema, so they can be used regardless the message type. A sample of a system context property is FILE.ReceivedFilename.

the problem of promoting context properties:
When you create a context property, not based on a message field, and you assign a value inside an orchestration, it is written, but not promoted. What does it means? it means that you cannot route the message based on this property.
Some days ago I had the situation where needed exactly this: content based routing based on a custom context property that had a value calculated inside an orchestration.

A common solution is to create a custom pipeline component that writes and promotes properties. Jon Flanders has created a good generic component to do this: the ContextAdder Pipeline Component.

the trick:
In my case, the pipeline solution is not an option, since I’m routing between orchestrations via Direct Port Binding. Also, coding a custom component to promote a property seems to complex for me…
Somebody told me a good trick to promote context properties inside an orchestration, easy and direct:
Create a CorrelationSet based on the property.
Even if you are not going to use it, when you initialize a CorrelationSet, the Orchestration engine makes the promotion of the properties involved, since correlation is just an special kind of routing.

So now I have some dummy CorrelationSets, that I call <Property>Promote_CorrelationSet

nice and easy! :-)

Fun with BAM and orchestration execution time

Abstract: I’ve using BAM to measure execution time in some orchestrations. I cannot use HAT because I needed to do some calculations to rest waiting time of the external processes. It’s extremely easy. Things get funny when we needed to get more that one row per orchestration instance (because of loops which are there…)

Today I’ve doing this BAM at a customer’s. I’m going back home in a high velocity train, but it’s a long trip, so I think this post is going to get long… I’ll divide it in 2 parts:

Part 1- The easy sample
The scenario is quite simple: some orchestrations provide a synchronous web service façade for a legacy system. The logic for integration with lecagy includes files interchange, sockets comminications and some transformations from/to Xml and HL7 flat files.
What we want: stress the solution and get detailed information about orchestration execution time. This execution measured time should not include web service communications or file interchange with the legacy system, because the intention is to measure BizTalk performance, regardless of network bandwidth or external dependencies.

Here is the sample:

 

In the simplified sample (actual processes are a little more complex :-), we must measure time between start (Receive 1) and Send 1, and time between Receive 2 and last send back (Send 2). Time between Send 1 and Receive 2 must be taken away.

All this information is in tracking database, and can be queried at a basic level with HAT. Probably it’s not very complex to query the tracking database to get these values. Since each shape has a start and end time, it’s just a question of making the right queries.
Anyways, as I have only one working day to prepare the environment for the tests, I don’t have time to study the tracking DB internals. I’ve done the easy way: BAM milestones. It’s easy because there’s no need to know about BizTalk internals (tracking db design) and flexible because BAM definitions can be designed and deployed independently to the orchestration development.

So, I’ll show how to do it in just 10 minutes (for the above sample):

1- Create BAM definition with the BAM Structures Mega Designer (aka the little Excel sheet :-). These definitions include 4 Activity Items, all of them as milestones. I’ll not use duration view items to catch intervals, instead I’ll do a Transact/SQL DATEDIFF later.
Time estimated: 3 minutes

2- Export the BAM definition to an Xml file, and deploy it in the server with BAM Management Utility (bm.exe).
Use the Tracking Profile Editor to assign each activity item to orchestration shapes.
Don’t forget to assign an Activity Id, tipically an Id field takes from the message schema.
Deploy the profiles to the server.
Time estimated: 3 minutes

3- Only have 4 minutes left, so I’ll assume there is some test automation :-).
Launch the tests (with NUnit in my case).
Open SQL Server Query Analyzer, connect to the DB server, switch to the BAMPrimaryImport database and open the corresponding BAM view. There you can see the milestones catched.
Now you can calculate execution times with basic date adds and rests. It you want to do it with SQL DATEDIFFs, execute something like this:
SELECT [fields],
DATEDIFF(ms, [Milestone1], [Milestone2]) As Segment1,
DATEDIFF(ms, [Milestone3], [Milestone4]) As Segment2,
(Segment1+Segment2) As TotalTime
Time estimated: 5 minutes (ouch, 1 minute late!)

well, actually the last part is not true… you cannot use Segment1 and Segment2 aliases in the last line, so you’ll need to repeat the DATEDIFF statements. I've included in this way to improve readability.
Also, another nice way to do it is query the standard views with Excel and play with columns and formulas.

Now we are ready to launch stress tests and query orchestration execution times easily. The stress tests will be launched via Application Center Test, so we can control concurrent users, test timing, etc.

Part 2- I have a Loop and BAM Activity items cannot be assigned
The second orchestration has been more tricky. It’s a 1-to-many façade, so it has a loop to send many messages to the legacy systems. Something like:

 

The problem: Activity items cannot be assigned to shapes inside the loop. Why? because each activity item is a row, and the Activity Id is taken from a message field that is unique for the orchestration.
The solution: use the API to add new BAM rows for each loop iteration. The Activity Id of each child-row will be the parent Activity Id plus the counter of the loop, so they can be related easily in a query.
The BAM API is very straightforward. One object, three methods: The EventStream object is used to create a new activity (BeginActivity method), update rows (UpdateActivity) of close the activity (EndActivity).
You can use a DirectEventStream object that writes the data in the BAMPrimaryImport database directly. This can impact the performance because it’s synchronous.
You can either use the BufferedEventStream object that use a asynchronous store-and-forward. It does not guarantee that the event will be inserted in BAM immediately, but it does not affect caller performance.
Here is the tricky thing: There is a property called ConnectionString that points to the database. If you use the DirectEventStream, ConnectionString points to BAMPrimaryImport database. But if you use BufferedEventStream, its ConnectionString points to the MessageBox database, because it’s used for store-and-forward.

My code between the orchestration and a helper component does more or less the following:
For each loop iteration:
- Begin an new Activity, using [ActivityId].[LoopCounter]
- Update with a milestone before sending to legacy
- Update with a milestone after receiving from legacy
- Close the Activity

So, for each orchestration instance, I have in the BAM database:
- One row with
    - Orchestration Activity Id
    - Milestones for BeginOrch and EndOrch
- Many “child” rows (the loop) with
    - Orchestration Activity Id + Counter
    - Milestones for BeginSend and EndReceive

The query to aggregate everything (courtesy of eXtreme.net, because I don’t have quick Transact/SQL skills), return a single row with the calculated fields for TotalTime and TotalTimeWithoutLegacyStuff. It uses DATEDIFF for the differences, and SUBSTRING to relate different parent and child Activity IDs

 

Calling A Receive Pipeline Inside an Orchestration in BizTalk 2006

Impresive! from Steve W Thomas:

For starter, why would you want to call a Receive Pipeline from within an Orchestration?  I had to struggle for a bit to come up with a good reason… I can find it useful in debatching Scenarios that require mapping prior to debatching or for debatching into smaller batches using a map.  I could also find it useful when working with flat file.

[...]

Super Cool: Supports receiving multiple messages returned from the pipeline and can use enumeration to process each message.

more on his blog

Why (good) Xml is much better than plain text

There are many reasons, sure, and probably there are also reasons why plain text files can be better, but I would like to remark just only one reason, just because I fighting with it right now:

Xml is human readable

Or at least, it should be.

I’m dealing with the HL7 standard for healthcare. HL7 files are text files with some strange delimiters such ^ and |. Luckily we can use the BizTalk HL7 Accelerator, that allow us to abstract from the HL7 details.

A sample of an HL7 file:

MSH|^~\&|REG|MCM|BTS||199601121005||ADT^A04|000001|P|2.2
EVN|A04|199601121005||01||199601121000
PID|||191919^^^MYHOS^MR~123-45-6789^^^USSSA^SS|253763|SMITH^JOHN^Q||19560129|M|||123MAIN^^BUFFALO^NY^98052^""||(123)555-0100||S|M|10199925^^^MYHOS^AN|123-45-6789
PD1|S|F|NormalString^A^+1^-1^ISO^simpletext&Test&HCD^GI^simpletext&NormalString&ISO^I|NormalString^Test&Test^Test^Test

^Test^Test^AE^simpletext^simpletext&Test&ISO
^P^NormalString^M10^MC^simpletext&NormalString&HCD^A|N|simpletext|I|I|N|NormalString^+1^M11^

simpletext&NormalString&L,M,N^RRI^simpletext&
NormalString&HCD|NOVALUE^NormalString^Test^Test^NormalString^Test|N
PV1|1|I|2000^2012^01^hey&test&DNS^test^test^test^test^test||||004777^MILLER^CONNIE^A.|||SUR||||2|A0

Where is the Patient Name? is “the substring between the fifth and the sixth | (pipe), in the third line (the line starting with PID). And remember, spaces are represented as ^(strange little hat)”

The HL7 Accelerator comes with Xsd schemas to map these flat files. A sample message type ADT A04 (the above) looks something like this (just a small piece):

<ns0:ADT_A04_22_GLO_DEF xmlns:ns0="http://microsoft.com/HealthCare/HL7/2X">
 <EVN_EventType>
  <EVN.1_EventTypeCode>A04</EVN.1_EventTypeCode>
  <EVN.2_DateTimeOfEvent>199601121005</EVN.2_DateTimeOfEvent>
  <EVN.3_DateTimePlannedEvent>199601121000</EVN.3_DateTimePlannedEvent>
  <EVN.4_EventReasonCode>01</EVN.4_EventReasonCode>
 </EVN_EventType>
 <PID_PatientIdentification>
  <PID.1_SetIdPatientId>191919</PID.1_SetIdPatientId>
  <PID.2_PatientIdExternalId>
    <PID.5_PatientName>
       <PN.0_FamiliyName>Doe</PN.0_FamiliyName>
       <PN.1_GivenName>John</PN.1_GivenName>
    </PID.5_PatientName>
[…]

we still deal with HL7 codes and semantic structure, but it’s much easier to work the Patient Name. It's located in “the FamilyName element under PatientIdentification” :-)

Fun with ConfigFramework (updated)

I've been playing around with a funny error message in ConfigFramework. It says "The Operation Completed Successfully", but it's a Retry/Cancel message box, where you get looped until you press cancel... The details found in the log file talk about failiure to modify a COM+ Application. It also happened to a colleage, as he has posted more details in his eXtreme.NET blog.

I solved it changing the identity of MSDTC to the Network Service system identity and restarting MSDTC (I also relaxed the security allowing remote administration of MSDTC, but I'm not sure if it helped). Anyways, I think you can change these settings back to the old values after running ConfigFramework, since the error is about creating and configuring the COM+ App, not about running it.

update: this "The operation completed successfully" translates to "Connection with the transaction manager was lost". So it's in deed a problem of servers communicating via MSDTC. My concrete case is due to use local accounts for MSDTC identity in a workgroup environment, so the servers cannot authenticate each other. This case is documented in Bug: MSDTC Fails to Mutually Authenticate

Helpful articles: in case you want more documentation:

New DTC Functionality in Windows Server 2003 SP1

FIX: The SqlConnection.Open function generates a ComException error message if the Distributed Transaction Coordinator (DTC) service is restarted or failed over

Turning Off Remote Procedure Call Security on Windows 2003 Server

 

Here's the pic, just for curious ones:

Compiled BizTalk 2004 Help File (.chm), indexed with MSN Desktop Search

Marty Wasznicky has posted a compiled BizTalk 2004 Help File (.chm) check it out in his blog (11 Mb)

The most interesting stuff is that it can be indexed using MSN Desktop Search (as any other .chm file).... that means querying BizTalk Documentation from the Windows Taskbar. Cool, since I've tried to index the Visual Studio Combined Help files without success :-(

New whitepaper about BizTalk Messaging Internals

There is a new whitepaper on MSDN about messaging. It's called BizTalk Server 2004: A Messaging Engine Overview,  but it's quite a deep dive into messaging. From my point of view, the most interesting stuff is how it explains the Pub/Sub internals in terms of MessageBox flow and stored procs involved in publishing and subscribing.

BizTalk messages based on .NET types instead of Xsd schemas

Abstract: You can create a BizTalk message from a custom .NET class, instead of using an Xsd schema. This practice has some pros and cons. Let’s see…

Usually, the most common process is to start creating Xsd schemas, and use them in Orchestrations, to define BizTalk messages. But, why would you like to have BizTalk messages based on Xml? It seems obvious that all messages should be Xml based, but in fact, sometimes Xsd/Xml is not necessary and adds complexity.

When is Xsd/Xml needed?
Xml messages are very nice if you need Messaging. That is, interoperatibility, external communications, schema publication. But surprisingly, I find many situations where people are defining Xsd schemas for Xml messages where Xml is not needed at all. Well, Xml is still fashionable, so…
If a message lives inside an orchestration (is being passed from within orchestrations), but it’s never going to be published externally, or is never going to be sent/received externally, there’s no advantage in using Xml. In fact, there’s no reason to use Xml.

Xsd hell: how do you create a message from the scratch?
It’s a well known issue that creating an empty instance of a message inside an orchestration is not easy. Having the xml string hardcoded is not very elegant. Pointing to a file is not very elegant at all. You would need the path harcoded. Well, you can put it in a .config file, but… isn’t too complex to just create an empty message?
I’ve read some discussion about using Maps to create empty (or default valued) instances. Since I agree it’s a good idea, it’s still too complex to just create an instance of a message.
This message-from-the-scratch problem is inherent to the use of Xml, since a message is not instantiable.

Using a .NET class
You can use a .NET class, instead a Xsd schema. Define your orchestration message as a .NET type, using your class. Use references to XLANGs Base Types to promote your class properties to distinguised fields or properties.
Pros:
Instantiate it. Use constructors, destructors, static members for instance creation or whatever you want.
Use rich properties. Mark public properties as promoted or distinguished fields. Use get and set methods.
Reuse as objects in other projects.
Cons:
External publication and Interoperatibility. There is no Schema, so if you intend to publish it externally, what do you publish?
Dependency of XLANG assemblies, so limited reuse outside your BizTalk project.

Simple Sample:
Create a .NET class:
using System;
namespace STCEAI.Messages
{
 [Serializable]
 public class SimpleMessage
 {
  private string _id;

  public SimpleMessage()
  {
  }

  [Microsoft.XLANGs.BaseTypes.DistinguishedField]
  public string Id
  {
   get{ return _id;}
   set{ _id = value;}
  }

  public static SimpleMessage Create()
  {
   SimpleMessage msg = new SimpleMessage();
   msg.Id = System.Guid.NewGuid().ToString();
   return msg;
  }
 }
}

Note the property is marked as Distinguished Field in order to make it visible from within the orchestration. This attribute is in Microsoft.BizTalk.XLANGs.BaseTypes.dll. You can also mark it as a promoted property, assign a Namespace, and include all the Xml serialization attributes as needed.

Usage inside an orchestration, in a Message Assignment Shape :
SimpleMessage_msg = new STCEAI.Messages.SimpleMessage();
SimpleMessage_msg.Id = myId;

or better:
SimpleMessage_msg = new STCEAI.Messages.SimpleMessage.Create();

What Videogame character are you?

Calculated shot... well, I've played Scorched Earth a lot in my life... :-)

I am a Scorched Earth Tank. When I have a mission, it consumes me; I will not be satisfied until the job is done. I have a strong sense of duty, and a strong sense of direction. Changes in the tide don't phase me - I always know which way the wind blows, and I know how to compensate for it. I get on poorly with people like myself. What Video Game Character Are You?

Real asynchronous vs. Simulated asynchronous

Abstract: In the asynchronous world, we can talk about Real async and Simulated async. Each one has its own pros and cons. Let’s see a simplified sample of each case.

Sample scenario, let’s assume two systems, A and B:
   1.-   A sends a message to B.
   2.-   B processes the request from A.
   3.-   B returns a response to A.

Constraint: The processing of the request (step 2) takes some unpredictable time, so we cannot afford A to have an open connection waiting for the response from B at step 3. We need an asynchronous model, but it can be a real async or simulated async.

Real Async
In a Real async scenario all the communications are one-way, fire-and-forget. A sends the request to B and closes the connection. Once the processing is finished, B starts a new connection with A and sends a new message, the response.

The characteristics are:

• Each message goes in a true one-way communication.
• Both client and server must implement listeners --> from the communications point of view, both are client and servers.
• Both A and B must be aware of the other system's endpoint.
• Bandwidth and CPU are optimized, and there are no blocking points.

here is a sample picture. Arrows shows who start the communications:

Probably, the most important issue is the second point: both A and B are clients and servers (or consumers and providers). Both systems must implement a listener or message sink. This is not suitable in many cases when A is a client app. So we can go for a simulated async.

 

Simulated Async
In the simulated approach, almost all the communications are still one-way, but B never starts a new connection; instead, B just puts the response available for A. It’s A responsibility to get the response, doing some polling.

The characteristics are:

• All the communications are started by A. B does not deal with communications issues. 
• B is not even aware of A. If there are many As, B does not need to know.
• Easy to implement. Or at least, easier than Real Async.
• Bandwidth overload, because of the polling, as well some CPU consumption on A.

Here is the sample picture, modified showing simulated async via polling. Arrows shows who start the communications:

In this case, the response is not sent (PUT); it’s retrieved (GET), so it’s not a one-way communication.

These two models can extend to the infinite with many variations, but I think here are the two most basic ones.

 

Question: Secure the access or secure the content?

Abstract: this seems an easy question, but I have not found a proper answer yet… feel free to give your answer and a little justification.

The complete question is:
What should I secure? the access to the resources, or the content of the resources?

Securing the access means controlling who sees the resources, in terms of who can read files, databases, etc.
Securing the content means encrypting the file content or the data inside databases, so everybody can read but only a few can understand.

Meanwhile you think a response,
- There is an application, let’s say, and ASP.NET application.
- Then there is IIS authentication, let’s say, Windows Integrated Authentication.
- Then there is .NET Framework Code Access Security settings, let’s say, system administrators configuring execution permissions to the assembly.
- Then there is more Code Access Security (developer’s), so the .NET assembly asks declaratively for a read permission to a resource file.
- Then there is the resource file, encrypted, of course, to store a Connection String.
- Then there is a connection to a database engine, with user/password challenge for authentication.
- Then there is database authorization, give access to a concrete database object.
- Then there is a query, that returns an encrypted column (lovely Yukon).
- Then there is a database-level encryption user key, using a password or passphrase provided by the user, so the executing assembly can read the column data.
- So the column data goes in clear to the assembly, which returns the information to the IIS to be returned to the human-being at the other side of the network. Of course IIS uses a HTTPS connection, beware of hackers…

back to que question... Encrypt files or protect them from being read? or both? most important: why?

Orchestration Error handling: Fix-and-Log

Abstract: An exception has been thrown, what should I do? --> At least two actions: Fix the situation and Log it

Orchestrations tend to fail. Sure. This is because the nature of integration: Orchestrations deal with external applications, and expect behaviors that are not always as expected.
Exception handlers should be everywhere where errors can happen. The rule of thumb is that “Suspended/Non Resumable state should never occur”.
Each exception handler should perform at least two actions: fix the situation and log it.

Fix the situation
Fixing means not allowing the orchestration fail into a suspended state, and notify all the caller process. This is usually done by creating an error response message, or valid response that contains no data, but some detail on the error, so the flow can continue in some way.
Note that using the same response message or a different and specialized fault message usually depends on the message schemas and other requirements.

Log
A common practice when handling with errors is to log full message or context data (or both!) using the Windows Event Log or any other log framework. Typically, developers log data about the exception, the state of the orchestration, the message that raised the error, etc.
With this practice is very easy to duplicate a lot of data, because BizTalk already stores all this data for you. This information can be found in the tracking database, using HAT. The problem with HAT is that is difficult to find useful info if you are not sure what you are looking for.
So what kind of information should you log in case of error? -> Enough info to:

1. know what kind of error happen.
2. know where it has happen
3. be able to find the details using HAT

Nothing more. This is the tip: don't log data that is logged by BizTalk. If there is an exception, BizTalk has the details. If message tracking is enabled, do not log message contents. Learn to use HAT efficiently, and you’ll save time and logging code.
Here is an easy sample of parallel Fix-and-Log: