Keith Lim's Biztalk BAM Blog

BAM continuation between WF/WCF application and BizTalk application

Awhile back a user asked me this question on BAM continuation from a WF/WCF application to a BizTalk application:

I have a process which starts a workflow and then an orchestration. I want to create an activity cross the workflow and the orchestration. So I create a ContinuationToken in the WF IC and create a ContinuationID in the Orchestration TPE. However, it does not seem work. The data from WF and the data from Orch are written into two records in the Completed table.

My questions are:

Is it possible for continuation to happen between WF/WCF and Orchestration? What specific steps I should be aware in order for that to happen? Any sample?

Here is my answer:  if you have a WCF data that is also available (flowed to) Biztalk Orchestration, you can use it to form a continuation token.   The important piece of information here is that you must prefix it with a constant and the constant must be the same name as the ContinuationID folder.

Here is a snippet of IC for a WCF application, in this continuation token node. Here the TokenID will be something like WCF2Orch1PO123 if POID contains PO123.

      <ic:ContinuationToken>

        <ic:Expression>

          <ic:Operation Name="Constant">

            <ic:Argument>WCF2Orch1</ic:Argument>

          </ic:Operation>

          <wcf:Operation Name="XPath">

            <wcf:Argument>//s:Body//po:POID</wcf:Argument>

          </wcf:Operation>

          <ic:Operation Name="Concatenate" />

        </ic:Expression>

      </ic:ContinuationToken>

In the TPE, you would then need to create a ContinuationID folder and rename it to “WCF2Orch1” and map the value under it to a piece of data that contains the flowed value of PO123.

*  When you deploy the tracking profile, you will get a warning that the tracking profile is missing the Continuation folder, but since we know what we are doing :), just ignore and proceed.

Steps to facilitate the creation of WCF Interceptor Configuration file.

The BAM Interceptor for WCF does not come with a UI tool to create the Interceptor Configuration file, and from the Beta II customer feedback thus far, this is the main pain point as well.   I will try to provide some steps and guidelines to help ease this concern. 

A common scenario is when the WCF application configuration file is updated and the IC is created and deployed, there is not data intercepted and persisted into the BAM PITDB (Primary Import Tables Database).   When the event log is checked out without error and that the bm.exe deploy-interceptor is successful, most likely the culprit is in the IC – somewhere in the IC, the XPath is incorrectly set.

Here is what I would do:

Enable WCF logging/tracing to produce the wcf log.  In the config file of the “interceptee” application (either or both client and service), splice the following highlighted code.

<configuration>

  <system.serviceModel>

    <diagnostics>

      <messageLogging maxMessagesToLog="30000"

              logEntireMessage="true"

              logMessagesAtServiceLevel="true"

              logMalformedMessages="true"

              logMessagesAtTransportLevel="true">

      </messageLogging>

    </diagnostics>

  </system.serviceModel>

  <system.diagnostics>

    <sources>

      <source name="System.ServiceModel" switchValue="Verbose, ActivityTracing" >

        <listeners>

          <add name="xml" />

        </listeners>

      </source>

      <source name="System.ServiceModel.MessageLogging" switchValue="Verbose">

        <listeners>

          <add name="xml" />

        </listeners>

      </source>

      <source name="Microsoft BizTalk Bam Interceptors" switchValue="All" >

        <listeners>

          <add name="xml"/>

          <add name="myListener" type="System.Diagnostics.TextWriterTraceListener"

               initializeData="TextWriterOutput.log" />

        </listeners>

      </source>

    </sources>

    <sharedListeners>

      <add name="xml"

           type="System.Diagnostics.XmlWriterTraceListener"

           initializeData="Traces.e2e" />

    </sharedListeners>

    <trace autoflush="true" />

  </system.diagnostics>

</configuration>

Rerun the application.  This should generate a WCF trace file: *.e2e.

Use the SvcTraceViewer.exe tool (comes with WCF SDK installation) to view the wcf log and copy out the message into a .xml file.  

 Here in this snapshot of the tool, you should copy out the s:envelop node. 

Use and xpath tool (such as VisualXPath) to drill down to the specific area of the message to produce a valid XPath.  Copy and past this into the interceptor configuration file.

Redeploy the IC and rerun the application.

Hopefully the above steps would help surface the problematic XPaths in the IC and speed up your work.

BAM Interceptor for WCF

Many of you guys are eagerly anticipating the release of the Microsoft BizTalk Server 2006 R2… and the only definite answer to that is it will be available in Q3 of 2007.  In the meantime, the Beta 2 release of it can be downloaded from this link: http://connect.microsoft.com/

In this series of blogs I will again as usual, delve on the BAM side of BizTalk offering and answer any questions from you.  In R2, there are two new interceptors added, namely the BAM Interceptor for WCF (Windows Communication Foundation) and the BAM Interceptor for WF (Windows Workflow Foundation).   Of the two new interceptors, I will, for the time being, concentrate on the WCF Interceptor.  To find out more on WF Interceptor, please checkout my colleague, Andy Shen’s blog site: http://blogs.msdn.com/andyshen/

Before I get into the BAM Interceptor for WCF, here is a summary of some howto’s from my past entries to those who are new to Biztalk BAM:

• Introduction and means to BAM-enabled a business process.
• Writing BAM events that are transactionally consistent with BizTalk pipelines using MessagingEventStream.
• Writing BAM events that are transactionally consistent with BizTalk orchestration using OrchestrationEventStream.
• Continuation scenarios with mix of TPE-enabled and BAM  EventStream API tracking.
• Programmatically retrieving data from BAMPrimaryImport database using supportable API.
• Programmatically retrieving server and database info of Biztalk MessageBox and BAM PrimaryImport database.

So what is this BAM Interceptor for WCF?  In a nutshell, it extends the capability of BAM interception to any WCF application of it message without having to recompile the WCF application.  (This is analogous to using BAM Tracking Profile Editor to map and enable interception on BizTalk application without having to recompile the BizTalk application.)

The integration of WCF application and BAM interception is achieved through WCF configuration file by adding a custom behavior to the WCF application endpoint node.  A separate file called IC (Interceptor Configuration) is required to filter and map WCF data of interest to corresponding BAM activity. 

Where in WCF application are the data intercepted?  It supports the following interception points for WCF client, service and callback applications:

·         Before the method call (Request)

·         After the method call (Reply)

·         When the fault occurs in method call (Fault)

What can be intercepted?  Basically, anything you can see in the soap envelope can be tracked, but more often than not, you are interested in the data in the soap body where information of the method parameters is located.

Here are the basic steps to enable BAM interception for a WCF application:

1.      Create BAM activity and deploy.

2.      On a WCF application, update its config file.

3.      Create a corresponding IC file and deploy.

4.      Execute the application.

5.      View data in Portal.

Only step 2 are 3 are new to this release and you can see details information on this in the BizTalk Server Documentation under the heading (I will update this to online link when it is available):

Business Process Management | Using Business Activity Monitoring | Implementing BAM Solutions | Using the BAM WCF and WF Interceptors

Additional resource:

MSDN forum specific for Microsoft BAM: http://forums.microsoft.com/MSDN/ShowForum.aspx?ForumID=1473&SiteID=1

To be continued…

How to programatically retrieve data from BAMPrimaryImport using supportable API

Eventhough there are Portal UI and Excel that would allow you to query and access BAM activity data, it seems that there are some of you who would like to programmatically access/query this data from a custom application.

In BizTalk Server Documentation, currently in its 2006 pre-released version, you will find information on the BAMManagementService API.  (Under Technical Reference | Developers Reference | BizTalk Server 2006 .Net Class Reference | Microsoft.BizTalk.Bam.Webservices.Management ). 

The BamManagementService class is what we are interested. This is a webservice API and you can also see its methods by pointing to the box where Biztalk is installed. (replace localhost with the machine name in the link below). 

http://localhost/BAM/BamManagementService/BamManagementService.asmx

• AddSubscription
• CreateAlert
• DeleteAlert
• GenerateEventForArtifact
• GetAlert
• GetAlertsSummary
• GetCapabilities
• GetPivotViewXml
• GetReferences
• GetRelatedActivityInstances
• GetViewDetailsAsXml
• GetViewSummaryForCurrentUser
• RemoveSubscription
• UpdateAlert
• UpdateSubscription

For example, the above GetViewSummaryForCurrentUser would allow you to retrieve all views that the user has access.  And it would allow you to retrieve the activity names from there.  See code example in help doc for this.

Once you have retrieve the activity name, you can then connect to the database and call the views that below to the specific activity. 

Sample code from help doc (some of these are still been updated):

using WS = BamManagementService;

class Program
{
    static void Main(string[] args)
    {
        GetViewSummary();

        Console.WriteLine("Hit Enter when ready.");
        Console.ReadLine();
    }

    static void GetViewSummary()
    {
        // Create a web service proxy object.
        using (WS.BamManagementService managementService = new WS.BamManagementService())
        {
            // BAM Management Web Service requires client to be authenticated.
            // The client must be a member of the BAM Portal Users group.
            managementService.Credentials = System.Net.CredentialCache.DefaultCredentials;

            // Get the list of all BAM views available to the current user.
            WS.ViewSummaryList viewSummaryList = managementService.GetViewSummaryForCurrentUser();

            // Print the info for all views on the screen.
            foreach (WS.ViewSummary viewSummary in viewSummaryList.View)
            {
                Console.WriteLine("View: {0}", viewSummary.Name);
                // Each view is defined on one or more BAM activities.
                foreach (WS.ActivitySummary activitySummary in viewSummary.Activity)
                {
                    // The activity data can be on a different server/database if distributed
                    // BAM infrastructure is deployed.
                    Console.WriteLine("    Activity: {0} {1}",
                        activitySummary.Name,
                        activitySummary.IsRemote ? "[ Remote ]" : "");
                    Console.WriteLine("        Server: {0}", activitySummary.Server);
                    Console.WriteLine("        Database: {0}", activitySummary.Database);
                    Console.WriteLine("        BAM Portal URL: {0}", activitySummary.PortalUrl);

                    // OLAP and Real-time aggregations aggregations can be defined.
                    if (activitySummary.Aggregation != null)
                    {
                        Console.WriteLine("        Aggregations:");
                        foreach (WS.Aggregation aggregation in activitySummary.Aggregation)
                        {
                            Console.WriteLine("            Name: {0} *** Type: {1} *** Cube Name: {2}",
                                aggregation.Name,
                                aggregation.Type,
                                aggregation.CubeName);
                            if (aggregation.PivotView != null)
                            {
                                foreach (WS.PivotView pivotView in aggregation.PivotView)
                                {
                                    Console.WriteLine("                PivotView: {0}", pivotView.Name);
                                }
                            }
                        }
                    }
                }

                Console.WriteLine();
            }

            if (viewSummaryList.PartialResults)
            {
                // Distributed BAM infrastructure is deployed and one of the referenced databases
                // is not accesible.
                Console.WriteLine("One or more of the referenced databases are not available. " +
                    "The given list of views may not be complete.");
            }
        }
    }
} 

Sample Output:

View: SalesManagerView
    Activity: PurchaseOrder 
        Server: purchasesrv
        Database: BAMPrimaryImport
        BAM Portal URL: http://purchasesrv/BAM
        Aggregations:
            Name: POCube *** Type: OLAP *** Cube Name: POCube
            Name: POByLocation *** Type: RTA *** Cube Name: POCube
    Activity: Shipment [ Remote ]
        Server: shipsrv
        Database: BAMPrimaryImport
        BAM Portal URL: http://shipsrv/BAM
        Aggregations:
            Name: SHCube *** Type: OLAP *** Cube Name: SHCube
            Name: SHByLocation *** Type: RTA *** Cube Name: SHCube

View: SupplyManagerView
    Activity: SupplyOrder 
        Server: purchasesrv
        Database: BAMPrimaryImport
        BAM Portal URL: http://purchasesrv/BAM
        Aggregations:
            Name: SupplyCube *** Type: OLAP *** Cube Name: SupplyCube
                PivotView: SupplyCost
            Name: SupplyTimes *** Type: RTA *** Cube Name: SupplyCube
                PivotView: SupplyTimes

Hit Enter when ready.

Soliciting Biztalk BAM scenarios and issues

How great it would be if every features we put in to the software is what customers need/want.   Anyway, I am creating this entry to solicit your input on how you use or plan to use BizTalk BAM.   Please response as a comment with your scnenario.

Do also tell us what feature you like and dislike.

Thanks,
Keith

BAM API: Using OrchestrationEventStream

Since there is not much documentation* on this yet on how to use OrchestrationEventStream (an EventStream that is in synch with Orchestration), here is a quick info on the howto:

This new feature is found in Microsoft.BizTalk.Bam.XLANGs assembly.  From you BizTalk project, you can write the following in your orchestration, e.g. Expression Editor (click on the appropriate shape in orchestration to launch this):

// Sample code for using OrchestrationEventStream from an orchestration, no need to new it:
Microsoft.BizTalk.Bam.EventObservation.OrchestrationEventStream.BeginActivity( "PurchaseOrder", "PO123");

Note: this only applies to Biztalk 2006.

* To use this in Biztalk 2004 SP1, you will need a hot fix, its kb article can be found here: http://www.kbalertz.com/Feedback_900083.aspx,

Sample code to retrieve server and database info of Biztalk MessageBox and PIT

Since there is a request for this, here is an example to retrieve server name and database for Biztalk MessageBox and BAM Primary Import database.  Since this is accessing internal implementation, this code is not supportable (use at your own risk) and users will be required to maintain this code on their own when doing upgrade.

Here are the steps:

Create a console appplication project in Visual Studio,

Cut and paste the code below and run

Sample output:

InfoFound Management Server KW0081 Database BizTalkMgmtDb
InfoFound MsgBox Server KW0081 Database BizTalkMsgBoxDb
InfoFound PIT Server KW0081 Database BAMPrimaryImport
InfoFound DTA Server KW0081 Database BizTalkDTADb

Sample Code:

Continuation scenarios with mixed of TPE-enabled and non-TPE-Enabled tracking:

It has been a while since I last blogged on this.  Let's continue from where I left off. 

Let's pick two continuation scenarios: the first, a TPE segment continues to a non-TPE segment, and the reverse being the second.

1.      TPE-enabled segment continues to a non-TPE-enabled segment.

In this scenario, you would need only a Continuation folder. The naming of this folder is important since the concatenation of the folder name and its mapped value will form the continuation TokenID used in non-TPE-enabled segment.  For example, if the Continuation folder name is "TPE2OES" and the mapped value has "PO123" for instance, the continuation TokenID would be "TPE2OESPO123".  Once you have mapped other data items and milestones, you will deploy* this.

In the related orchestration, in the process somewhere downstream, you will need to insert the following:

// Continuation segment using OrchestrationEventstream**

// Where ContinuationTokenID based on this instance contains the value ""TPE2OESPO123".***

Microsoft.BizTalk.Bam.EventObservation.OrchestrationEventStream.UpdateActivity("PurchaseOrder",  ContinuationTokenID,

        "Column1", Value1,

        "Column2", Value2);

Microsoft.BizTalk.Bam.EventObservation.OrchestrationEventStream.EndActivity("PurchaseOrder",  ContinuationTokenID);

2.      Non-TPE-enabled segment continues to a TPE-enabled segment.

In the reverse scenario where an activity begins from an non-TPE segment,  the following code would be required:

// Sample main segment of continuation using OrchestrationEventStream.***

// Activity in this instance contains: "PO124"

Microsoft.BizTalk.Bam.EventObservation.OrchestrationEventStream.BeginActivity("PurchaseOrder",  ActivityID);

Microsoft.BizTalk.Bam.EventObservation.OrchestrationEventStream.UpdateActivity("PurchaseOrder",  ActivityID,

        "Column1", Value1,

        "Column2", Value2);

// CONTINUATION LINE HERE:

Microsoft.BizTalk.Bam.EventObservation.OrchestrationEventStream.EnableContinuation("PurchaseOrder",  ActivityID, "OES2TPE" + ActivityID);

Microsoft.BizTalk.Bam.EventObservation.OrchestrationEventStream.EndActivity("PurchaseOrder",  ActivityID);

To create the continue-to segment, in your tracking profile, you would need to add a ContinuationID folder, name this folder as "OES2TPE" and map the relevant item (one that will give you the above "PO124" for this particular instance of message.  Deploy* this and drop a message and see the beauty flows :)

*   You will get a warning that the tracking profile is missing the ContinuationID or Continuation folder, but since we know what we are doing :), just ignore and proceed.

** Note: there is no BeginActivity call since this is a continued-to segment. 

***The above can easilly be changed to used DirectEventStream or BufferedEventStream. If you do so, make sure you call the Flush method explicity or set the flush threshold to 1 or greater in the constructor.

DirectEventStream vs BufferedEventStream

Which EventStream to use:

DirectEventStream (DES) vs BufferedEventStream (BES)

Use DES when you don't want any latency of data appearing in BAMPrimaryImport database.   Since this is synchronous call, any error will be thrown back to the calling application.

Use BES if your concern is throughput.  This is particularly suited for scenarios with spurts of large chuck of events.  A small penalty for using this is that there will be small latency between when the event is sent and the event appearing in BAMPrimaryImport database.

For more info on EventStream: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/sdk/htm/frlrfmicrosoftbiztalkbameventobservationeventstreamclasstopic.asp

How to write BAM Events from pipeline components.

In BizTalk 2004, to write any BAM event from the BizTalk pipeline, you will need to write some code :)  To simplify the learning, I will illustrate by adapting an SDK sample.

Adapt the SDK CustomComponent (found in .../SDK/Samples/Pipelines/CustomComponent) sample, you will find two directories:

• FixMsg - the custom pipeline component (you will need to splice BAM code here) 
• PipelineComponentSample - custom pipeline and will use the above custom pipeline component.

Modify the FixMsg to include the below sample (Search for Execute method):

public IBaseMessage Execute(IPipelineContext pc, IBaseMessage inmsg)
{

IBaseMessagePart bodyPart = inmsg.BodyPart;
if (bodyPart!=null)
{

...
...
//Splice BAM Code here:
WriteBAMEvent(pc, "Execute", inmsg);

}
return inmsg;

}

// Add a new method to write to BAM

private void WriteBAMEvent(IPipelineContext context, string StageDescription, IBaseMessage inmsg)
{

EventStream _BAMEventStr = null;
// Create an EventStream instance
try
{

// Use can use BES/DES or MES
// DES - DirectEventStream - writes direct to BAMPrimaryImport (Synchronous call, no latency)
//_BAMEventStr = new DirectEventStream("Integrated Security=SSPI;Data Source=.;Initial Catalog=BAMPrimaryImport",1);

// BES - BufferedEventStream - writes to MessageBox (Asynchronous call, no wait in call)
// _BAMEventStr = new BufferedEventStream("Integrated Security=SSPI;Data Source=.;Initial Catalog=BizTalkMsgBoxDb",1);

// MES - MessagingEventStream - Async and participates in Pipeline transaction context).
// - Note: Available in BizTalk 2004 with SP1 patch and Biztalk 2006
_BAMEventStr = context.GetEventStream();

}

catch(Exception e)
{

throw (e);

}

// Sample BAM Event Code here:
string ActivityID;
//Create a new, unique activity identifier to use as the ActivityID in BAM
ActivityID="BAM_" + Guid.NewGuid().ToString() + "_" + DateTime.Now + "_" + StageDescription;

//Start the activity record identified by ActivityID
_BAMEventStr.BeginActivity("PurchaseOrder",ActivityID);

_BAMEventStr.UpdateActivity("PurchaseOrder",ActivityID, "ProductName","QFE1117");

_BAMEventStr.EndActivity("PurchaseOrder",ActivityID);

// Optional Flush
//_BAMEventStr.Flush();

}

Reference on MessagingEventStream:

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/sdkmref[bts06]/html/M_Microsoft_BizTalk_Component_Interop_IPipelineContext_GetEventStream.asp

Reference on EventStream:

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/sdkmref[bts06]/html/T_Microsoft_BizTalk_Bam_EventObservation_EventStream.asp

To understand the above and details on how to fetch message specific data, please refer to sectio on Developing Pipeline Components in the help or use this link:
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/sdk/htm/ebiz_prog_pipe_qdri.asp
You would at least need to implement these interfaces.
IBaseComponent, IComponent, IComponentUI.

The above is tedious and is the only means for writing to BAM in Biztalk 2004. In Biztalk 2006, you can do some of this easily via TPE drag and drop.

Means to BAM-Tracked “BAMified” Business Process Data:

Means to BAM-Tracked “BAMified” Business Process Data:

There are several means to enable business process data to be tracked in BAM. In Voyager (BizTalk 2004), programmatically there are the DirectEventStream and BufferedEventStream classes, and non-programmatically there is the Tracking Profile Editor (TPE) drag-and-drop mapping mechanism.

In Pathfinder (BizTalk 2006), both of these areas are further enhanced.  In brief, programmatically, there are the addition of MessagingEventStream and OrchestrationEventStream which are EventStreams that provide transaction consistent processing in the context of Biztalk pipelines and orchestrations respectively.

In the drag-and-drop TPE, in additional to the tracking from orchestrations, tracking can now be done from pipelines as well.

TPE, a shift from orchestration to activity centric view:

There is a fundamental shift in tracking profile view; what used to be an orchestration centric view has now changed to an activity centric view.  Data that you want to be tracked thru TPE for a single activity definition must be mapped in a single tracking profile.   For example, if you have a continuation scenario from one orchestration to orchestration, or a pipeline to an orchestration to a pipeline, all of the information must be presented in one single tracking profile.   In other words, one tracked activity must have only one tracking profile (*.btt) file.

So, what are these Continuation and ContinuationID folders?

A continuation folder is needed for every TPE-enabled tracking that is a source* segment of a continuation.  (Note: I am saying a source and not a main segment since in a multi- continuation scenario, there would be more than one source segments but only one main segment.)

On the other hand, a continuationID folder is needed for every TPE-enabled segment is continued to or the target.

Example:

For a simple scenario where you would continue from Orhestration1 to Orchestration2,  employing the non-programmatic mechanism via TPE drag-and-drop, you would need to create a pair of Continuation and ContinuationID folder.  The name of the folder must be the same, a good naming convention would be to go according to what it does such as “Cont_Orch1_2_Orch2”.  You would then need map the content of each of this folder with a schema item from each orchestration that would contain the same data as it passes from one orchestration to another.   Typically, this piece of data is from the message pay load schema.  This data must be unique from one instance message to another to avoid data from one activity instance polluting another.

You must be curious as to why, such a lengthy explanation.   This is because there are cases where you would not need Continuation and ContinationID folder in a pair.  In cases where you want have a mixed of TPE and non-TPE (DES/BES/MES/OES), you would then need to know exactly when to use Continuation and when to use ContinuationID folders.

Explain it in another way, if you have TPE continues to non-TPE segment, you would need a Continuation folder, and if you have non-TPE continues to TPE segment, you would need a ContinuationID folder.

Now when you do this, what would be the TokenID?   To be continue…

Biztalk BAM Blogging Introduction

As 2005 is coming to a close, I want to take this opportunity to wish you a happy 2006; this also means we are inching closer towards the release of BizTalk 2006 :)

What is this blog good for...you may ask?  This blog is for you the users.  :)  The topic that it focuses on is BizTalk in general and BizTalk BAM specifically.  I will use this as a central depository (referenced from my forum replies) as well as direct replies from which to disseminate insights and personal experience from working with this product on a daily basis. 

You, as the user can help me better help you.  Please use the comment (on this posting) to ask for anything you may want me to delve into... of course, this got to be BizTalk and BAM related :)

Happy BAMifying,

Keith Lim