Kevin B Smith's WebLog

My blog has moved...

2007-03-07T01:33:00Z

I've moved my blogs' home, you can find it here now. Once we ship the book I plan to blog on some wider topics than BizTalk, I'll share some of the interesting things that I've been looking at recently.

Its been a while....the book, BizUnit and my job

2007-03-05T23:49:00Z

Darren, Ewan and myself are really close to getting the book, Professional BizTalk Server 2006 out of the door, it's been hard work but hopefully it'll be worth while. The book offers insight in to how BizTalk works under the hood which will enable you to really exploit the architecture, this is blended with the lessons learned designing, developing, and operating some of the most mission critical BizTalk solutions that have been deployed around the world by Microsoft Consulting Services. I knew it would be a massive amount of work before we started, and it has definitely been that!!

As you may have seen GotDotNet is being phased out by Microsoft, because of this I've moved BizUnit to its new home at CodePlex, I have some great ideas for new features which, now the book is done I might actually get time to implement!! The latest version includes a bunch of work that we did on the last performance lab in Redmond that I was involved in, including some new test steps to enable it to be used to automate performance tests.

Aside form the book, a lots been going on for me. I left Microsoft last summer to work for a leading european investment bank, leaving after 8+ incredible years was not easy I can tell you. I had an amazing time working for Microsoft, a lot of great times. Since leaving I've been playing with a lot of different but very interesting technology, so I should get back to some blogging soon!

BizUnit 2006 is Released – BizTalk 2006 and Visual Studio 2005

2006-04-14T02:04:00Z

I’ve finally got around to finishing off BizUnit 2006 - v2.1 just in time for the BizTalk 2006 GA, this version is targeted at BizTalk 2006 and Visual Studio 2005 and is built using .Net 2.0. I’ve been using the new unit testing capability in Visual Studio 2005 to drive BizUnit for a while now instead of NUnit and have to say I’m pretty pleased with it.

In case you're not aware, BizUnit is a declarative test framework targeted at but not restricted to the automated testing of BizTalk solutions. BizUnit is fully extensible, v2.0 has had over 2077 downloads to date and there are many customers world wide using it to drive up the quality of their BizTalk solutions. Its approach is to enable test cases to be constructed from generic reusable test steps, test cases are defined in XML which allows them to be auto-generated and also enables the ‘fixing up’ of Url’s for different environments, e.g. test, staging and production environments.

This version adds some additional test steps:

FactBasedRuleEngineStep
PerfmonCountersStep
DBExecuteNonQueryStep
DBQueryReturnXmlStep
ExportDBDataToDataSetStep
HostConductorStep
ImportDatasetToDBStep

And some new Validation Steps:

CodeValidationStep
XmlValidationStepEx

I’ve also added wild card support for reading configuration, the following wild cards are supported, let me know if there are others you’d like to see supported:

%DateTime% - will replace the wild card with the current date time in the format HHmmss-ddMMyyyy
%ServerName% - will replace the wild card with the name of the server BizUnit is being executed on
%Guid% - will be replaced by a new Guid

For example, for the test step configuration below:

<SourcePath>..\..\..\TestData\InDoc1.xml</SourcePath>

<CreationPath>..\..\..\Rec_03\TransactionId_%Guid%_%ServerName%.xml</CreationPath>

</TestStep>

CreationPath becomes "..\..\..\Rec_03\TransactionId_12345678-D6AB-4aa9-A772-938972E3FD51_ZEUS001.xml"

If you're new to BizUnit here's a brief overview of how it works...

Test Case Format

A test case is made up of three stages, test setup, test execution and test cleanup, the cleanup stage is always executed (even if the main execution stage fails) and intended to leave the platform in the same state that it started.

Each stage may consist of zero or more test steps, test steps are in general autonomous, state can be flowed between them if required using the ‘context’ object that is passed to each test step by the framework.

BizUnit also has the notion of TestGroupSetup and TestGroupTearDown, these are test cases that are executed at the begginning and end of a suite of unit tests.

The diagram below illustrates the format of a test case.

In addition to test steps, BizUnit has the notion of validation steps and context loader steps. These can be thought of as sub-steps and can in general be independantly executed from any test step. For example, an MSMQ-read step might be used to read and validate both Xml and Flat File data from a queue, the same step can be used with both the RegExValidationStep and the XmlValidationStep to validate the data read.

A test step within a test case can be marked with the attribute - runConcurrently which causes subsequent test steps to be started before it has completed. In addition test steps maybe marked with the attribute - failOnError, setting it to false cause BizUnit to ignore a failure of that test step, this is particularly useful for the setup and cleanup stages of test cases.

Lets look at an Example Scenario...

BizUnit takes a black box approach to testing solutions, if you look at the scenario below, a BizTalk solution receives a request-response message over HTTP, the message is routed to an Orchestration which, sends a message to MSMQ and another to a FILE drop, the Orchestration waits for a FILE to be received, after which the Orchestration sends the response back to the waiting HTTP client. The solution also uses BAM, writing business data to the BAM database.

In order to test this scenario, a BizUnit test case is defined that has 5 test steps:

The HttpRequestResponseStep sends the request to the two-way receive port and waits for the response. This step is executed concurrently so that the other test steps may execute whilst it waiting for the response
The MSMQReadStep waits for a message to appear on an MSMQ queue, when it reads the message it uses the XmlValidationStep to perform schema validation and also execute a number of XPath expression to ensure the message contains the correct data
The FileValidateStep waits for a FILE to be written to a given directory, when it reads the FILE it validates the data using the RegExValidationStep validation step since the FILE picked up was a flat file format
The FileCreateStep creates a new FILE in the specified directory containing the data that the backend system would typically create. This allows the Orchestration to complete and send the response back to the waiting HttpRequestResponseStep step
Finally, DBQueryStep is used to check that all of the BAM data has been successfully written to the BAMPrimaryImportDB

Finally, I'd like to give a big thanks to the following who have contributed new steps, ideas, etc to this version of BizUnit, my apologies if I have missed off anyone, it’s not intentional just my disorganisation!, please let me know if that is the case and I’ll add you:

Jon Fancey

Mike Becker

Tanveer Rashid

Young Jun Hong

If you find bugs, have new steps or ideas/requirements for new features that you'd like incorporated let me know.

Enjoy!

Debugging BizTalk Maps and XSLT's...

2006-04-08T16:42:00Z

Did you ever want to debug a BizTalk map, or any XSLT for that matter? VS 2005 has a great new feature that enables you to do just that. If you are using BizTalk and want to debug your map here’s what you do:

Right click the BizTalk map and select “Validate Map”
In the output window you’ll see a link to the .xsl, click on the link to bring it up in VS
Right click the .xsl and select “View Source”
At that point you can set break points in your .xsl
Then select”XML / Debug XSLT” – at which point you’ll be able to step thru the XSLT

BizTalk Server Team Blog...

2006-03-23T23:30:00Z

Michael Woods, Kris Horrocks, Steve Sloan, and Mark Berman from the BizTalk Server product group have started up a blog which will I'm sure fill the void left after Scott changed groups, they already have a bunch of postings up there.

Concurrency and Coordination Runtime...

2006-01-16T23:54:00Z

I've been tracking the CCR internally within Microsoft for a while now, and now there's an interesting channel 9 interview with the CCR guys: http://channel9.msdn.com/Showpost.aspx?postid=143582 if you've ever done any amount of concurrent programming you may find it interesting. As we move towards multi-core CPU architectures concurrent programming will become essential in order to make apps scale. Enjoy!

The BizTalk Product Group is Hiring....

2006-01-13T03:34:00Z

Do you want to join Microsoft? Would you like to help to shape and build the next generation of BizTalk Server? The BizTalk product group is looking for talented individuals to help build the next generation of the product.

If you're interested or want to find out more about the opportunities email Kerry Van Voris your resume: kerryv@microsoft.com

Routing Messages to the MQ Dead Letter Queue

2005-11-04T10:40:00Z

I recently had a fun problem to solve for a client of mine, essentially they wanted to route messages to the MQ Series dead letter queue and have the MQ dead letter queue handler move those messages to the queue they defined in the dead letter.

The first problem was getting the message on the MQ dead letter queue, John and Anil gave some great pointers here, it turns out to send a message to the dead letter queue the message needs to be pre-pended with the dead letter header (DLH) and the MQMD_Format property needs to be set to MQ "MQDEAD " so that MQ knows to expect the DLH.

Serializing the Dead Letter Header

To achieve this I wrote a DLH utilities component that allows the various fields of the DLH header to be set, the component also serializes the DLH into a byte[] so that it may be pre-pended to the message. The component was called from a custom send pipeline component which allows the key fields to be set at design time, the pipeline component also sets the MQMD_Format property to the message context.

The format of the MQ DLH struct is as follows:

char[] strucId = new char[4]; // Structure identifier - MQCHAR4 StrucId

int version = 1; // Structure version number

int reason; // Reason message arrived on dead-letter

char[] destQName = new char[48];// Name of original destination queue

char[] destQMgrName = new char[48];// Name of orig dest queue manager

int encoding; // Numeric encoding of data that follows MQDLH

int codedCharSetId; // Char set identifier of data that follows MQDLH

char[] format = new char[8]; // Format name of data that follows MQDLH

int putApplType; // Type of app that put message on DLH

char[] putApplName = new char[28]; // Name of app that put msg on DLH

char[] putDate = new char[8]; // Date when message was put the DLH

char[] putTime = new char[8]; // Time when message was put the DLH

This struct needs to be serialised into a byte[] and pre-pended to the message:

public byte[] SerializeHeader()

{

byte[] header = new byte[172];

int index = 0;

byte[] tmp = null;

// strucId

int written = System.Text.Encoding.UTF8.GetBytes(strucId, 0, strucId.Length, header, index);

index += 4;

// version

tmp = BitConverter.GetBytes(version);

tmp.CopyTo(header, index);

index += 4;

// reason

tmp = BitConverter.GetBytes(reason);

tmp.CopyTo(header, index);

index += 4;

// destQName

written = System.Text.Encoding.UTF8.GetBytes(destQName, 0, destQName.Length, header, index);

index += 48;

// destQMgrName

written = System.Text.Encoding.UTF8.GetBytes(destQMgrName, 0, destQMgrName.Length, header, index);

index += 48;

// encoding

tmp = BitConverter.GetBytes(encoding);

tmp.CopyTo(header, index);

index += 4;

// codedCharSetId

tmp = BitConverter.GetBytes(codedCharSetId);

tmp.CopyTo(header, index);

index += 4;

// format

written = System.Text.Encoding.UTF8.GetBytes(format, 0, format.Length, header, index);

index += 8;

// putApplType

tmp = BitConverter.GetBytes(putApplType);

tmp.CopyTo(header, index);

index += 4;

// putApplName

written = System.Text.Encoding.UTF8.GetBytes(putApplName, 0, putApplName.Length, header, index);

index += 28;

// putDate - Format: yyyymmdd

written = System.Text.Encoding.UTF8.GetBytes(putDate, 0, putDate.Length, header, index);

index += 8;

// putTime - Format: hhmmss00

written = System.Text.Encoding.UTF8.GetBytes(putTime, 0, putTime.Length, header, index);

index += 8;

return header;

}

Pipeline Component

The MQ adapter does not directly support setting this property, so the custom send pipeline component is responsible for pre-pending the message data stream with the DLH and setting the message context property which the MQ adapter will then set on the MQ message:

private static PropertyBase mqmd_Format = new MQSeries.MQMD_Format();

private const string MQFMT_DEAD_LETTER_HEADER = "MQDEAD ";

// Set MQMD_Format property to MQFMT_DEAD_LETTER_HEADER - to indicate the message

// is prepended with the dead letter header...

inmsg.Context.Write( mqmd_Format.Name.Name,

mqmd_Format.Name.Namespace,

MQFMT_DEAD_LETTER_HEADER );

// Pre-pend the message body with the MQS dead letter header...

inmsg.BodyPart.Data = DeadLetterHelper.BuildDeadLetterMessage(

inmsg.BodyPart.GetOriginalDataStream(),

_DestinationQueue,

_QueueManager,

_ApplicationName );

All that is required then is for the send port to be configured to send the message to the SYSTEM.DEAD.LETTER.QUEUE.

Dead Letter Handler

Once the message is on the dead letter queue the dead handler (runmqdlq.exe) can be configured to take the message off the queue and put it on the destination queue defined in the DLH. This proved to be a little tricky to get working, thanks to Jason for his help in getting the dead letter handler working.

runmqdlq.exe SYSTEM.DEAD.LETTER.QUEUE QM_demoappserver < qrule.rul

The handler may be fed a rule (qrule.rul), the example here removes the DLH and puts the message on the queue specified in the DLH, one thing to watch out for is the rule below needs to have a CRLF at the end!!:

INPUTQM(QM_demoappserver) INPUTQ('SYSTEM.DEAD.LETTER.QUEUE') WAIT(NO)

ACTION(FWD) FWDQ(&DESTQ) HEADER(NO)

Dave Green is blogging...

2005-09-20T22:53:00Z

Dave Green has started his blog, welcome Dave!. Dave is the WWF Architect, and he's been having stacks of fun building out Microsoft's workflow platform, I have to say WWF looks really cool. Dave's a very smart cookie and has been building workflow engines for some time now, I'm sure he'll share some interesting insights around the workflow space. Dave, John and myself were the British faction in the BizTalk team before I defected back to Blighty!

BizUnit 2.0 - Automated Testing for BizTalk Solutions

2005-07-17T16:11:00Z

I've released BizUnit 2.0, in case you’ve not come across it before, BizUnit is a framework that I put together to enable the rapid development of automated functional testing for BizTalk solutions, the motivation was to provide a framework that would bring automated testing of BizTalk solutions to the masses!! BizTalk solutions are often mission critical, an automated testing approach helps to drive quality before go-live and is essential post go-live to ensure that patches to not cause regressions. BizUnit is not restricted to testing BizTalk solutions, but it is targeted at them.

BizUnit defines test cases as Xml documents, which, enables test cases to be rapidly written and also enables the same test cases to moved between different environments by fixing up the Url’s for the endpoints, for example the endpoint Url’s for a development environment will be different to a testing, and pre-production environments.

Also, by having the test cases defined as Xml, it allows them to be easily generated, some of the customers that I have worked with define their test matrixes in Excel, and then auto-generate the BizUnit test cases from Excel.

Test Case Format

A test case is made up of three stages, test setup, test execution and test cleanup, the cleanup stage is always executed and intended to leave the platform in the same state that it started.

Each stage may consist of zero or more test steps, test steps are in general autonomous, state can be flowed between them if required using the ‘context’ object that is passed to each test step.

How does it Work?

BizUnit takes a black box approach to testing BizTalk solutions, if you look at the scenario below, a BizTalk solution receives a request-response message over HTTP, the message is routed to an Orchestration which, sends a message to MSMQ and another to a FILE drop, the Orchestration waits for a FILE to be received, after which the Orchestration sends the response back to the waiting HTTP client. The solution also uses BAM, writing business data to the BAM database.

In order to test this scenario, a BizUnit test case is defined that has 5 test steps:

The HttpRequestResponseStep sends the request to the two-way receive port and waits for the response. This step is executed concurrently so that the other test steps may execute whilst it waiting for the response
The MSMQReadStep waits for a message to appear on an MSMQ queue, when it reads the message it uses the XmlValidationStep to perform schema validation and also execute a number of XPath expression to ensure the message contains the correct data
The FileValidateStep waits for a FILE to be written to a given directory, when it reads the FILE it validates the data using the RegExValidationStep validation step since the FILE picked up was a flat file format
The FileCreateStep creates a new FILE in the specified directory containing the data that the backend system would typically create. This allows the Orchestration to complete and send the response back to the waiting HttpRequestResponseStep step
Finally, DBQueryStep is used to check that all of the BAM data has been successfully written to the BAMPrimaryImportDB

BizUnit 2.0 Enhancements

A number of enhancements have been added to BizUnit 2.0, firstly, there are now some 37 test steps, which is vastly improved from the previous version. A lot of people have contributed test steps which I’ve included in this version. Briefly, here’s some of the enhancements:

Concurrent test step execution option – allows multiple test steps to be executed at once
Context – allows state to be flowed between test steps, think of the scenario where a FILE is written the content of which specifies where the response should be written
Test group setup / tear down stages - allows group wide setup and cleanup
Help – all of the test steps have help in a .chm (accessed from Program Files\Microsoft Services BizTalkApplicationFramework\BizUnit 2.0\Documentation. The help contains Xml configuration examples for each test step that can be copy/pasted into a test case, along which a description of each configuration option etc
FailOnError – flag to allow failed test cases to be ignored, this is useful during the cleanup stage

Here’s a list of all of the test steps in BizUnit build 2.0.1062.0, the documentation provides more details around what these steps do and how to use them:

BAMDeploymentStep

CheckPop3MailStep

ContextManipulatorStep

CrossReferenceDataVerificationStep

CrossReferenceSeedClearStep

CrossReferenceSeedLoadStep

DatabaseDeleteStep

DatabaseRowCountStep

DBQueryStep

DelayStep

DotNetObjectInvokerStep

EventLogCheckStep

EventLogClearStep

ExecuteCommandStep

FileCreateStep

FileDeleteMultipleStep

FileDeleteStep

FileMoveStep

FileMultiValidateStep

FilesExistStep

FilesMoveStep

FileValidateStep

HttpPostStep

HttpRequestResponseStep

MSMQCreateQueueStep

MSMQDeleteQueueStep

MSMQQueuePurgeStep

MSMQReadStep

MSMQWriteStep

OrchestrationConductorStep

ReceiveLocationEnabledStep

ReceivePortConductorStep

RenameDirectoryStep

RuleEngineStep

SMTPReadStep

SOAPHTTPRequestResponseStep

WaitOnFileStep

MQSeriesClearQueueStep

MQSeriesGetStep

MQSeriesPutStep

OutlookReadStep

Validation Steps:

BinaryValidationStep

ContextValidationStep

RegExValidationStep

XmlValidationStep

Context Loader Steps:

RegExContextLoader

TextContextLoader

XmlContextLoader

Test Coverage:

So, you’ve developed all of your BVT’s (Build Verification Tests) for your BizTalk solution using BizUnit, how do you know whether you’ve done a good job, and how much test coverage you have? Enter Jason Births “Orchestration Profiler”. This is a great tool that you can use in conjunction with BizUnit to ensure that you have adequate test coverage, it produces a detailed graphical report of what the test coverage is like, I highly recommend using it with BizUnit.

Finally, I’d like to give my special thanks to a number of people that have contributed test steps, requirements and ideas for BizUnit 2.0, my apologies if I have missed off anyone, it’s not intentional just my disorganisation!, please let me know if that is the case:

Dave Regan

Tanveer Rashid

Daren Jefford

Kevin Purcell

Karina Apostolides

Jon Bonnick

Brian Milburn

Rahmatullah Khan

Future work:

A couple of the guys above have done some great work building a Visual Studio plugin for BizUnit and a command line driver. I didn’t have time to get it in this release, but it’ll be following soon. If you find any bugs, or have test steps you'd like to contribute please sned them along to me and I'll get them into the next drop.

Enjoy!

Announcement: Get Connected to Free Product Support and Tremendous Online Collaboration

2005-06-22T09:38:00Z

The product group have asked me to post this announcement to let you know about the great online support that we have available, I strongly encourage you to take a look because (1) its free J, (2) there is a lot of good traffic thru it and (3) a lot of the folks in the product group monitor and answer questions for their specific feature areas.

The general BizTalk newsgroup can be found here

Get Connected to Free Product Support and Tremendous Online Collaboration

Have you ever wanted to speak to Microsoft developers of a specific feature of BizTalk Server? I am sure your answer was “Yes let me at them”, so the Business Process Integration Division is extending an invitation to all customers to join our key feature developers, program managers, and testers in the following newsgroups:

microsoft.public.biztalk.accelerator.forsuppliers
microsoft.public.biztalk.newuser
microsoft.public.biztalk.accelerator.rosettanet
microsoft.public.biztalk.admin
microsoft.public.biztalk.appintegration
microsoft.public.biztalk.framework
microsoft.public.biztalk.general
microsoft.public.biztalk.library
microsoft.public.biztalk.nonxml
microsoft.public.biztalk.orchestration
microsoft.public.biztalk.sdk
microsoft.public.biztalk.server
microsoft.public.biztalk.setup
microsoft.public.biztalk.tools
microsoft.public.biztalk.xlangs
microsoft.public.biztalk.xsharp

We’ve been working very hard over the past year to connect with folks just like you and want to include you in our community of Most Valuable Professionals (MVP), developers, information technology professionals, chief information officers, chief executive officers, or any other role within large, medium, and small companies that hang out in our online newsgroup communities. We want to have you join in this vibrant online community to ask those questions you always wanted to ask but did not know where to go. Well, now you know where to go, we want you to come on in and join us!

If you are new to BizTalk Server, try out the NewUser newsgroup, Microsoft.public.biztalk.newuser. We are encouraging novice users to post question here.

We’re offering two levels of interaction with Microsoft Corporation employees as follows:

Managed Newsgroup Support
Unmanaged Newsgroup Support

Managed Newsgroup Support

MSDN managed newsgroups are available in English to MSDN Universal, Enterprise, Professional and Operating Systems subscribers to receive free technical support on select Microsoft technologies as well as to share ideas with other subscribers. MSDN managed newsgroups provide:

Unlimited on-line technical support - keep your PSS incidents
A commitment to respond to your post within two business days
Over 200 newsgroups to choose from
Spam protection for your e-mail address when posting items

Go to the following URL to sign up: http://msdn.microsoft.com/newsgroups/managed . These newsgroups are monitored by Microsoft support engineers and product group team member as described above.

Unmanaged Newsgroup Support

MSDN unmanaged newsgroups are available to all individuals.

Go to the following URL to participate: http://msdn.microsoft.com/newsgroups. These newsgroups are monitored by Microsoft product group members, other customers like you, most valuable professionals, and various other individuals.

Questions, suggestions, and direct feedback can be sent to me.

James Fort

BPI Community Lead

mailto:jfort@microsoft.com

Controlling the CLR Thread pool hosted by BizTalk

2005-05-04T22:55:00Z

The CLR thread pool is global to the process, meaning that all components using the thread pool in a given process share the same thread pool and therefore compete for it. This for the most part is a good thing as it is more efficient in terms of system resources. When you do performance testing/tuning there are scenarios when you need to control the thread pool, for example when using the SOAP Adapter.

I came across this little gem last week, I hadn’t realized that Q886966 added the registry keys MinCompletionPortThreads, and MinWorkerThreads and then SP1 added the MaxWorkerThreads key below. These all exist under:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\BTSSvc{Guid}\CLR Hosting

MinWorkerThreads - the minimum number of worker threads that are available in the thread pool

MaxWorkerThreads – the number of worker threads in the thread pool

MinCompletionPortThreads – default 150

MinWorkerThreads can be tuned to address the problem due to the slow speed at which the thread pool can sometimes add new threads, this typically happens during service startup or during a burst of new load after a lull period. MaxWorkerThreads can be used to increase the number of threads in the thread pool, and this can be used to address the “out of threads issue” while using the SOAP adapter.

Note, as with most BizTalk registry keys they don’t exist by default, you’ll need to create them in order to use their non-default values.

BAM: Tracking Portal

2005-04-29T22:45:00Z

As we all know BizTalk Server 2004 provides a great set of enabling technologies to address a number of business problems, specifically around integration and business process management, the very nature of this technology means that BizTalk is typically deployed into the very heart of the enterprise and often has very mission critical applications running on it. With this of course comes a big responsibility in terms of how we build real enterprise class applications on BizTalk. Over the coming weeks and months I intend to push out to a broader community some of the learning’s and findings that I beleive make a real different in terms of increasing the chances that a BizTalk Server deployment will be successful, not only in terms of getting it to go-live but also around ensuring it’s cost of ownership is kept to a minimum, I’ll try to keep these all under the theme of Enterprise Architecture. We’re also doing a much bigger piece of work around this but more about that another time :-).

The first of these areas is the Tracking Portal, Darren has been hounding me (and rightly so) to blog some stuff on this for quite a while now, so dude, here it is…finally :-). The Tracking Portal often takes a back seat to getting the solution out of the door, the fact of the matter is however that tracking is core functionality in my opinion that needs to be designed in from the outset. A Tracking Portal really needs to surface two views into the solution, first it needs to provide a view into the business for the users of the applications sometimes this means MI but other times simply the business view of long running transactions, most people get this and also that BAM is well placed to provide this, and I have to say what great technology BAM is and it usually provides the ‘wow’ to the business users. The second view is often neglected, but it’s equally as important, it needs to provide a view into the solution from a functional perspective for the solution support team to use in order to troubleshooting the live system. There are other tools like HAT that can be used to troubleshoot, but HAT does not have the understanding of what combination of message flows constitute a business transaction, and typically this is what both the business and support users care about rather than individual discrete messages.

The approach that we have been using over here in jolly old Blighty for a while now, is to build the Tracking Portal using BAM to capture the key state and business transaction information, and SQL Reporting Services, (another technology that I think is very cool indeed), to render the Business and Support views from the tracking database. Using this approach the Tracking Portal can be put together very cheaply with a rich reach UI, and in my experience providing these views, especially the Support view could get you out of some very difficult situations after you goes live. Ok, so let’s dig a little deeper…

Using BAM to Capture Key Business Transaction Information

There are essentially three way you can use BAM, TPE, BufferedEventStream or DirectEventStream, the first TPE, allows you to define what tracking data is captured at specific points in an Orchestration, while this is great to setup quickly it has the problem that every time you change your Orchestration you need to redefine your TPE. The second problem is that you can’t use it in the pipeline, for some scenarios this presents a break in your tracking data between the time the business transaction reaches the adapter to the time the message was published by the messaging engine and routed to the appropriate orchestration. For me, these two reasons typical cause me to discount TPE.

The other two are very similar to each other in usage, the main difference being in the way that the tracking data is transferred to the BAMPriamryImport DB, the BufferedEventStream is asynchronously processed which essentially means is less of a performance hit, while DirectEventStream causes the tracking data to be written directly to the BAMPriamryImport database.

In general the BufferedEventStream is preferable for performance reasons. As I mentioned earlier, to get a full view of how business transitions flow it generally means that the BufferedEventStream needs to be called from a custom pipeline component in the receive pipeline, the orchestrations, and again in a custom pipeline component on the outbound path, so providing a simple stateless wrapper API around the calls to the BufferedEventStream is a good approach, this API can then be called from a pipeline component or an orchestration.

In order to tie up the individual messages into a message flow which represents a ‘business transaction’, a unique business transaction ID is needed, often there is the notion of some unique identifier for message flows depending on the scenario, but in the scenarios where there isn’t, the system context property InterchangeID maybe used since it is guaranteed to be unique and will be flowed with the message even as the message is cloned through each stage in the pipeline. If the message is copied in the orchestration it is the user who responsible for ‘flowing’ the InterchangeID.

Each new business transaction will start a new Activity, this is done by calling the EventStream.BeginActivity() API, an activity is ended by calling EventStream.EndActivity(), of course for business transactions that span the receive pipeline, and orchestration etc, BAM has the concept of continuing an activity, for this the EventStream.EnableContinuation() API is called passing in a unique identifier to tie the activities together. The property BufferedEventStream.FlushThreshold is used to control the frequency at which the events are flushed to the BAM database, in general setting this to ‘one’ works well. Neither the DirectEventStream or the BufferedEventStream are serializable currently, the implications of using them from Orchestrations are that they need to be created each time they are used rather than defining a local variable in the Orchestration, this may sound like an issue, but in the past we’ve taken customer scenarios into the perf lab and profiled the BizTalk solution to understand the cost associated with using the EventStream in this manner, in the grand scheme of things the cost of creating it and using it with a flush threshold of one is not significant.

Pipeline components can get an EventStream from the pipeline context using IPipelineContext.GetEventStream(), this was added in QFE 1117, and is in SP1, this ensures transactional integrity between the tracking data and the message box interactions.

One final point is that it is possible to track message bodies also using BAM, though the restriction is that there is a limit to the size of the message bodies of 3885 bytes, which isn’t that big. Having said that, using this approach outlined here, we’ve measured the cost of this tracking on a complex customer scenario, and found it to be in the order of 20% slower than the same solution with no tracking, which is pretty impressive.

Using SQL Reporting Services to Build the Portal

Ok, so now we have a very flexible mechanism to collect tracking data for business transactions at run-time, we need a slick way to render the web based tracking portal, enter SQL Reporting Services. SQL RS can be used to build the tracking portal views (business focused and support focused) very easily, it allows you to use drag and drop techniques to build the UI and define the layout of the report. Since the portal is built on SQL RS, it’s really easy to provide rich querying capabilities as well as hierarchical views that for example allow you to look at a business transaction then drill down to the individual message flows. The approach here is to point SQL RS at the BAM views that generated when the activities are deployed, and allow SQL RS to query over these views.

BizUnit v2.0 is imminent...

2005-04-29T20:34:00Z

Version 2 of BizUnit is pretty close to compleation, the new version has the notion of passing state between the individual test steps plus plenty of new test steps not ony from myself but also from a number of customers and Microsoft partners.

As I mentioned previously, for those of you that haven't used BizUnit, it's a framework for the rapid development of automated testing, it was initially targetted at BizTalk but it's not by any means restricted to it. A number of my customers, past and present have had great success using it.

If you have test steps that you like to have included in Version 2, please let me know, I'm aiming to get it finished in the next month or so.

Processing Large Interchanges

2005-04-27T21:18:00Z

Recently I’ve have had quite a few customers asking about the processing of large interchanges, and restrictions around them. Firstly, let's make sure we're on the same page regarding what an interchange is, it's a single message that contains many individual messages that are disassembled in the receive pipeline, an example of this would be a flat file which is disassembled (using the FlatFile dissassembler) to produce many smaller messages, each of which are published independently.

As you’ll know BizTalk Server 2004 has a strong large message story, the engine jumps through hops in order to keep a flat memory model at run-time. If you consider the implications of this on a large interchanges they are quite interesting...

Interchanges are processed using one atomic transaction, meaning that either all of the messages disassembled from the interchange are published to the message box or none of them are. But clearly if the messages disassembled from an interchange were committed to the message box only after the interchange had been entirely disassembled, the engine would reach an out of memory condition for large interchanges. In order to keep the memory model flat the engine breaks the interchange into many sub-batches under the scope of a transaction, the transaction is committed only once the interchange has been entirely disassembled successfully. Because the interchange is sub-divided the memory for each sub-batch may be relinquished in order to keep the memory model flat. This all sounds well and good, how ever, each one of these messages keeps a SQL lock, and SQL does not have a limitless number of locks!!

For large interchanges, where large is say 100,000, it’s pretty easy to reach this lock limit when concurrently processing large interchanges. The reason being due to the multi-threaded nature of the engine it is possible to have many interchanges being processed at once. While out of the box the engine restricts the number of active threads per CPU, a context switch will cause another thread to be released and potentially kick off the processing of another large interchange, hence more locks will be taken.

If you’ve not seen it, the performance white paper has a simple equation that can be used to estimate the maximum size of an interchange:

Maximum number of messages per interchange <= 200,000 / (Number of CPUs * BatchSize * MessagingThreadPoolSize)

So, around this, there are a few options that you can employ to reach the out of locks scenario for large interchanges:

-         Pre-split the interchange before it hits BizTalk, the size can be determined using the formula above
-         Use 64bit SQL, I believe there are more locks
-         Write a custom adapter that pre-splits the interchange
-         Write an orchestration to split the interchange

The last point to consider is that smaller interchanges allow for more efficient processing so the raw through put will be greatly increased by using smaller interchange sizes, again there is some good data around this in the performance white paper.