"From the Field"- Neil Thompson : BizTalk

BizTalk Server 2006 - Enterprise Production Considerations - Part 4 - BAM

neilth — Tue, 23 Oct 2007 19:12:00 GMT

Obligatory Introduction

BAM is such a good name for this technology for a few reasons. The first is that it reminds me of the kid Bambam on the Flinstones (You know, the baby with the club that had a thing for Pebbles). Very powerful and very cool. Also because "BAM" is the sound my forehead makes while it collides with the palm of my hand as I discover I could have simply used BAM instead of writing a custom reporting app.

Sample to Whet the Appetite

Here is a fictitious BAM portal Aggregation for one of my lab projects

The Basics

This article will be a fairly unstructured summary of my experiences with BAM under BTS 2006. If there is interest, I will create some more in-depth posts on the topic. I'll be brief here because most of you know this already. The basic way BAM works is the following:

Create a Business Activity showing the LOGICAL progression and points of interest in a process (using Excel, or Visio ODBA)
Deploy that Activity (as .xls or .xml to the biztalk server) using BM.exe
Associate low level technical events (i.e. hitting a shape in an orch or a method in a .net class) with items you defined in the activity (using the BAM API or the Tracking Profile Editor (TPE)

The end result of this is that you can take the mess of code, orchestrations etc and transform it back into a logical/business view that a CEO could understand. All in all, very cool.

Some Pragmatic Advice

"DO" List

Try both ODBA and Excel for creating activities (I like excel better, but hey, who cares?)
Become familiar with the BM.exe utility. You really only need the "deploy-all" and "remove-all" commands to start with
Keep your activity files versioned! If you deploy an activity, then change the activity file, then try to undeploy you may have problems. Then you have to undeploy by manually deleting tables and rows in the BAMPrimaryImport table.
I recommend that you manipulate your activity definition in Excel. When you are ready to deploy it, export it to xml and then deploy the xml. That way if you fiddle with the Excel file, you can still undeploy the activity using the same xml you used to deploy it.
Before you re-export your activity to XML, undeploy the activity. Or even better export your Activity to a new version of the xml (i.e. with a version number in the file name
Start with the tracking profile editor, master it, then move onto the BAM API calls.

"DON'T" List

Don't try to map milestones from Message Constructs or Loops (or other special cases). That is advanced and you may need to use the BAM API call to make that work as you would expect
Don't put spaces or special characters in your Activity Names. It can lead to problems later where the Event Log will say that "stored procedure does not exist" and your BAM Aggregation will not work.
Don't have a group shape as the last shape in the orchestration you are tracking in TPE. For some reason it won't work. Just add any shape (i.e. Expression) after the group to fix the issue.

Continuations & Relationships

I can't do these justice in this entry so this will be a bit shallow. I will probably blog on this separately. The more comments i see below requesting this, the sooner I will get around to it.

Continuations allow a single activity defintion to span muliple orchestrations or even other classes/systems seamlessly. It can also allow you to use different unique identifiers accross those systems to When using TPE to create a continuation. Create a continuation and name it, then create a continuationID and name it the exact same thing. Map some kind of ID value (PO ID, RequestID, etc.) from a messaging shape (receive, construct, send as described in the above DO List) in the first orchestration to the continuation. Then in the second orchestration map that same ID value to the continuationId. The idea is that the engine will recognize that those to values match and will correlate them so that both instances can contribute data to the same Activity instance

Use Continuations when the Relationship is 1:1
Use Relationships when the Relationship is 1:Many

Real-Time vs. Scheduled Aggregations

There is a little button in the Excel view (if you have installed the BAM addon) that will mark the aggregation as RealTime or Scheduled. Again, this is complex so I will summarize below.

	RealTime	Scheduled
Maintenance	Low maintencance - The data is kept in the relational tables and periodically cleaned. All you have to do is set the TimeSlice and TimeWindow values to determine the trimming behavior	High Maintenance - This data is kept in cubes so you need to have Analisys server setup, schedule the packages, and make sure they execute as expected
Amount of Data	Small Amount of Data - Because this is kept in the relational tables the volume of data has to be kept rather trim	Large Amount of Data - Because this is kept in cubes you can have almost unlimited amount of data
Accuracy of Data	Very Accurate - This provides up to the second view of data and can show in progress activities	The data you have is correct, but you will be missing any data collected since the last execution of the packages
Comprehensive Measures	Limited - You can not use MIN,MAX, and apparently AVG functions in RTA	Unlimited - You can use any type of measure in scheduled packages
Performance	This does contend with SQL and BizTalk for resources on the production box.	This is presumably a separate machine serving as a DataWarehouse so it should not contend with the production server for resources. (except to execute the packages)

BAM Alerts

For the most part, real time data is better for driving alerts, as they are usually time sensitive.

Common Problems & Solutions

Problem: The BAM Activity and the Tracking Profile have been deployed, the process has been run, but there is no data displayed for the activity in the BAM portal or LiveData Excel sheet.

Solution: When you deploy a non-Real Time Aggregation (this is default) while using SQL 2005. It will create the <package>_DM and <package>_AN packages. These packages need to be run to populate the BAM tables and get you the BAM Data. To run these packages you should select "Integration Services" in the SQL Management Console (not database engine). Once you have found them, run them. Hopefully, you're data will appear.

Problem: The BAM portal or LiveData Excel sheet shows "BLANK" for one column and one row in the Aggregation display. It seems like the measures, and the milestones are not being combined into the same activity. Also, the data shows up in two rows (or more) in the BAMPrimaryImport.dbo.bam_<ActivityName_Completed table.

Solution: This could be a number of things, but if it was the same problem that I had the solution is the following. When you are in the TPE and you are associating data from the message, be sure to use the orchestration schedule. Right click on your receive, Message Construct, or Send shape and select Message Payload and map the items from there. Many people go to the "Select Event Source" and use the "Messaging Payload" from there. YOU ONLY WANT TO DO THAT IF YOU ARE JUST USING SEND RECEIVE PORTS TO MOVE MESSAGES. If you are using an orchestration, map everything from the orchestration schedule.

Problem: Your BAM aggregation is not working and you see "Could not find stored procedure" in your event log.

Solution: This is probably caused becaues you have a "space" or some other bad character in your Activity Name. Undeploy your activity, rename it so it does not have the space. Redeploy it. Adjust your TPE (if required). This should fix the problem.

Wish List

Not being able to have spaces in the Activity Name AND not being warned about it is a serious limitation
The length of Activity Names and Item names is too short. I wish for longer names.
Measures using Average also seem to not show for RealTime Aggregations (at least they did not in mine).
I would like to see this integrated with the Visual Studio Environment to a much greater degree. My vision is that we use Office and VS and nothing else.
I also want to be able to deploy BAM stuff with MSI easier

BizTalk Server 2006 - Enterprise Production Considerations - Part 2 - Host and Host Instances

neilth — Tue, 31 Jul 2007 15:57:00 GMT

I've seen one too many BizTalk groups with poorly configured Host/Host Instances. So I've decided to create a BASIC post on the subject. This is a pretty sophisticated topic having to do with scaling, high availability, resource management etc. so please don't assume that what I'm about to say is universally appropriate. I'll give a few suggestions and more importantly, I'll give the reasons behind those suggestions which should give you the background you need to create your own configuration wisely.

Suggested Starting Point

Enable "Allow host tracking" on the default in-process BizTalk Application Host
1. Do NOT enable "Allow host tracking" on any other Host
  Confirm with the BTS docs if you like, but only one host has to have this enabled
2. Do not use this host for any ports or orchestrations, this is dedicated for host tracking
Create at least 3 in-process hosts for each logical application that you have. (Use at least 2 hosts if you are not using any orchestrations)

One for Send Operations [Bind your send points to this host]
One for Receive Operations [Bind your receive ports to this host, except HTTP or SOAP]
One for Processing [Bind your orchestrations to this host, if you have any]
A case by case examination can be made here for additional hosts, usually for:

High volume orchestrations or those you wish to have granular control over. These get a dedicated host
Out of process hosts for HTTP/SOAP/Custom protocols . These get additional Hosts/Instances

Application Isolation
Having separate hosts for your applications allows you to restart the host instances for that application (perhaps because of a re-deployment) without interrupting the other apps.

Scaling Out
Having a host for Send/Receive/Process allows you to distribute your application accross several servers in your BizTalk group in a flexible manner. You can increase the number of host instances running to "beef up" any combination of these three types of operations as required.

Memory Utilization
As you well know, 32 bit systems have limits regarding how much memory can be used by a single host (process). A single host on a 32GB system will really only use about 1.8GB - 2GB or so. If you want to take full advantage of all of that juicy memory you'll have to create more host instances and let each of them take their 2GB bites out of it. (As a rule of thumb, be sure to leave 4GB for the OS). This is not universally true (i.e. 64bit systems are different)

High Availability
Have at least two host instances running for every host that you have. This way, any single node can fail and the other node will continue to operate. This can become more complicated for Receive nodes so check the HA configuration for whatever adapter types you have running. For instance, FTP does not support locking so having two host instances reading from it can cause duplicate messages. This has to have just one host instance reading from with (which I tend to call a "Singleton" adapter configuration). In this case there is one host that is clustered for high availability

Security
You may also need to create additional hosts so that you can use different credentials to access different systems. The security context is determined by the Host/Instance which can be very helpful in scenarios that are using AD to authenticate your requested operations. (i.e. File Adapter, SQL Adapter, and I believe the MS CRM adapter, among others)

Some Caveats

Too many hosts.

My BizTalk groups seldom have more than 22 hosts or so (7 apps + default host for tracking) So if you begin to experience problems with Groups having more hosts than that I recommend you consider too many hosts as a possible cause. There could be RAM availability issues at these levels.
Configurations in BTSNTsvc.exe.config may give you weird results because you may be assuming that they apply to the whole group, when really they apply to the host instances
Again, in the BTSNTsvc.exe.config, the changes in this file are picked up when the host instance restarts. You may have changed the configurations, restarted only a handful of your hosts. Later down the road, you restart other host instances that then begin to have problems because of the contents of the BTSNTsvc.exe.config. This is VERY hard to trouble shoot and I have two recommendations.
1. Don't change BTSNTsvc.exe.config unless you absolutely MUST.
2. If you change BTSNTsvc.exe.config restart ALL of your host instances immediately so you can quantify the affect of the changes immediately.

Single Box groups
1. If you're running BizTalk all on one box then it hardly helps you to be able to partition out the operations. In this case, I would just create one host per application so you can still have Application Isolation, without worrying about the scale out features or HA (which you can't really use with just one box anyways)

Agree, Disagree, Don't Care? Please comment below

BizTalk Server 2006 - Enterprise Production Considerations - Part 1 - Production Server Topology

neilth — Thu, 26 Jul 2007 17:50:00 GMT

Introduction:

This post is the first in a series of posts that will describe a number of production considerations for Enterprise customers (“Enterprise” loosely defined as those with moderately complex BizTalk implementations)

This is in response to a great deal of confusion that I have encountered in the field and hopefully I can kill about 500 birds with 1 stone by publishing these findings (no offence to bird enthusiasts J ).

Today’s Topic – Starting the Production Infrastructure

BizTalk Server 2006 has a dizzying array of scalability options, but many options can mean complexity. This post will specifically discuss the most common “starting points” for production enterprise BizTalk systems and compare/contrast them so you can hopefully figure out which one best suits your needs.

The X + Y notation means:

X Dedicated BizTalk boxes
Y SQL Server Nodes

Server Configurations

Before we get started here, I have to emphasize that these are all common STARTING configurations. Configurations can (and do) scale out and up depending on your requirements.

Also, all of these configurations have some notion of high availability, no single node failure will kill the infrastructure. Way more options are available if you sacrifice High Availability (but that’s pretty unusual in the Enterprise isn’t it? J )

This configuration has the minimal number of servers involved but it does present some complexity problems because the BizTalk hosts must be clustered if you are running the ENTSSO service as a clustered service. You either cluster the hosts or live with having an un-clustered Enterprise Single Sign- on Master Secret Server. Either option could be best depending on your requirements.

This configuration is a pretty good starting point for the cost-sensitive enterprise. This allows separation of BizTalk from SQL (a good thing) while maintaining redundancy by having two BizTalk nodes that can “cover for each other” in case one of them goes down. The ENTSSO MSS can be installed as a clustered service on the SQL cluster without creating complications for the BizTalk hosts (i.e. they don’t have to be clustered, which is also a good thing in many cases)

This is probably my favorite starting point. It is the lowest cost, and scaling out with more BizTalk servers is really easy. If you’re not sure which one to pick, take this one.

This configuration is an improvement on the 2+2 configuration because it really allows the orchestrations and business logic to become intensive without having much of an impact on the Send/Receive operations (even though the Send/Receive operations can still interfere with each other). This is for enterprise applications with moderate Send/Receive activity and fairly intensive business logic.

Of the configurations being considered in this article, this is the most capable (and most expensive). This allows full separation between BizTalk and SQL, and also between the Send/Receive/Process operations within BizTalk. This configuration is for Enterprises with intensive Send/Receive/Process requirements.

Notice the additional SQL server, a pretty good rule of thumb is a ratio of 3:1 for BizTalk:SQL servers. Although this one is 3:1 with only 2 SQL servers, a failure of one node would change that ratio to 6:1. Seeing as how this configuration is for high volume scenarios, that would be bad. Invest a couple of bucks and have a passive node in the wings to keep things running if a SQL node goes down!

Comparison Chart

	0+2	2+2	4+2	6+3
High Availability	Yes	Yes	Yes	Yes
Performance	Lowest	Medium	Medium	Best
Cost (# Servers)	Lowest	Medium	Medium	Highest
Ease of clustering SSO MSS	Hard	Easy	Easy	Easy
Separation of SQL and BizTalk	No	Yes	Yes	Yes
Dedicated Receive Nodes	No	No	No	Yes
Dedicated Send Nodes	No	No	No	Yes
Dedicated Process Nodes	No	No	Yes	Yes
Easy to Scale out	Complex	Yes	Yes	Yes

Call to Action

I know what I want to write about, but this blog isn't for me! What topics do you want to see in the next article? Drop in a request in the comments section below and I'll review before I start writing the next article (I promise)

Configuration
SAN/Storage
Troubleshooting/Debugging
High Availability
other....

BizTalk gets "General Network Error" under heavy load

neilth — Fri, 06 Jul 2007 00:53:00 GMT

This is a fairly common scenario after Windows 2003 SP1 (or greater) is installed, so I thought I better put in my two cents

Scenario: BizTalk seems to lose connectivity to the MsgBox (and all other databases on that SQL instance)

Problem: The operating system (w2k3 sp1) running the SQL server instance is misinterpreting the heavy TCPIP connection traffic from the BizTalk App server as a Denial of Service (DOS) attack.

Clarification: This is an operating system level issue. This is not BizTalk or SQL per se (though they are common victims of this security feature). This is a security feature for the OS that affects how the OS interprets large amounts of incoming connections from individual clients.

Factors:

1. Windows Server 2003 SP1 (or higher) is installed on the SQL server hosting the BizTalk DBs

2. SynAttackProtect is still set to “1” on the SQL server

3. Concurrent TCPIP connection requests from a specific BizTalk app server exceeds the critical threshold (Sorry, don’t know what that threshold is!)

Diagnosing:

Look for the following event entries:

Event Type: Error
Event Source: BizTalk Server 2004
Event Category: (1)
Event ID: 6913
Computer: %ServerNameHere%

Description: SQLServer, BizTalkMsgBoxDb, [DBNETLIB][ConnectionWrite (send()).]General network error. Check your network documentation..

Event Type: Warning
Event Source: BizTalk Server 2004
Event Category: (1)
Event ID: 5410
Computer: %ServerNameHere%

Description: An error has occurred that requires the BizTalk service to terminate. The most common causes are an unexpected out of memory error and an inability to connect or a loss of connectivity to one of the BizTalk databases. The service will shutdown and auto-restart in 1 minute. If the problematic database remains unavailable, this cycle will repeat.

Error message: [DBNETLIB][ConnectionWrite (send()).]General network error. Check your network documentation.

Error source:
BizTalk host name: BizTalkHost
BizTalk Windows service name: BTSSvc$BizTalkHost.

Fixes:

1. Set the SynAttackProtect registry setting as described in http://support.microsoft.com/kb/899599

2. ALSO take a look at the Group policy for that SQL server to be sure that the Group Policy does not periodically UNDO the setting change that you did in step 1. I have been in the situation where I fixed the issue, but then the GP Update or reboot undid the fix! (Put the server in a separate OU if you have to, but make sure you isolate the box from policies that would reset this registry key)

Limitations

1. I don’t know how to disable this feature for a specific set of clients (i.e. SynAttackProtect is off for the BizTalk app servers but remains on for other servers). The setting seems to be at the protocol level so I wouldn’t even know where to start looking for network set definition here.

Is there something that I can do to make this blog more helpful to you? I welcome your feedback!

Excessive Blocking when using BizTalk SQL Adapter

neilth — Wed, 20 Jun 2007 06:00:00 GMT

-------------------------------------
Addendum – June 22 2007: Under absolutely NO circumstances should you modify the data model of the BizTalk databases in any way. You should only optimize your own application databases.
-------------------------------------

I have seen this issue in many places, which still surprises me because there are a lot of blog entries about this. For what it's worth, here's one more.

Scenario: BizTalk solution makes moderate to heavy use of SQL adapter in calling stored procedures in a database.

Problem: SQL server shows many blocking (perhaps deadlocking) SPIDs, most of which are owned by the BizTalk host instance account.

Factors:

The longer the procedure takes to execute, the worse off you are

Big tables, poor logic, bad indexes, bad db maintenance, disk contention etc.

The more concurrent procedure executions the worse off you are

short polling interval
multiple ports
multiple host instances

Diagnosing:

Enterprise Manager or Management Studio can give some insight to active and blocking spids (check to see who owns the blocking spid and use dbcc inputbuffer to see the sql it is executing)
Try using sp_who,sp_who2,and sp_who3, they may give you lots of insight
Call MS PSS (or download pssdiag for yourself from the MS download site) and have them configure a pssdiag trace for you. Look especially closely at the blocker output and the profiler trace. Use rml.exe (read80trace) to generate a report of the "worst offenders" sql scripts or procedures.

Fixes:

As per documentation, don't use T-SQL transactions in your proc. The SQL Adapter already has this all wrapped up in a distributed transaction so relax!
As per documentation, you can try setting the transaction isolation level to READ COMMITTED, or perhaps REPEATABLE READ. If you are running SERIALIZABLE that is a great way to tank performance and cause blocking.
Speed up the procs (Archive, purge, index, create views, run db maintenance, buy a big SAN, whatever. The locks won't kill you if they don't last)
Don't go silly with coarse locking hints. Keep your lock few in number and small in granuarity. If you must manually lock, then please read up on the READPAST hint - especially for queue or "trigger" tables. (Thanks to Kunjal K for this hint)
Reduce concurrency - You can try lengthening your polling interval or reducing the number of host instances that run that port.

Didn't find what you were looking for? Try one of these:

Is there something that I can do to make this blog more helpful to you?
I welcome your feedback!