Tiho's blog

The infamous BAM EventBus Service event ID 25

2006-12-21T22:19:00Z

There are 8 tables in each BizTalk Message Box database which store the tracking data before it is imported by the BAM EventBus Service (a.k.a. TDDS or Tracking Data Decoding Service) into the DTA/HAT and the BAM Primary Import databases. The tables are named TrackingData_x_y where 0 ≤ x ≤ 1 and 0 ≤ y ≤ 3. It is absolutely critical that the sizes of these tables do not increase over time. If the sizes are increasing than either TDDS is not able to catch up with the load or it is not working properly. There is a performance counter that can be used to monitor the size of the tracking data – BizTalk:Messsage Box:General Counters\Tracking Data Size.

The first step to troubleshoot issues with the tracking data is to examine the Windows Event Log on the BizTalk Tracking Host machine for errors from the BAM EventBus Service. Error information along with the serialized original tracking data might be available in the TDDS_FailedTrackingData tables in the DTA/HAT and the BAM Primary Import databases. Another place where error information might be available is the TDDS_Heartbeats table in the BizTalk Management database.

In this post I will focus on one very common cause which manifests as an error from the BAM EventBus Service in the Windows Event Log with event ID 25. The error should look like this:

Event Type: Error
Event Source: BAM EventBus Service
Event Category: None
Event ID: 25
Date: 12/20/2006
Time: 10:38:27 AM
User: N/A
Computer: TRACKINGHOST
Description:
Either another TDDS is processing the same data or there is an orphaned 
session in SQL server holding TDDS lock.Timeout expired. The timeout 
period elapsed prior to completion of the operation or the server is not 
responding. SQLServer: DBSERVER, Database: BAMPrimaryImport.

This error will usually show up every 5 minutes or so. Below are the steps to resolve it.

First, make sure that you are not hitting any of the issues described in this article. If this is the case, then killing the orphaned sessions or restarting the SQL Server machine will resolve the issue. You can use the information in this article to identify orphaned sessions.

If the issue is still there, then you most certainly have permissions problems. The account under which TDDS (and the BizTalk Tracking Host) is running must have execute permissions for the following stored procedures in the BizTalk Message Box database: TDDS_RedisterTDDSAccess and TDDS_GetNumTrackingPartitions. The same account must also have execute permissions for the TDDS_Lock stored procedure in the DTA/HAT and the BAM Primary Import databases.

The permissions are set correctly when BizTalk Server is first installed but might be altered later directly or indirectly by manually setting explicit deny permissions for the specific account or any group that it is a member of.

The best way to check that permissions are indeed the problem is to open Task Manager on the Tracking Host machine and on the Processes tab check the account under which the BTSNTSvc.exe process is running. This is the Tracking Host account. Now start SQL Query Analyzer or SQL Management Studio (or any of the command line counterparts) with the same user credentials as BTSNTSvc.exe. You can use the “runas” command or right-click on a shortcut and choose “Run as…”. I suggest that you do this test on the Tracking Host machine to rule out any network connectivity issues.

After you have connected to the correct SQL database execute the following:

In all BizTalk Message Box databases:

DECLARE @RC int
DECLARE @retVal int
EXEC @RC = [BizTalkMsgBoxDb].[dbo].[TDDS_RegisterTDDSAccess] @retVal OUTPUT 
SELECT @RC
GO

DECLARE @RC int
DECLARE @nPartitions tinyint
EXEC @RC = [BizTalkMsgBoxDb].[dbo].[TDDS_GetNumTrackingPartitions] @nPartitions OUTPUT 
SELECT @RC
GO

In the BizTalk DTA and BAM Primary Import databases:

DECLARE @RC int
DECLARE @resource nvarchar(128)
DECLARE @milisecTimeout int
DECLARE @retVal int
SELECT @resource = N'Foo'
SELECT @milisecTimeout = 5000
EXEC @RC = [BAMPrimaryImport].[dbo].[TDDS_Lock] @resource, @milisecTimeout, @retVal OUTPUT 
SELECT @retVal
SELECT @RC
GO

The results of these queries should be 0 and there should not be any errors. If you get “EXECUTE permission denied” errors then grant the corresponding execute permissions until you can execute the queries without any errors. At this point TDDS should start moving the tracking data from the Message Box databases to the DTA and the BAM databases.

BizTalk Server 2004 Service Pack 2 has been released

2006-10-30T23:53:00Z

Download it from here: http://www.microsoft.com/downloads/details.aspx?FamilyId=D20B4510-E5A6-4D7B-87A1-4BD52BDD57B8&displaylang=en

Image File Execution Options or how to start the debugger automatically

2006-09-09T10:36:00Z

I'm putting this here so that I don't forget where to look for it later when I need it as it has already happened in the past. Mike has a great post which is full of links to other great posts on the topic.

How do I get data into BAM Archive?

2006-09-02T03:36:00Z

"... It seems that the DTS/SSIS package BAM_DM_<activity name> should be moving it into the Instance and Relationship tables in the BAM Archive database but it doesn’t."

The BAM Data Maintenance DTS/SSIS package will only move data that is out of the activity online window. This is a setting on the BAM activities which can be changed with bm.exe. More specifically, look at the get-activitywindow, set-activitywindow commands of bm.exe.

Also if you have an OLAP aggregation defined in you observation model (BAM definition), then the Data Maintenance DTS/SSIS package will not move out of the BAM Primary Import database any data that is not yet processed in the OLAP cube by running the BAM Cube Update DTS/SSIS package - BAM_AN_<cube name>. So you need to run the Cube Update DTS/SSIS package before running the Data Maintenance DTS/SSIS package.

More information on this here.

Developing with BAM and BAM FAQ are now available

2006-08-23T02:47:00Z

I meant to post this earier so you might have already seen it but here it is.

A short FAQ covering common questions and a document on using the BAM APIs.

How is BAM Archive database used?

2006-08-23T02:24:00Z

I get different flavors of this question a lot so I’ll put some Q&A here.

Q. Is there a way to view the data stored in the BAM Archive database in the BAM Portal?
A. The short answer is NO.
The BAM Archive database is the final destination for the BAM data. Once the online data is archived, it is no longer useful for BAM with only one exception that I know of – the upgrade of the BAM OLAP cubes from SQL 2000 to SQL 2005. Neither the BAM Portal nor any other BAM component will use data from the Archiving database. Also, querying the data there is not a good idea because (1) you don’t have the computed columns and the real-time aggregations and (2) you don’t have indexes. The expectation is that all “queriable” data will be in the activity online window and thus in the BAM Primary Import database.

Q. However, can I use the data stored in the BAM Archive database from my own application?
A. By all means. This is an article which shows how to create a partitioned view for all the data for a single BAM activity in the BAM Archive database.

bts_CleanupMsgbox doesn’t work well with TDDS

2006-08-22T00:31:00Z

DISCLAIMER --
The bts_CleanupMsgbox stored procedure IS NOT SUPPORTED. IT IS NOT TO BE USED ON PRODUCTION SYSTEMS!!! The steps to workaround the bts_CleanupMsgbox issue outlined below ARE NOT TO BE USED ON PRODUCTION SYSTEMS!!! You can use bts_CleanupMsgbox and the steps in this posting in a test environment at your own risk. This posting is provided "AS IS" with no warranties, and confers no rights.

If you don’t know about bts_CleanupMsgbox or you don’t use it then you don’t need to worry about anything in the rest of this post.

If you want to learn how to use bts_CleanupMsgbox in test environment go here.

If you use the bts_CleanupMsgbox stored procedure on you BizTalk Message Box databases then be advised that it doesn’t work well with TDDS. You might loose significant amount of BAM and/or HAT data. bts_CleanupMsgbox will be fixed soon but until then here’s what you can do to avoid issues.

Procedure to use bts_CleanupMsgbox safely with regard to TDDS:
1. Read the DISCLAIMER at the beginning of this post. Never use these steps on a production system.
2. Stop all tracking hosts (BizTalk processes hosting TDDS).
3. Run bts_CleanupMsgbox.
4. Truncate the TDDS_StreamStatus tables in the BAMPrimaryImport database and the BizTalkDTADb by executing the following statement:
truncate table [TDDS_StreamStatus]
5. Restart all tracking hosts.

This posting is provided "AS IS" with no warranties, and confers no rights.

BAM Web Services and SQL Server mixed mode authentication

2006-06-14T04:56:00Z

This is an issue I see every once in while so let's see what causes it and how to diagnose and fix it.

You have created and deployed a BAM definition with some BAM views in it but when you hit the BAM Portal with IE all you see in the "My Views" pane on the left is an error message "Views or Activites may be missing because one or more database(s) could not be contacted." followed by "No view to display" message. Now what?

Your first step to troubleshooting such issues is the Windows Event Log on the machine hosting the IIS server running the BAM Portal and the BAM web services. The BAM event log entries will be in the Application event log. You can filter on event source equal to "BAM Portal" or "BAM Web Service". I will usually look at the last few errors from "BAM Web Service".

So you open the Windows Event Viewer and you see one error from "BAM Web Service" with event ID 12033. Double-click on it and the description says:

Referenced database 'BAMPrimaryImport' on server 'SERVERNAME' is not accessible. The error is:

System.NullReferenceException: Object reference not set to an instance of an object.
at Microsoft.BizTalk.Bam.WebServices.AsyncResult.End(AsyncResult asyncResult)
at Microsoft.BizTalk.Bam.WebServices.Management.GetViewSummaryAsyncResult.End(IAsyncResult result)
at Microsoft.BizTalk.Bam.WebServices.Management.BamManagementService.EndGetViewSummaryForDatabase(IAsyncResult result)
at Microsoft.BizTalk.Bam.WebServices.Management.BamManagementService.GetViewSummaryForCurrentUser().

Now what?

The second step to troubleshooting BAM Web Services issues is to... enable tracing. To turn on tracing go to the "Program Files\Microsoft BizTalk Server 2006\BAMPortal\BAMManagementService" directory (or "...\BAMQueryService") and open the web.config file with notepad. Uncomment the <system.diagnostics> element towards the end of the file. By default the traces will be saved in "C:\temp\BamManagementServiceTrace.log" or "C:\temp\BamQueryServiceTrace.log". Make sure that the IIS worker process user account (usually Network Service or ASPNET) and the BAM web service user have write permissions for the trace log file directory (C:\temp by default).

Turn on tracing for the BAM Management Web Service and refresh the BAM Portal home page in IE. Now open the trace file "C:\temp\BamManagementServiceTrace.log" and search for the string "Error" from the top of the file to find the first error. In this case the first error is:

<BizTalk2006> 06/13/2006 17:00:36 6 Microsoft.BizTalk.Bam.WebServices.TraceHelper ThrowHelperSoapException Error 11007 Cannot determine the SID for the user.

Bingo! This is the root cause of the problem.

Now some history behind this problem. The BAM Web Services authenticate all users using Windows authentication and authorize them based on their Windows identity. But there is a special case - DBO. DBO is a special account which has permissions to all BAM views. So the BAM Web Services will try to find the Windows identity for the DBO user for the BAM Primary Import database and they will fail if this is a SQL account.

BAM does not support anything other than Windows authentication for SQL Server. If everything else seems to work, there is no guarantee that it actually does and it will continue to work in the future. True, the error is not intuitive but it is correct.

How to resolve the problem now that we have identified it? It is really easy indeed. Just change the DBO accounts for all BAM databases to Windows accounts. You can do this using the sp_changedbowner system stored procedure.

To set DBO to the "domain\user" account run:

use BAMPrimaryImport
go
sp_changedbowner 'domain\user'
go

Navigate to the BAM Portal and verify that you see all BAM views and no errors in the left pane on the home page.

There is a pretty nice BizTalk Installation Guide, which you can use the next time you setup BizTalk Server to avoid such issues.

Check out the new BizTalk Server Developer Center and TechCenter

2006-05-02T22:00:00Z

You can find some orchestrations code samples and FAQ at the BizTalk Developer Center.

And the cool virtual lab at the BizTalk TechCenter. Learn what's new in BAM in BizTalk Server 2006.

Drop all indexes from a SQL table

2006-01-17T02:16:00Z

After spending some time to find out that there is no easy way to drop all indexes from a SQL table I came up with this script.

DECLARE @indexName NVARCHAR(128)
DECLARE @dropIndexSql NVARCHAR(4000)
DECLARE tableIndexes CURSOR FOR
SELECT name FROM sysindexes
WHERE id = OBJECT_ID(N'tableName') AND 
  indid > 0 AND indid < 255 AND
  INDEXPROPERTY(id, name, 'IsStatistics') = 0
ORDER BY indid DESC
OPEN tableIndexes
FETCH NEXT FROM tableIndexes INTO @indexName
WHILE @@fetch_status = 0
BEGIN
  SET @dropIndexSql = N'DROP INDEX tableName.' + @indexName
  EXEC sp_executesql @dropIndexSql
  FETCH NEXT FROM tableIndexes INTO @indexName
END
CLOSE tableIndexes
DEALLOCATE tableIndexes

Use it at your own discretion. It should work fine on both SQL Server 2000 and SQL Server 2005.

Update on September 1st, 2006:

Thanks to Anthony Jones who pointed out that the script should drop the indexes in decscending order of indid. If the table has a clustered index it will be indid 1 which if dropped first will cause all other indexes to be rebuilt only to be subsequently dropped. I updated the script. Enjoy!

BizTalk Server Beta1 is alive

2005-08-04T05:11:00Z

As some of you already know Microsoft BizTalk Server 2006 has been available for some time now. Visit http://www.microsoft.com/biztalk/evaluation/bts2006beta.mspx for overview of the changes and instructions on how to get the beta.

And don't forget to post all you questions and suggestions to the BizTalk Server Beta newsgroups (microsoft.beta.biztalk). This is very valuable feedback for us.

More on BAM and BizTalk Server 2006

2005-06-16T07:35:00Z

Gilles started a series on the new BAM Portal. Scott Woodgate is presenting the new features and improvements in BizTalk Server 2006. See what he has to say about BAM.

New BAM Management Utility (bm.exe) Commands

2005-06-16T06:52:00Z

I’m really exited about all the great new features that we’ve put in BAM in BizTalk Server 2006 but I figured out that we’ve made some changes to the bm.exe commands and you need to get familiar with those changes before everything else.

The information below is based on the Community Technical Preview (CTP) build of BizTalk Server 2006 that was distributed to TechEd attendees at TechEd USA 2005 in Orlando. Some things might change before we ship the betas and the product. If you haven’t got a copy of the CTP build yet, don’t stop reading but get ready to try the new features soon when we ship Beta 1.

To see the list of all commands just run bm.exe without and parameters. To get detailed help for a specific command run:

bm.exe help -Command:<command>

For example:

bm.exe help -Command:deploy-all

Note that there is no space between “-Command:” and “deploy-all”. And don’t worry – if you get any command wrong, you will see an error message along with the detailed help for that command and a few examples to help you fix your input.

Here is a table with the bm.exe commands in BizTalk Server 2004 and their equivalents in BizTalk Server 2006.

BizTalk Server 2004	BizTalk Server 2006	Example
bm.exe dbsetup	bm.exe setup-databases	bm.exe setup-databases -ConfigFile:BamConfiguration.xml
bm.exe deploy	bm.exe deploy-all	bm.exe deploy-all -DefinitionFile:SalesManagerView.xls
bm.exe undeploy all	bm.exe remove-all	bm.exe remove-all -DefinitionFile:SalesManagerView.xls
bm.exe undeploy view	bm.exe remove-view	bm.exe remove-view -Name:SalesManagerView
bm.exe undeploy #1	bm.exe remove-view bm.exe remove-activity	bm.exe remove-view -Name:SalesManagerView bm.exe remove-activity -Name:PurchaseOrder
bm.exe add	bm.exe add-account	bm.exe add-account -AccountName:domain\user -View:SalesManagerView
bm.exe remove	bm.exe remove-account	bm.exe remove-account -AccountName:domain\user -View:SalesManagerView
bm.exe list	bm.exe get-accounts	bm.exe get-accounts -View:SalesManagerView
bm.exe listchanges	bm.exe get-changes	bm.exe get-changes

Note that the BAM configuration file is now passed only to the setup-databases command. You can use the “-Server:” and “-Database:” parameters with almost all commands to specify BAM Primary Import database to be used for the command or you can specify the default BAM Primary Import sever and database in the bm.exe configuration file (bm.exe.config). If you don’t specify either of these, the defaults will be used – server locahost and database BAMPrimaryImport.

For example:

bm.exe deploy-all -DefinitionFile:SalesManagerView.xls -Server:SalesServer -Database:BamSalesPI

The new remove-all command will remove all activities and views defined in the BAM definition file. You can use the remove-view or remove-activity commands to remove a single view or activity.

The new deploy-all command will create BAM live-data Excel workbook in the same directory in which the Excel file with the activity and view definitions resides.

One thing I really like is that now you don’t need to export the BAM definition XML from the BAM Excel spreadsheet to use TPE. You can simply deploy your BAM Excel spreadsheet using bm.exe and TPE will get the activities definitions directly from the BAM Primary Import database. Then you can map the activities checkpoints to BizTalk orchestrations and start tracking your data.

I hope this overview will get you started quickly. Next I will write about the new update commands as well as the alerting commands in bm.exe and the distributed BAM deployment. Sounds exciting?

BizTalk Server 2006 Community Technology Preview on TechEd 2005

2005-06-10T02:49:00Z

If you enjoy being a part of this year's Microsoft TechEd and if you work in the Information Worker space make sure you visit the Business Activity Monitoring (BAM) hands on lab for a glimpse on some of the new features which will be part of Biz Talk Server 2006. Or even better - grab a CD with the CTP bits of Biz Talk Server 2006 and try it yourself. I will soon start posing about some new exiting features in BAM in Biz Talk Server 2006, so stay tuned!

An early look at Indigo

2005-02-11T06:12:00Z

Here is a good summary on Indigo.