Derek Tan's Blog

Modifying Length of a Text Data Item of an Activity

2006-01-19T23:46:00Z

Modifying the size of a nvarchar column of a data item of an activity

Applies to: Microsoft BizTalk Server 2004

You have already deployed a BAM definition with a text data item but now you want to change the maximum length of data item. The recommended way would be to remove the bam deployment and re-deploy the BAM definition with the new length for the data item. However, it is not always possible as you may already have data in the BAM tables and cannot lose the data.

This blog will detail how you may modify the length of the text data item without having to remove and re-deploy. Note that this is unsupported by Microsoft.

1. Modify the columns of tables in Bam Primary Import database

bam_<Activity Name>_Active
bam_<Activity Name>_Completed

ALTER TABLE <Table Name>

ALTER COLUMN <Data Item Name> NVARCHAR(<New Length>)

bam_<View Name>_<Activity Name>View_ActiveInstancesSnapshot if a view has been deployed

ALTER TABLE <Table Name>

ALTER COLUMN <Alias Name> NVARCHAR(<New Length>)

2. Modify the stored procedure in Bam Primary Import database

bam_<Activity Name>_PrimaryImport

Modify the stored procedure input parameter for the nvarchar data item to its new length.

3. Modify the columns of tables in Bam Archive database

bam_<Activity Name>_Instances

ALTER TABLE <Table Name>

ALTER COLUMN <Data Item Name> NVARCHAR(<New Length>)

4. Modify the columns of tables in Bam Star Schema database

bam_<View Name>_<Activity Name>View _Staging

ALTER TABLE <Table Name>

ALTER COLUMN <Alias Name> NVARCHAR(<New Length>)

bam_<Cube Name>_Dim_<Data Dimension>

ALTER TABLE <Table Name>

ALTER COLUMN <Data Dimension Member Name> NVARCHAR(<New Length>)

Moving Time Window of BAM Aggregated Alerts

2006-01-10T00:53:00Z

In BizTalk 2006, you can define alerts based on real-time aggregations or scheduled aggregations based on analysis cubes. For aggregated alerts, you have the option to specify a time window. This binds the alerts to be aggregated within a time past to the present.

For example: Say there is a purchase order activity, time dimension “Received Time” that has Year, Month and Day as its members. There is also a measure Total Sales. Let’s say you want to define such that if the Total Sales of all purchase orders received for the past 7 days are greater than $1000, you receive an alert. You would specify “Received Time” and 7 days as the time window, measure as “Total Sales”, the threshold at $1000 and threshold operator as “>”. Note that the time window is always relative to the present time.

Note: Real-time aggregated alerts without a time window only makes sense when the alert has a progress dimension member and the instances that are aggregated are not completed. This is because only a slice of data is available in the real-time aggregated tables and the duration which the slice spans is dependent on the real-time aggregation window. For example, it does not make sense to specify a real-time aggregated alert that queries total sales of purchase orders that are shipped (i.e. purchase orders are completed). One has to bound it within a time window: Total sales of purchase orders that are shipped for the past 7 days. Queries like “total sales of purchase orders that are in fulfilling state” is valid because incomplete instances are not discounted from the real-time aggregation.

BAM integration with Microsoft SQL Server Reporting Services 2005

2006-01-10T00:39:00Z

The steps to create a report based on data in Bam infrastructure is no different from what one would do for other SQL data sources. It is recommended that you read the “Creating a Report with Report Designer” section in SQL Server 2005 Books Online on how to create a report using the Microsoft SQL Server Report Designer.

Report Wizard

“Select a data source” page

Bam provides two data sources that Reporting Services may point to:

1. Bam Primary Import database for views on activity instances or real-time aggregations. Make sure to select Type=”Microsoft SQL Server” and Connection String=”Data Source=<server name>; Initial Catalog=<database name>”, where <server name> and <database name> are the server and database names of your Bam Primary Import database.

2. Bam Analysis database for querying the analysis cube. Make sure to select Type=”Microsoft SQL Server Analysis Server” and Connection String=”Data Source=<server name>; Initial Catalog=<database name>”, where <server name> and <database name> are the server and database names of your Bam Analysis database.

“Design the Query” page

For the Bam Primary Import database, there are two types of views:

1. dbo.bam_<view name>_<activity name>View_View. This is the view for instance data.

2. dbo.bam_<view name>_<real time aggregation pivot table name>_RTAView. This is the view for the real-time aggregations.

You can type in “select * from view” to return the desired result set.

For the Bam Analysis database, click on the query builder and drag and drop the dimensions and measures of the cube named <view name> to return the desired result set.

Other steps

Go through the rest of the steps to decide what and how you want the data to be presented.

What is a Progress Dimension? What are a Progress Stage and a Progress Milestone?

2006-01-09T23:29:00Z

BAM introduces a new type of dimension, the Progress Dimension. It allows aggregations of data within categories of progress status of activities. For example, the valid categories of status of a purchase order activity may be evaluating, fulfilling, shipped and denied. With this dimension, one can find out the total number or sales of purchase orders that are denied within a certain period.

In BizTalk 2006, the terms progress stage and milestone are used in place of child and sibling. A progress stage represents a transient state of an activity instance. On the other hand, once a milestone is reached for an activity instance, it would not be undone. Suppose you define for a purchase order instance and below is a representation of the report you want:

Progress Dimension			Total Sales($)
Received	Evaluating		500
	Approved	Fulfilling	200
		Shipped	700
	Denied		100

Received, Approved, Shipped and Denied are progress milestones. Evaluating and Fulfilling are progress stages.

Backup and Restore for BAM databases

2006-01-09T22:57:00Z

The following applies to BAM in BizTalk 2006. It would work similarly in BizTalk 2004, without SQL NS databases.

Deployment and Removal

To ensure consistent schema across all BAM databases, run the “Backup BizTalk Server” SQL job for each deployment of activities and/or views. In addition, backup the DTS packages, BAM analysis, star schema and archive databases.

Runtime

To ensure databases are in sync after a restore, it is necessary to back up the BAM databases in the following order:

Run SQL agent job “Backup BizTalk Server” to back up BAM Primary Import, SQL NS as well as other BizTalk databases.
Run BAM analysis and data maintenance DTS/SSIS packages for all activities and cubes respectively.
Backup BAM star schema, analysis and archive databases.

The BAM star schema, analysis and archive databases are modified when the analysis and data maintenance DTS/SSIS packages are executed. Hence, in case of failure of Primary Import, restore the backup of star schema, analysis and archive databases BEFORE the last successful backup of Primary Import.

It is possible that the archive database may fail in between the successful execution of the data maintenance DTS/SSIS package and the next backup of the database. In order to ensure that at any time a complete set of archived data can be recovered, modify the data maintenance DTS/SSIS package for each activity to insert a step to backup the Archive database before the last step “End Archiving”. The “End Archiving” step deletes the archived data from Primary Import database, updates metadata tables and views.

BAM without BizTalk Configuration (Part 3)

2006-01-06T23:58:00Z

The BAM web service needs to be able to access the SQL NS binaries in order to set up alerts. The following will set up the BAM portal so that you can define alerrts via the portal.

Grant Permissions to Microsoft SQL Notification Services bin folder

Grant the web service identity “Traverse Folder / Execute Files”, “List Folder / Read Data”, “List Folder / Read Data”, “Read Attributes”, “Read Extended Attributes” and “Read Permissions” permissions for the following bin folder:

In SQL 2000, the bin folder can be found at <SQL NS install location>\<Installed version>.
In SQL 2005, the bin folder can be found at <SQL Server install location>\90\NotificationServices\<Installed version>.

BAM without BizTalk Configuration (Part 2)

2006-01-06T22:43:00Z

The following assumes that you are using Internet Information Services (IIS) 6.0 or higher and your operating system is Microsoft Windows Server 2003.

1. Create Application Pool

a. Click Start->Programs->Administrative Tools->Internet Information Services (IIS) Manager.

b. Right mouse click on “Application Pools”->New->“Application Pools”.

c. Type in name (say, “BAMAppPool”) of application pool to create.

d. On the properties page of the application pool, choose an application pool identity. Click OK when done.

e. Make sure that the application pool identity is added to the “IIS_WPG” local group and if Sharepoint is configured, make sure it is also added to the “STS_WPG” group.

2. Create Virtual Directory

a. Using Internet Information Services (IIS) Manager, create a virutal directory named “BAM”.

b. Expand “Web Sites” node->Right mouse click on “Default Web Site”->New->Virtual Directory.

c. In the “Virtual Directory Creation Wizard”, type in the virtual directory name, choose “<install location>\BAMPortal” as the content directory and only allow “Read” permissions.

d. On the properties page of the virtual directory, in the “Virtual Directory” tab, create an application named (say) “BAM”, choose “Script only” execute permissions and the application pool “BAMAppPool” created in previous step. Click “Edit” on the Authentication and access control tab. Uncheck the “Enable anonymous access” checkbox on the “Directory Security” tab and make sure that Integrated Windows Authentication checkbox is checked. Click OK when done.

e. On the properties page of the “BAMManagementService”, create an application named (say) “BAMManagementService”, choose “Script only” execute permissions and the application pool “BAMAppPool” created in previous step. Click “Edit” on the Authentication and access control tab. Uncheck the “Enable anonymous access” checkbox on the “Directory Security” tab and make sure that Integrated Windows Authentication checkbox is checked. Click OK when done.

f. On the properties page of the “BAMQueryService”, create an application named (say) “BAMQueryService”, choose “Script only” execute permissions and the application pool “BAMAppPool” created in previous step. Click “Edit” on the Authentication and access control tab. Uncheck the “Enable anonymous access” checkbox on the “Directory Security” tab and make sure that Integrated Windows Authentication checkbox is checked. Click OK when done.

3. Setup web.config For Web Services

There are 3 web.config files to create.

a. Web.config in <install location>\BAMPortal folder

i. Make a copy of web.config.tmpl in <install location>\BAMPortal folder and renamed it to web.config.

ii. Open the web.config file, replace “localhost” with the name of your server hosting the web service for the following two lines:

iii. Set culture to “neutral” for the assembly.

iv. Replace “Everyone” with the group that you want to use for controlling access to the portal for the following line:

v. Save the web.config file.

b. Web.config in <install location>\BAMPortal\BAMManagementService folder

i. Make a copy of web.config.tmpl in <install location>\BAMPortal\ BAMManagementService folder and renamed it to web.config.

ii. Open the web.config file, replace “localhost” and “BAMPrimaryImport” with the name of your primary import server and database respectively.

iii. Replace “Everyone” with the group that you want to use for controlling access to the web service for the following line:

iv. Save the web.config file.

c. Web.config in <install location>\BAMPortal\BAMQueryService folder

i. Make a copy of web.config.tmpl in <install location>\BAMPortal\ BAMQueryService folder and renamed it to web.config.

ii. Open the web.config file, replace “localhost” and “BAMPrimaryImport” with the name of your primary import server and database respectively.

iii. Replace “Everyone” with the group that you want to use for controlling access to the web service for the following line:

iv. Save the web.config file.

4. Use ASP.Net To Encrypt Web Service Credentials

a. From the command prompt, type the following:

aspnet_setreg.exe -k:"SOFTWARE\Microsoft\BizTalk Server\3.0\BAM\WebServices\identity" -u:<yourDomainName\userName> -p:<password>

b. Press Enter.

Note For more information about aspnet_setreg.exe, see Knowledge Base article 329290, "Use the ASP.NET Utility to Encrypt Credentials and Session State Connection Strings" located at http://go.microsoft.com/fwlink/?LinkID=16728

5. Grant Permissions in Registry

WARNING: If you use Registry Editor incorrectly, you may cause serious problems that may require you to reinstall your operating system. Microsoft cannot guarantee that you can solve problems that result from using Registry Editor incorrectly. Use Registry Editor (regedit.exe) at your own risk.

a. Grant the web service identity “Query Value”, “Enumerate Subkeys”, “Notify” and “Read Control” permissions to the following registry key.

i. HKLM\SOFTWARE\Microsoft\BizTalk Server\3.0

b. Grant the application pool identity “Query Value”, “Enumerate Subkeys”, “Notify” and “Read Control” permissions to the following registry key.

i. HKLM\SOFTWARE\Microsoft\BizTalk Server\3.0\BAM\WebServices\identity\ASPNET_SETREG

6. Grant Permissions to Folders

a. Grant the application pool identity “Traverse Folder / Execute File”, “List Folder / Read Data”, “Read Attributes”, “Read Extended Attributes”, “Create File / Write Data”, “Create Folders / Append Data”, “Write Attributes”, “Write Extended Attributes” and “Read Permissions” permissions to the Windows temp folder, its sub folders and files.

7. Update Bam Configuration

a. Insert the following as a child of BAMConfiguration element in the BamConfiguration.xml, replacing “yourServer” with the server where you want to install the BAM web service.

<GlobalProperty Name="BAMVRoot">http://yourServer/BAM </GlobalProperty>

b. Run "bm.exe update-config -FileName:BamConfiguration.xml"

8. Grant Permissions in BAM Primary Import Database

a. Add the web service identity to SQL Logins.

b. Add the web service identity to the “BAM_ManagementWS” role in the BAM Primary Import Database

9. Troubleshoot

a. I get an error in the event log that says that the "BAM Portal" event source cannot be created.

i. Create a key "BAM Portal" under HKLM\SYSTEM\CurrentControlSet\Services\Eventlog\Application

ii. Under the newly created key, create a key-value pair.

Key: "EventMessageFile".

Value: <.Net Framework Path>\<Installed Version>\EventLogMessages.dll

Type: Expandable string value.

If you have alerts configured, refer to my blog "BAM without BizTalk Configuration (Part 3)" for the additional step.

BAM without BizTalk configuration (Part 1)

2006-01-05T03:18:00Z

In BizTalk 2006, if you do not want to create a BizTalk group, yet would still like to use BAM. You can create the BAM infrastructure is supported without having to configure BizTalk. The steps are as follows:

Define a Bam Configuration file. The following is an example of what it may look like, including an optional section on configuring BAM alerts which requires Microsoft SQL Notification Services to be installed. Note that CubeUpdateDTS, Bam Star Schema and Analysis databases are also optional. Replace “yourServer” with the server/s which you want to install the databases and share as well as “yourEmailAddress” with the address which you want the email alerts to be sent from.

<?xml version="1.0" encoding="UTF-8"?>

<Property Name="ServerName">yourServer</Property>

<Property Name="DatabaseName">BAMPrimaryImport</Property>

</DeploymentUnit>

<Property Name="ServerName">yourServer</Property>

<Property Name="DatabaseName">BAMStarSchema</Property>

</DeploymentUnit>

<Property Name="ServerName">yourServer</Property>

<Property Name="DatabaseName">BAMAnalysis</Property>

</DeploymentUnit>

<Property Name="ServerName">yourServer</Property>

<Property Name="DatabaseName">BAMArchive</Property>

</DeploymentUnit>

</DeploymentUnit>

</DeploymentUnit>

<Property Name="DBServer">yourServer</Property>

<Property Name="ApplicationDatabaseName">Application</Property>

<Property Name="InstanceDatabaseName">BAMAlerts</Property>

<Property Name="GeneratorServerName">yourServer</Property>

<Property Name="ProviderServerName">yourServer</Property>

<Property Name="DistributorServerName">yourServer</Property>

<Property Name="SmtpServerName">yourServer</Property>

<Property Name="AlertMailFrom">yourEmailAddress</Property>

<Property Name="FileDropUNC">\\yourServer\alerts</Property>

</DeploymentUnit>

Run "bm.exe setup-databases -ConfigFile:BamConfiguration.xml -NSUser:myDomain\myNSUser" When prompted, enter the password for the user. This will setup the databases as well as the SQL NS services for BAM alerts.

You can then instrument your application and use Direct or Buffered EventStream API to monitor your business. Refer to the programmer's reference for use of this API:

http://www.msdn.microsoft.com/library/default.asp?url=/library/en-us/sdk/htm/frlrfmicrosoftbiztalkbameventobservation.asp

Note that the BAM portal is not yet set up with the above steps. Refer to my blog "BAM without BizTalk configuration (Part 2)" on how to set up the BAM portal.

Addition of BAM Analysis after initial configuration

2006-01-05T02:38:00Z

In BizTalk 2004, in order to add BAM Analysis after an initial configuration without, one can use a command in bm.exe to create the BAM Analysis as well as the BAM Star Schema databases. The star schema database acts us a staging database for the building of the BAM cube. The steps are:

Modify the BamConfiguration.xml in the \Microsoft BizTalk 2004\Tracking folder. Add the following and type in the server and database names for the star schema and analysis databases.

<DeploymentUnit Name="StarSchemaDatabase">
   <Property Name="ServerName"></Property>
   <Property Name="DatabaseName"></Property>
</DeploymentUnit>
<DeploymentUnit Name="AnalysisDatabase">
   <Property Name="ServerName"></Property>
   <Property Name="DatabaseName"></Property>
</DeploymentUnit>

2. Run "bm dbsetup BamConfiguration.xml" from the same folder.

In BizTalk 2006, re-run the BizTalk Server Configuration (Start Menu->Programs->Microsoft BizTalk Server 2006->BizTalk Server Configuration)

Real-time Aggregation Online Window

2006-01-05T01:51:00Z

In BAM, instead of waiting for the execution of data analysis task to update the cube aggregations at scheduled intervals, one can define specific slices of data to be aggregated in real-time. For example, in a departmental store scenario, a store manager may want to know what the sales of a promotional item is. Typically, he may have to wait for the report on the next day because the cube aggregations are updated daily at midnight. With the real-time aggregation feature in BAM, he may query the real-time views via Excel pivot tables, to know the up-to-date (to the minute) sales of the promotional item for today. In BizTalk 2006, the user may also query via the BAM portal and even set alerts based on a threshold.

BAM updates the real-time aggregation tables as and when the data arrives at the BAM infrastructure. To maintain this table, an online window is defined. This online window defines how much aggregate data is to be kept in the tables. Note that if the online window is set to (say) 1 day, only data a day earlier till now would be aggregated. Typically, you would want to set the online window slightly larger than the time span the queries are based on. In BizTalk 2004, one can pre-defined the online window in the BamConfiguration.xml and this will be applied when the BAM infrastructure is created. If this online window for the real-time aggregation needs to be changed, the corresponding row in the bam_metadata_RealTimeAggregations table can then be modified. In BizTalk 2006, the option SetRTAWindow in bm.exe should be used instead of modifying the table directly and the default online window is 3600 minutes which is the same as in BizTalk 2004.

More information can be found at: http://www.msdn.microsoft.com/library/default.asp?url=/library/en-us/sdk/htm/ebiz_prog_bam2_qkng.asp

How to represent date time data in Bam in local time

2005-06-11T03:38:00Z

If you are using the tracking profile editor to map orchestration shapes/events to BAM activity, the date time value that appears in views and cubes will be in UTC (Coordinated Universal Time) time. With a little tweaking of a stored procedure for each activity, you can display the date time value in your local time.

The stored procedure to modify is: bam_<Activity Name>_PrimaryImport. Add the following couple of lines in this stored procedure:

DECLARE @TimeDifference int
SELECT @TimeDifference = ROUND(DATEDIFF(mi, GetUTCDate(), GetDate()), -1)

For each variable of datetime type passed into this stored procedure, add the following:

SELECT @yourVariable = DATEADD(mi, @TimeDifference, @yourVariable)

That's it!

I want to thank Seong Joon Kwak for asking the question and going through a couple of solutions before I arrive at this simple solution.

Continuation vs Relationship - When to use?

2005-03-03T00:17:00Z

If an orchestration calls another orchestration multiple times, use relationship to model the process. Examples where this apply may include

A purchase order processing with each line item processed by another orchestration.
An exception handling orchestration that is invoked anywhere in the project where an exception may occur

To illustrate, suppose orchestration (Orch1) calls orchestration (Orch2) n times, where n is determined at runtime. To model this, create two activities (Act1) and (Act2) in the same Bam definition file which have data items and business milestones corresponding to Orch1 and Orch2 respectively. Using the Tracking Profile Editor (TPE), map the data items and milestones as usual. In addition, follow the steps at http://www.msdn.microsoft.com/library/default.asp?url=/library/en-us/sdk/htm/ebiz_prog_tracking_hgdm.asp to map the relationship. At runtime, the bam_Act1_AllRelationships view contains the mapping between instances of Act1 and Act2. Using this view and the bam_<ViewName>_<ActivityName>_View views for both Act1 and Act2, you can create queries to give information that you need for the entire process.

Another situation where modelling relationship may be more natural is when there are two different activities, like purchase order processing and shipment processing, and a single view is not what the user is after. At the same time, the user wants to relate these two activities in a way so that one can go from a purchase order number to find the corresponding shipment record.

If there are multiple orchestrations but each orchestration is called only once, use continuation to model. The view that comes with the deployment of Bam spans the entire project. Note that in this case only one activity is needed. Other situations where continuation may be used is when events are sent from different artifacts, like pipelines and customized code. The Bam End-To-End (BizTalk sample) in the SDK shows how continuation is used to monitor events from pipelines across to orchestrations: http://www.msdn.microsoft.com/library/default.asp?url=/library/en-us/sdk/htm/ebiz_sdk_samples_bam_vgrs.asp

Introduction

2005-02-28T21:40:00Z

Thanks for coming to this blog. I've been active in answering newsgroup questions on Microsoft BizTalk Server; my specialty is in the Business Activity Monitoring (BAM) technology. After some time, I find myself repeating the answers/solutions to similar questions posted by different people. It may be very well that customers and partners are finding it hard to find the answers to their questions in the newsgroup. So my primary intention is to provide a location to gather up these questions and reply in a succinct way. Also, this forum allows me to explain in greater details aspects of the BAM technology than is usually possible in replying to questions in newsgroups. Please note that any solutions included here do not in any way imply that they are officially supported by Microsoft.

Usual disclaimer: The postings and any code are provided "AS IS" with no warranties.