BizTalk 09 and SQL R2 not Supported\BizTalk 09 and Project References

In a recent case the customer was attempting to install BizTalk 09 with SQL 2008 R2. It fails with the following error.

"Failed to create Management database 'BizTalkMgmtDB' on server 'XXXXX'. Error '0x80040E1d' occurred."

The server name is represented by the "XXXXX".

No follow up has been done to explain the problem. The BizTalk 09 product site does not mention support for this version of SQL.

Building Projects with BizTalk 09

Beware when building projects with BizTalk 09. There is a problem with references. There are various manifestations of the problem. Artifacts from referenced projects may not be available. Messages types requiring a reference to another project may not be available. Maps requiring schemas from referenced types are not available. The hot fix is under development and should be available in the near future. This could be a bigger problem as more symptoms are discovered. One fix may not fit all.

Unable to deploy the MSI Application that contains WCF bindings to BizTalk

I have seen few customers running into this issue. This forced me to write this blog.

Background: On a development environment, my customer had a BizTalk application. Application has few WCF bindings. They created MSI file that can be used to install the application onto production machines. When they try to deploy the application to target machines, following error is encountered

Validation failed for 1 resource(s).

Invalid URL "http://localhost:90/" provided as destination location. Valid format of URL is "http://localhost[:port]/path".

Environment: BizTalk 2006, Windows 2003

Cause: During the deployment of MSI application, For WCF bindings, BizTalk would verify whether URL is in <http://localhost[:port]/path> format or not. If it is not in the specified format BizTalk would display the error similar to the above.

In this specific case WCF service is hosted directly under root directory (under localhost) and hence it is referenced by http://localhost. When BizTalk checks the URL it finds that actual URL is not represented in http://localhost[:port]/path format (because it expects to see that WCF service is hosted under a Virtual directory, but not under root directly) and it errors out.  

Resolution: You can work around this issue by following the below steps.

1) In the source machine host WCF service under virtual directory (i.e. For

example instead of hosting WCF service directly under localhost host it under localhost/WCFService)

2) Modify the receive location properties of WCF Receive port and specify the modified URI. It should be like /virtualdirectoryname/servicename.svc.

3) Export the package now

4) Make sure while exporting, under the “Specify IIS Hosts” window make sure Hostname is set to localhost and Path should contain /virtualdirectoryname (If we just have “/” here it would error out)

5) Import the MSI file in the target computer

Hope this helps!

Inserting parent child records with Identity column using WCF-SQL Adapter in one transaction

There would be scenarios where you are inserting into a Parent table which have got an auto incrementing or an Identity field and you need to get this identity value back and use it to insert into child table records. And that too all, in a single transaction.

In Native SQL Adapter, this kind of composite operation can be achieved using updategram by capturing the value generated for the Identity field using the updg:at-identity attribute. This is documented at whitepaper: Working With BizTalk Adapter for SQL Server. But when it comes to WCF-SQL Adapter (Adapter Pack 2.0), since it would not work with updategram, only way I could figure out is to write a Stored Procedure which will do all the stuff in a single transaction and you can then call it from the Send Port. Of course, you can do the same in multiple operations without calling Stored Procedure where you first write to Parent table and get the identity value back as response and then write to child table; but this would not be a single transaction and any failures will writing to child table will not rollback the insertion into Parent table by itself.

Now, when it comes to writing Stored Procedure, if you just need to insert into one parent record and one child record, it would be simpler. But what about if there are multiple child records for a single parent record? For this, we would need to create an Xml message and pass this to the Stored Procedure as an Xml parameter. And then inside the stored procedure, we need to retrieve the values from the Xml message and insert it to our Parent and Child tables. Parsing the Xml message inside Procedure will be not an easy task if the message schema is complicated (especially for EDI Schemas :) ) and will involve good knowledge of writing Xpath Queries. But this becomes really simpler with use of Table Types and Table-Valued Parameters in SQL 2008. (Thanks to Mustansir Doctor for pointing me to this new feature of SQL 2008)

Here is an example of how we can achieve the same using Table Types and Table-Valued Parameters.

Lets say we have two tables Orders & Order Details in SQL 2008 with Primary Foreign relationship  and OrderId is an auto incrementing field.

Now, first step will be to create Table Types for the above two tables:

We do not need to have OrderId field in the Table Type created as this is an incrementing field and we do not need to pass this in parameters passed to Stored Procedure.

Now, create a stored procedure to which we will pass this Table Type as parameter and inside it we will first insert into Orders table and read out the Identity value and use it to insert into OrderDetails table. Also, notice that reading out values from the Table Type objects is as simple as reading out form a Table.

Now, what all we have to do is to use Consume Adapter Service through Visual Studio to generate Schema.

It will generate two schemas TableType.dbo.xsd and TypedProcedure.dbo.xsd along with a binding file for the WCF-SQL Adapter.

TableType.dbo.xsd

 TypedProcedure.dbo.xsd

TypedProcedure.dbo.xsd contains the request and response schema for WCF-SQL Adapter.

Now, only thing remaining is to construct a message of the request schema type inside the Orchestration and send it to WCF-SQL Adapter which will execute the Stored Procedure. I am not going into it more detail on how we do it inside Orchestration as it will be similar to any other operation we do using WCF-SQL Adapter.

You can see that using the Table Types simplify the things to a large extent and we do not need to write a single line of code to parse the XML.

MSDTC issue in BTS 2009 virtualization environment

This is a repost of Ahmed Irkakene’s mail to his fellow BizTalk colleagues.

======================================================================

I have been working with a customer last time to install several BTS 2009 systems.   Customer was cloning  his initial VMware system.

As a result he is facing problem with MSDTC:  “MSDTC not configured correctly.”

MSDTC component, during its transactional operations, needs basically SID and NetBios name to be resolved.

As you imagine without MSDTC, BizTalk server can’t do much.

When cloning system, the customer should:

1. Use sysprep tool to get a new SID for each system  ( DO NOT newsid tool as it’s NOT supported )

ð As a consequence, the system will be removed from its domain.

2. Re-install properly MSDTC

Before joining the domain again, please follow the steps below to make sure the MSDTC is correctly installed.

a. MSDTC –Uninstall

b. Reboot

c. MSDTC –Install

d. Reboot

Make sure the domain replication is occurred ( or force it to occur), otherwise you may get the following error even if the system joined the domain:

The security database on the server does not have a computer account for this workstation trust relationship

Also, one more important thing ( mainly when using static IP address but still valid for dynamic ones ):

As mentioned earlier, MSDTC needs name resolution and therefore “Enable NetBIOS over TCP/IP” should be enabled on all systems (adapters network) involved in the scenario.

Troubleshooting BTARN Web Pages and more on BTARN x64

Support for BTARN often involves issues connecting with the partner. Unfortunately the web pages deployed for this purpose are not the best when it comes to troubleshooting. It can be problematic to tell a customer they need to deploy new pages with increased tracking capabilities into their production environment. This is especially true if it is 3 AM. This was generally the situation prior to BTARN 3.3.

With .Net 2.0 some cut and paste changes to the web.config file often provide the same information without the need to redeploy pages to the production environment. The configuration changes provided here should be pasted in after the closing </system.web> tag. Then make sure the logging location exists and is available for output.

<system.diagnostics>

<trace autoflush="true" />

<sources>

<source name="System.Net">

<listeners>

<add name="System.Net"/>

</listeners>

</source>

<source name="System.Net.HttpListener">

<listeners>

<add name="System.Net"/>

</listeners>

</source>

<source name="System.Net.Sockets">

<listeners>

<add name="System.Net"/>

</listeners>

</source>

<source name="System.Net.Cache">

<listeners>

<add name="System.Net"/>

</listeners>

</source>

</sources>

<sharedListeners>

<add

name="System.Net"

type="System.Diagnostics.TextWriterTraceListener"

initializeData="c:\\trace\\System.Net.trace.log"

/>

</sharedListeners>

<switches>

<add name="System.Net" value="Verbose" />

<add name="System.Net.Sockets" value="Verbose" />

<add name="System.Net.Cache" value="Verbose" />

<add name="System.Net.HttpListener" value="Verbose" />

</switches>

</system.diagnostics>

A log will be created with great deal of detail. This includes the messages being passed. The log eliminates the previous issues with same-box transfers, SSL encryption, and sometimes limited access to network resources. Armed with this information it should be straight forward to understand the problem. The same information from BTARN 3.0 would require many lines of code added to the web pages to get the same information and as mentioned before redeployment of the enhanced pages.

BTARN and Windows 08 x64

In my previous blog I published help when using x64 with BTARN. This is particularly bothersome for Windows 03 x64 because BTARN is only available as a 32 bit application. Another little discovery has been detected with Windows 08 x64.

It turns out Windows 08 uses TLS protocol 1.0 as the default when attempting connections for BTARN (any HTTP). This was not the case with Windows 03. In some cases this turns out to be a problem. Close examination of the conversation shows the "Client Hello" being responded to with an "Encryption Error" or some other server drop. When things work properly the first attempt to use the TLS protocol should be followed with a second attempt to revert back to SSL 3.0 if the server cannot use TLS 1.0. Some servers don't allow the second attempt.

The fix for this situation is to change the code in the BTARN web pages. The following line of code should be added.

//this line for compatibility with Win08

ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3;

//rest of the code

HttpWebRequest outerRequest = null;

ErrorLevel level = ErrorLevel.ProxyToOuterFailure;

This code sets a property. Location is not critical except I would keep it close to the HttpWebRequest object so it is easy to follow like listed above. This will prevent the use of the TLS protocol for this communication.

BizTalk DevCenter - check it out!

The BizTalk support team is now adding content to the BizTalk Developer Center on MSDN, especially the Support page:

http://msdn.microsoft.com/en-us/biztalk/default.aspx

Our goal is to continually update these pages with the following information:

• Common call generators, including Symptoms and potential Resolutions
• Potential scenarios and their fixes
• Contributing to the monthly theme
• Providing learning materials, including some of the same BizTalk 2009 training materials that the BizTalk support team used:

BizTalk Server Training from Microsoft Customer Service and Support
http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=459bd96f-0c52-49ac-bb78-e1868c2d3121

We encourage you to browse the BizTalk DevCenter. New information is being added on a regular basis.

Thanks!
Mandi

Error due to missing HIPPA Schema Trigger Field Annotations

Recently, I was working on a support incident where the customer have modified 837I HIPPA schema by adding a child node under DTP segment. But, while we tried to validate an instance against the schema at both design time and run time, we will get error “Object reference not set to an instance of an object”.

Below is the error thrown by EDIReceive pipeline at run time.

Event Type: Error
Event Source: BizTalk Server 2009 EDI
Event Category: None
Event ID: 4097
Description:
Fatal error encountered in EDI Disassembler, error information is Object reference not set to an instance of an object.

The issue would not happen if we were using the default 837I HIPPA schema which comes out of the box with BizTalk. Now, it means that something going wrong with the node which we have added to schema and the error is also of not much help here. Also, if we remove the data from the input instance corresponding to the node added in the schema, it would get successfully parse.  

After we did a thorough examination of schema (all thanks to Farida for this), we found that the child node added under DTP segment in the schema was missing the HIPPA Schema Trigger Field Annotations. First learn what are HIPPA Schema Trigger Field Annotations, for that click here. Here it was happening that, for DTP segment, we had other default child nodes which have got their Trigger Field Annotations specifying what their qualifier values (or trigger values) are. Now when the EDI Disassembler encounters DTP segment followed by the qualifier value, it should be able to resolve it to the corresponding XML child node under DTP segment. But when it encountered DTP segment for the child node added to the schema, it was not able to resolve the qualifier value to any node since there is no child node matching that qualifier value and hence the error.

Once we added the Trigger Field Annotations to the node added manually, it resolved the error. Note that, for adding the trigger info, we have to open the schema in a text editor.

EDI/AS2 in BizTalk 2009 : How to get the BatchId for a specific BatchName

One of the new features in BizTalk 2009 EDI is the possibility to create multiple batch configurations for each party.
Each batch will have a specific name and specific batchID and can be managed individually.

For a specific batch name you can get the corresponding batch ID. For example you may need this for an external trigger message for that batch.

If you want to do that programmatically, you may expect you can just call Partner.GetBatchIdForBatchName method by passing the batch name.
http://msdn.microsoft.com/en-us/library/microsoft.biztalk.edi.partneragreementmanager.partner.getbatchidforbatchname.aspx

But looking at the signature of that method, it is probably not what you want. Instead of expecting just a string for the batch name, this method wants two int, the partyID and the messageType. But if you just want to pass in the batch name, you do not get very far with that method.

To get the batch ID for a batch name, we have a stored procedure you can call from code: edi_GetBatchIdForBatchName .
Calling 
  exec edi_GetBatchIdForBatchName 'MyBatch'
will return the ID of the batch name MyBatch.

Note: both calling GetBatchIdForBatchName and edi_GetBatchIdForBatchName directly is not supported.

Manuel Stern

Configuring BTARN on x64 boxes

As x64 O/S's start to become the norm documenting the additional steps required to configure BTARN to work with these may be helpful. Most of this information is documented in the troubleshooting or "known issues" section of the product documentation. The most important thing to know is BTARN is 32-bit.

The information provided here addresses Windows Server 03 x64 and 08. The changes are to IIS so these would apply to other operating systems depending on the version of IIS. BizTalk provides configuration guidance to enable x64 processing. This is not compatible with BTARN. Review these steps if 404's or 500's are showing up in the application log when attempting to operate BTARN.

Web Service Extensions

Windows 03 (IIS 6.0)

Open IIS manager and check the web extensions. Make sure the BizTalk externsion is not set to the x64 version. It should be "C:\Program Files (x86)\Microsoft BizTalk Server 2006\HttpReceive". The next configuration is a couple of command lines. The first sets processing to 32-bit and the second installs .Net.

cscript %SystemDrive%\inetpub\AdminScripts\adsutil.vbs set w3svc/AppPools/Enable32bitAppOnWin64 1

C:\WINDOWS\Microsoft.NET\Framework64\v2.0.50215>aspnet_regiis.exe -i

This may not be necessary if the ASP .Net 2.0 32-bit entry is visible and enabled under "Web Extensions". If this is not displayed the above code will make the extension available. Be sure to set the value to "allowed".

Windows 08 (IIS 7.0)

Again, BTARN has to run as a 32-bit application. The bigger issue with 08 is the relocation of several configuration options. Here are the instructions to install the BizTalk ISAPI filter from the BizTalk documentation:

Click Start, point to All Programs, point to Administrative Tools, and then click Internet Information Services (IIS) Manager.

Select the root Web server entry and in the Features View, double-click Handler Mappings and then in the Actions pane, click Add Script Map.

Note - Configuring the script mapping at the Web server level will cause this mapping to apply to all child Web sites. If you wish to restrict the mapping to a specific Web site or virtual folder, select the target site or folder instead of the Web server.

In the Add Script Map dialog box, enter BtsHttpReceive.dll in the Request path field.

In the Executable field, click the ellipsis (.) button and browse to drive:\Program Files\Microsoft BizTalk Server 2009\HttpReceive. Select BtsHttpReceive.dll, and then click OK.

Enter BizTalk HTTP Receive in the Name field, and then click Request Restrictions.

In the Request Restrictions dialog box, select the Verbs tab and then select One of the following verbs. Enter POST as the verb.

On the Access tab, select Script, and then click OK.

Click OK and when prompted to allow the ISAPI extension, click Yes.

This works up to a point. Note the location would implement the x64 version. Since BTARN only works as a 32-bit use "C:\Program Files (x86)\Microsoft BizTalk Server 2009\HttpReceive" instead.

Application Pools (app pool)

Make sure to use a custom app pool for each BTARN web virtual folder. Configure them to run under the same identity as the BizTalk service. This is the same for IIS 6.0 or 7.0. Move to the individual web folders and select the app pool for each.

That's it for IIS 6.0. IIS 7.0 requires additional configuration under each app pool. Select the "Advanced" configuration option and look for "Enable 32-bit Applications" setting and change it to "True". This should get BTARN working. Use the "LoopBack" tutorial to verify the configuration.

Client Authentication (ELEMICA)

Client authentication is not required to get BTARN up and running. However, in many cases this is required by partners. The existing procedures work for Windows 03 x64, but 08 is a different animal. The "ELEMICA" whitepaper (http://msdn.microsoft.com/en-us/library/cc563935.aspx) provides the bulk of configuration details. The process for 08 is easier but not unless instruction is provided. The client authentication certificate installation requires more explanation.

Open MMC and add the "Certificates" add-in. Select the machine account and open the "Personal" store. Import the private key used for client authentication.

Right click the new entry and select "All tasks". Select "Manage Keys" and give "Everyone" read access to the key.

The whitepaper includes a link to replacement web pages required to reference the certificate used for client authentication. The documentation is not so clear on the configuration of the web page. The additional parameter in the web.config must point to the public key of the client authentication certificate. This is the unusual situation where both the public and private keys are required.

Call orchestration with ref parameter in a loop

Sometime back I worked on a case where the spool would go very high and I have been thinking to blog this for some time now and today I decided I will process this awaiting queued item in my mind.

I had a customer who was seeing a surge in the spool count whenever an orchestration instance started and spool would go back to its original count only once this instance get completed. As we all know, the high count of spool impact Biztalk processing, the customer was too worried about it.

Looking into the orchestration, they have got a loop shape which could loop up to 40K times or even more. In this loop, they were calling an orchestration with Biztalk message being passed as ref parameter.

We found that with REF we need to keep messages around until the orchestration completes, so that is why the spool count keeps growing till the loop completes and once the orchestration instance completes, Biztalk jobs would clean up messages from spool. This is by design.

To demonstrate this, we got a simple orchestration with a loop inside. This will loop for 50 times only.

Inside the loop, we are making a call to another orchestration and passing Biztalk messages as REF parameter.

If we test it, and monitor the spool count in perfmon, we will see that spool keeps growing until the loop completes and once the orchestration instance completes, Biztalk jobs are quick to clear the spool.

As a workaround for REF parameter, if we pass a message as IN parameter and have one more OUT parameter of the same type in the called orchestration to return the result, I see that spool growth is lower than while using REF parameter, but still for very large loops spool count can grow substantially.

Also, if possible we can try using variables instead - which can be reused as opposed to message.

So moral of the story is we have to avoid large loops with call orchestration shape passing ref parameter. We would suggest using smaller loop, or removing the call structure by using expression Shape or custom object etc.

How to join two schemas in a map when they contain namespaces

I worked on an issue where we were receiving two different messages (Message1 and Message2) inside orchestration each with multiple records. Now, we have to join these messages in a map inside orchestration based on TRANID and PROCID. So basically, we have to get the value of TRANID from Message1 and look for the record inside Message2 where value of PROCID matches to value of TRANID and get the value of SUM element and populate it in the output message and this process have to be repeated for every TRAN record inside Message1.

Input Message1

Input Message2

Output Message

Though, there is already a solution for this which could be found at the blog <http://geekswithblogs.net/synboogaloo/archive/2005/04/22/37335.aspx>, but only thing different is that in this case, input schema contains namespace which would complicate the XPATH Query used in XSLT Call Template. Let us figure it out later, how to quickly build XPATH query using namespaces in this scenario.

1) First, create the schema for the three messages.

2) Now create orchestration like below. It will receive these two input messages, constructs the output message using map inside Transform shape and then sends the output to a file location. Since we have to receive two different messages using File adapter and we do not have any unique identifier for correlation purpose, I used BTS.ReceivePortName for correlation. Though, this may not be very practical, I just want to demonstrate joining of two schemas when they have got namespace. But yes, make sure that both the receive locations are in the same Receive Port.

3) In the transform shape, select the msgInput1 and msgInput2 as Inputs and msgOutput as Output. Open the map now.

4) In the map, put a looping functoid as in the figure below. Link TRANID and DESC from input to output. The main thing remaining is to retrieve value of SUM from INPUT2 when TRANID matches to PROCID. Now drop a scripting functoid to the map and connect it to TRANID as Input and SUM as Output. It should look something like below.

5) Now, for the scripting functoid, things would have been something like below if the schema does not have the namespaces. But here, this XPATH query inside the ‘Inline XSLT Call Template’ will give us only blank results.

6) So we need to use the namespaces prefix in out query to get the desired output. Now we need to find out the namespace prefix being used in map for INPUT 2 schema. To find it out, we need to view the XSLT of the map. Therefore, we have to validate the map file first. So, right click the map file and say validate map. Now in the output, we will get a link to the map XSLT. Open this XSL file. Now in this XSL file, we can find the prefix used for the namespace “http:\\Input2” and that is ‘s1’.

7) Now, use the prefix "s1" inside the XPATH Query as below. There could be other ways to use XPATH Query but this was the best I could figure out.

8) Deploy the solution and test it. That’s all.

I believe there would be several other solutions for this and if you know one, please share it. Would really be interested in knowing that.

Throttling

In Biztalk sometimes we see that performance of server is going down. Suddenly server has become slow.

Most of times this would be because of Host throttling.

Now, what is Host Throttling? :) ... Refer Link http://msdn.microsoft.com/en-us/library/aa559591.aspx

The throttling mechanism moderates the workload of the host instance to ensure that the workload does not exceed the capacity of the host instance or any downstream host instances. 

There are lot of scenarios because of which Host throttles. It can be because of Low Process Memory, High DB Size,high thread count and some more

It is very easy to find out reason for throttling .For this ..Go to Start--> Run -->Type perfmon --> Go to report (As shown in Following Figure)

After Going to Report , Select Counters Biztalk: Message Agent .

And Then Select all Counters and all Instances , Click Add.

There are two record which I have highlighted, This are throttling State for Publishing and Delivery, This state will let us know if Biztalk is throttling or not

If value of this state is Zero it means there is no throttling . But if value of State other than Zero than it is throttling and using below table you can know what type of throttling it is.

Message delivery throttling state

A flag indicating whether the system is throttling message delivery (affecting XLANG message processing and outbound transports).

• 0: Not throttling
• 1: Throttling due to imbalanced message delivery rate (input rate exceeds output rate)
• 3: Throttling due to high in-process message count
• 4: Throttling due to process memory pressure
• 5: Throttling due to system memory pressure
• 9: Throttling due to high thread count
• 10: Throttling due to user override on delivery

Message publishing throttling state

A flag indicating whether the system is throttling message publishing (affecting XLANG message processing and inbound transports).

• 0: Not throttling
• 2: Throttling due to imbalanced message publishing rate (input rate exceeds output rate)
• 4: Throttling due to process memory pressure
• 5: Throttling due to system memory pressure
• 6: Throttling due to database growth
• 8: Throttling due to high session count
• 9: Throttling due to high thread count
• 11: Throttling due to user override on publishing

Refer Following link for more information http://msdn.microsoft.com/en-us/library/aa578302.aspx

So By looking at state you can know Reason for Throttling. Depending on the state you can find the Solution

For Eg: If Message Publishing Throttling State is 6 then it is because of High DB Size . High DB Size can be if you are not properly maintaining Biztalk DB. This can be because Not all Required Jobs are running . You need to have look at Jobs here.

If Message Publishing Throttling State is 4, it can be because of  Low Process Memory Pressure.

You can go  to Host , Properties and Throttling Threshold. Here you can increase Process Memory and check if throttling is gone.

Host has lot of setting related to throttling . You can refer to this article http://msdn.microsoft.com/en-us/library/aa559628.aspx to understand more about this settings.

Hope this helps.

Messages not appearing in the dropdown box of Transform configuration

If we are using Multi-part Message inside Orchestration and we have any part which is of simple type, then we cannot directly access any message part of this Multi-part Message from the transform shape inside Orchestration.

This is a limitation with the Multi-part Message type object. If the Multi-part Message type contains ONLY schema types, then only it can be directly accessed in a transform shape, otherwise we will not be able to access any of the parts from the transform shape. This is common when we try referencing a WSDL that contains both simple and complex type’s message parts, the resulting parts of the Multi-part Message type are not available for selection in the transform configuration of the orchestration.

Let’s say we referred to a WSDL file and the resulting Multi-part Message Type contains parts both simple and schema types. In the below figure, the Multi-part message type contains 2 parts which are simple types (outlined in red) and 1 part which is schema type (outlined in green).

Now, create a message of this Web Message Type.

If we drop a transform shape in the orchestration, we will not see any message in the drop down box. It means that we cannot directly access any part of this Multi-part message from the transform shape.

So now the question is how do we construct this Multi-part Message so that we can call our web-service? Lets figure it out.

The workaround is first create a message in the Orchestration View for the schema type part of multi-part message.

Then, we can now select this message in the transform configuration. Now we can easily construct this part of the multi-part message in the transform shape.

Now, only thing remaining is to construct the Multi-part message. Once we have construct the schema type parts, we can easily construct the Multi-part message using Message Assignment Shape as shown in below figure. Here we have directly assigned values to the simple type parts and assigned the above create schema type message to the schema part of Multi-part message.

So, this is how my orchestration looks like.

BAM tracking data gone missing

Recently, I worked on a BAM issue and I wanted to share with you all some interesting facts which I found while researching on this.

Customer implemented BAM Activities with Event Streams. They were using OrchestrationEventStreams within Orchestrations.

This is how their flow looks like in short:

Orch1:  Start Orch1 –> BeginActivity –> UpdateActivity –> Enable Continuation –> Send message to Orch2 so that it gets instantiated and passing Continuation Token to it --> End Activity
Orch2:  Start Orch2 --> Update Activity with Continuation Token  --> End Activity with Continuation Token

Customer observed that sometimes there is a row in the BAM Active table with ActivityId set to ContinuationToken and IsVisible set to NULL. This tracked BAM data is sitting idle in the active table and would not go to Completed table. Also, since the IsVisible is set to NULL, it would mean that it is not going to be shown through BAM view or BAM Portal, unless you write a custom query.

Researching on this, I found that if IsVisible is NULL, this would mean that BeginActivity was never called. So, what was happening in the customer case that that BAM events from Orch2 were getting processed and reaching BAMPrimaryImportDB while the BAM events from Orch1 never make up or they got stuck somewhere in between. Now, the question was how come the events from Orch1 were not getting processed though the events from Orch2, which got instantiated after Orch1, were getting processed. Also, keeping in mind, that the issue is intermittent.

Lets now understand how do the OrchesetrationEventStream(OES) API works. OES API’s are asynchronous. This means that API stores tracking data first in the BizTalk MessageBox database. Periodically the data is processed and persisted to the BAMPrimaryImport database by the Tracking Data Decode Service (TDDS).  There are four tables inside the MessageBox database which stores BAM Tracking data before it gets moved to BAMPrimaryImportDB i.e trackingdata_0_0, trackingdata_0_1, trackingdata_0_2, trackingdata_0_3 (note: the other four tables trackingdata_1_x store the tracking data for BiztalkDTADB). For a particular Orchestration instance, OES uses the Orchestration ID as the StreamID and all the events are written to the same table by the OES and TDDS can guarantee that the events are processed in the same order. http://msdn.microsoft.com/en-us/library/microsoft.biztalk.bam.eventobservation.bufferedeventstream.streamid.aspx

But since we have two different Orchestrations here, it would mean that all the BAM tracking data could go to different Tracking tables inside MsgBox and then we cannot guarantee that it would be processed in sequence. But since we are using Continuations, BAM guarantee that the end result after all the events gets processed (no matter what the sequence is), we should see correct result in the BAM Completed table. That’s the magic with BAM.

Now we queried on the four trackingdata tables inside MsgBox DB and found that the table trackingdata_0_0 has more than 50000 rows which keeps moving in the upward fashion and never decreased. It seems that somehow BAM data stored at this table is stuck there and TDDS is not able to move it to BAMPrimaryImport DB. It now clearly makes sense why we sometimes see BAM events from Orch2 inside BAMPrimaryImport DB and not from Orch1 as the events from Orch2 might have been stored at table other than trackingdata_0_0 and got successfully processed by TDDS. It also explains the intermittent nature of problem as other times the BAM events from Orch1 and Orch2 got written to table other than trackingdata_0_0. Also, sometimes both the events from Orch1 and Orch2 could get written to trackingdata_0_0 table and we would see no BAM data getting tracking in BAMPrimaryImport.

Later, on more digging, the issue was found external to Biztalk. The issue was all related to network and firewall between the Biztalk and SQL servers. After those issues were fixed, all data from trackingtable_0_0 table moved to the BAMPrimaryImportDB correctly.

Message for the File Send Adapter remains in active state

This was an interesting issue on which I worked with Shaheer (Biztalk Escalation Engineer) for 2 customers, so I thought I would share it with all.

Customer is using File Adapter at Send Port to write files to a file server. Sometimes, the send message stuck inside in Biztalk with active status and a 0 Kb file is created on the share on the file server. The message remains in active state until we restart the host instance, and then the message is written to the file share.

We took the Biztalk traces and although we did not see anything for the send port other than them being stuck, at the same time that the send ports were stuck, we could see the receive locations hitting the same file server were throwing error 0x800703E5 and Win32 Error = 56 in the traces and even sometimes 80070038

[filelistener]Network connection to location (\\<fileshare>) is down. Error = 80070038

--- 0x80070038 means:
ERROR_TOO_MANY_CMDS winerror.h
# The network BIOS command limit has been reached.

--- 0x800703E5 means:
ERROR_IO_PENDING winerror.h
Overlapped I/O operation is in progress.

--- Win32 Error = 56 means:
Network Bios Command limit has been reached.

This means the network bios command limit has been reached between the Biztalk Sever and the file server. This can happen when there is a very high load of SMB traffic between Windows servers since the default network bios command limit may not be high enough to handle this load. This will effect any SMB traffic between BTS & that file server so even though the error in trace was happening for receive location, it would apply to send port hitting same file server also.

Now comes the resolution part:

Need to follow the below steps as per KB 810886 to increase command limit by setting BOTH of the following registry keys on all the Biztalk Servers AND the File server AND any other file server that you may have problems with:
1. Click Start, click Run, type regedit, and then click OK.
2. Locate and then click the following key in the registry:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\lanmanworkstation\parameters
3. In the right pane, double-click the MaxCmds value and in the Value data box change the value to decimal 5000. (If the MaxCmds value does not exist, create it as a new REG_DWORD value of decimal 5000)
4. Locate and then click the following key in the registry:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\lanmanserver\parameters
5. In the right-pane, double-click the MaxMpxCt value and in the Value data box change the value to decimal 5000. (If the MaxMpxCt value does not exist, create it as a new REG_DWORD value of decimal 5000)
6. Quit Registry Editor.
7. Reboot the computer

We have said above value 5000, taking less cautious route to start with a high value and work your way down. If we want to be cautious, we can start by increasing it to 1000 and then see if we still have the problem and if so, increase it again. In a very high load environment, even 5000 may not be enough and the value would have to be increased further. Only thing we could think with high value even as high as 50,000 is that we may run into high cpu on file server if it can't handle the load.

[Note: The maximum number of simultaneous, active requests between an SMB client and the server is determined when a client/server session is negotiated. The maximum number of requests that a client supports is determined by the MaxCmds registry value. The maximum number of requests that a server supports is determined by the MaxMpxCt registry value. For a particular client and server pair, the number of simultaneous, active requests is the lesser of these two values. This is documented at KB 810886.]