The dot before the Net : Biztalk

Required NTFS permissions for BizTalk File Adapter connecting to network share

Sanket Bakshi — Sat, 06 Jun 2009 05:38:48 GMT

When connecting to a network file share using BizTalk File adapter, you might've noticed that just specifying "Modify" rights is not sufficient. In simple configuration terms, you would need to have a minimum of "Full Control" or the BizTalk Host Service Account. Obviously this would not gel well with the system administrators.

Tom Canter (http://blogs.neudesic.com/blogs/enterprise_integration/archive/2007/02/22/4260.aspx) has put down the exact permissions that you would require for the FILE adapter that would help alleviate some of the concerns for the admins.

Overall, here is what it boils down to –

For the FILE Receive Adapter

After you provide "Modify" rights, Click on "Advanced" button and on the Advanced Security Dialog, edit the permission entries for the BizTalk Service Account
Here you will notice that the "Delete Sub Folders and Files" option is not given by default. This is exactly the missing link for the receive adapter.
Simply set this property and you are good to go without "Full Control"

For the FILE Send Adapter

If you are NOT USING the "use temporary file while writing" attribute on your send port, only the following permissions are required on the permissions entry dialog
- Create Files / Write Data
If you are using the "use temporary file while writing" attribute on your send port, the following permissions are required on the permissions entry dialog
- Create Files / Write Data
- Delete Files
- Delete Sub Folders and Files
- Read Permissions

Trouble setting the BizTalk Backup Job?

Sanket Bakshi — Sat, 06 Jun 2009 05:19:00 GMT

Recently, I faced some of the issues below when configuring BizTalk Disaster Recovery at one of my customers. The BizTalk solution was implemented on HyperV and since earlier clustering was not supported for SQL 2005 (BTW, it is supported now), we had to separate the Message Box database and the Management, SSO & Tracking databases to ensure better solution performance. The overall configuration looked as below –

Due to this configuration, there were some minor complications that kicked in when setting up the backup jobs.

When configuring BizTalk with the above topology, the BizTalk configuration will create a SQL Linked Server between the Management database and the Message Box database. The backup jobs resides on the management database and the linked server is supposed to help communication with message box server and facilitate the backups.

When we first started with testing the backups job, we ran into the following issue –
- <Msgbox DB Server Name> is not configured for RPC. SQLSTATE 42000. Error 7411
As the error message points out, the first suspect is usually to check if the RPC service is working on the message box SQL Server. If BizTalk is performing its required tasks, then chances are that the issue is elsewhere. Since DTC requires RPC. And BizTalk cannot live without DTC.

The second step is to check the Linked Server properties and ensure that the "RPC Out" property is enabled. Enabling this allows RPC communication between the servers.
Secondly, if both the SQL Servers are configured to use different service accounts, make sure that both the service accounts should have access to the network share that creates the backup and log files. In the above case, the service account that was configured for the message box server did not had access to the network share. The BizTalk backup job in turn "instructs" the message box database server to perform the backup. Hence, the actual writing of the backup file happens with the SQL service account configured on the message box server. If you do not have this setup right, chances are that you will see the familiar "Access is denied" error message and the Backup job will fail.

Hopefully these experiences might save some time the next time you run into this configuration.

--Sanket

Consuming webservices from within BizTalk

Sanket Bakshi — Sat, 31 Mar 2007 16:47:00 GMT

BizTalk 2006 allows some really good interacting with webservices. If it is a simple webservice, we can very well consume it by adding a reference to it and then consuming it using the orchestrations. Here are some interesting scenarios that I encountered when dealing with webservices.

Can I consume the service?

Although Biztalk offers fairly good support for consuming webservices, there are some considerations that might affect the way these interactions are developed.

For instance, the Add Web Reference feature in BizTalk does not understand the import element in the WSDL. Multi dimensional arrays are not supported from within BizTalk & so on. An elabortate list of these considerations can be found here.

In some cases, you might run into one of these limitations - most probably when using non .net webservices. There can be a couple of ways in which we can deal with this scenario. This webservice call can be either wrapped within a .Net component or a webservice that can in turn be consumed by BizTalk.

Using SOAP Headers when consuming the webservice

When consuming a webservice with SOAP headers, the headers need to be written to the context of the web request message. In order to consume these webservices from within an orchestration We need to promote a property with same name as the SOAP Header & assign it to the message context. To achieve this, create a new property schema with target namespace as http://schemas.microsoft.com/BizTalk/2003/SOAPHeader. Each root element in this schema must match the name in the defined SOAP Header. Also, each root element that corresponds to the SOAP header should have its "Property Schema Base" property set to "MessageContextPropertyBase". Setting this property ensures that the element is visible in the list of elements in the message context.

Once done, then this property can be used in the expression editor to set the property on the request message as -

myWebserviceRequestMsg(MyPropSchemaName.MyHeaderName) = "<?xml version='1.0'?> <MySOAPHeaderName xmlns='http://SOAPHeaderSchemas.MyHeader'> <HeaderInfo>abcxyz</HeaderInfo></MySOAPHeaderName>"

The myWebserviceRequestMsg can now be sent directly to the webservice along with the associated SOAP Header.

Using Webservice request & response messages in Mapper

Most of the times, you need to resort to transformation either for the request message - before sending to the webservice or the response message after recieving from the service. In atrading sceanrio, where the trader hosts the suppliers catalog, the orchestration would need to get the catalog from the webservice & then use the transformation to include catalog in the traders database. However, if you are thinking of creating the map directly by adding a new map, the mapper UI does not allow specifying the webmessages as source or desctination parameters.

To overcome this, we can simply use the transform shape to create a new map for us. Specifying the web response message as the input to the transform shape allows it to create a new map with the web response schema. So instead of creating a new map, just drop a transform shape in the orchestration, create two message - one with the web response schema & other of your destination schema. Open the transformation configuration page & just select new map option there. After specifying your source & destination message, a new map will be created by using those schemas.

Hope this alleviates some blues for anyone trying to deal with webservices from BizTalk.

--Sanket

Creating public orchestrations

Sanket Bakshi — Mon, 19 Feb 2007 10:41:46 GMT

I am currently working on one of the Biztalk projects that contains a lot of lengthy orchestrations. In order to split them in manageable chunks, we are making intensive use of the call orchestration and start orchestration shapes. As long as the orchestrations remain in a single assembly, it works great. However, for complex solutions, it makes great sense to split these orchestrations into separate deployable assemblies - probably spanning across several BizTalk applications.

In such cases, orchestrations in an assembly are by default not visible to orchestrations in another assembly. This is due to the default type modifier for the orchestration. The Type Modified for the orchestration is similar to the access qualifier of C#. By default this is set to "Internal". Obviously, with Internal, the orchestration is not visible outside the assembly within which it is coded. So if you are trying to call this orchestration from some other assembly which has referenced this assembly, you would need to explicitly make the orchestration "visible" outside the assembly. This can be done by setting the Type Modifier as "public".

This helps a lot in componentization of the orchestrations in a complex Biztalk solution.

Adding Environment Specific Binding Files from Command Prompt

Sanket Bakshi — Tue, 13 Feb 2007 19:41:21 GMT

Biztalk 2006 provides the btstask utility to carry out most of the deployment operations from command prompt. However, the help available at the command prompt does not talk anything about adding environment specific binding files.

You can get elaborate information about it on MSDN at this location.-

In order to add the environment specific binding files, you need to run btstask.exe with the -P:TargetEnvironment parameter as-

btstask AddResource -ApplicationName:MyBTSAppName -Type:System.BizTalk:BizTalkBinding -Overwrite -Source:"MyProductionBindingFile.xml" -P:TargetEnvironment="PRODUCTION"

--Sanket

What to and what not to do with your MessageBox database server.

Sanket Bakshi — Fri, 12 Jan 2007 23:41:52 GMT

The BizTalk Core Engine Team has compiled an excellent list of things that you can or cannot do on your BizTalk Message Box SQL Server.

I am sure this will be the most sought after thing by all the database administrators trying to "maintain" the Biztalk databases.

Publishing Biztalk Orchestrations as webservices

Sanket Bakshi — Tue, 02 Jan 2007 15:06:18 GMT

When developing Biztalk applications that are exposed as webservices, it becomes quite a tedious job to expose them as webservices everytime there is a change in the orchestration. Even though Visual Studio integrates the BizTalk Web Service Publishing Wizard, it is definitely a cumbersome tool to use everytime.

To minimize the efforts, we can have a simple batch file that can do the job for us. Biztalk comes with a command line utility - btswebsvcpub.exe

Here is an example of how you can achieve this on the command prompt -

btswebsvcpub "C:\TestOrchSolution\TestOrch \bin\Deployment\MyTestOrch.dll" -Overwrite -Anonymous -TargetNamespace:http://testorch /service -ReceiveLocation -ApplicationName:MyTestApp

Furthermore, you can also use this trick for quickly deploying your assemblies from one environment to the other.

Biztalk Assembly Deployment goof-up

Sanket Bakshi — Tue, 26 Dec 2006 16:29:00 GMT

This, to me, seems a rare scenario that you might encounter.

But if you are using the "ReDeploy" feature in the Biztalk project (from Visual Studio) you might bump into this sometimes. When debugging your orchestration with HAT, you might encounter some strange behaviour - Biztalk simply skips some steps when performing the execution. You might notice something similar to this in the HAT -

The problem here is quite simple. When deploying a Biztalk artifact it is deployed in the GAC and also in one of the Biztalk database. The assembly in the GAC is used by the Biztalk host instance to perform the actual execution. The one in the Biztalk database is used for the tracking purpose so that it can be shown in the HAT. Now, lets say you add something to the orchestration (like I did with the Call shape immediately after the Decide) and when deploying this assembly you somehow missed the update in the database. In this case, the Biztalk Host instance does excute the newly added step and reports it to the database. However, when looking at it from the HAT, Biztalk is not aware of this step. Hence, it is shown as a blank and not executed.

I relied on the "Redeploy" property of the Biztalk project which states that it overwrites the current deployment with the new artifacts. However, it seems that redeploy does sometimes miss out the the updates either to the GAC or to the database. (Not sure of the reason why).

The best way to get out of this is to undeploy the Biztalk assemblies and also remove them from the GAC. I also realized that when I delete my Biztalk Application from the Administration Console, it does not remove the assemblies from the GAC. I had to manually remove the assemblies there.

Simply re-deploy the application after undeploying and removing the assemblies from the GAC and you will be good to go. Hope this helps solve some confusion when you see something like this happening with your Biztalk project.

Resolving "Unknown Errors" when working with Commerce Server Profile Adapter

Sanket Bakshi — Sat, 09 Dec 2006 13:56:00 GMT

Most of the times, when you encounter an error interacting with Commerce Server Profiles Adapter, the Adapter will give you back an error with a proper description of what might've went wrong. However, there are certain cases when the adapter simply emits a warning in the event log that mentions about an error with "Unknown Error Description".

This one seems quite tough to solve at the first instance, given that we do not have any further information about this. However, most of the times, as I noticed it this is more to do with some missing values in the message or some mis-configured properties on your send port.

To get the actual description of the error that has occured, you can do the following -

On the commerce server, create a new Profiles Webservice by inheriting from Microsoft.CommerceServer.Profiles.WebService.ProfilesWebService.
Now simply override the ImportProfile and UpdateProfile methods and call the base methods in them.
On the Biztalk Server, instead of pointing to the default Commerce Server Profile Webservice, point to your newly created webservice.
Then in this webservice, you can put a break-point and see why it fails. The actual exception information will be provided here for all the "Unknown Error Description" issues.

Once you know the reason, then it would definitely be quite easy to deal with it.

Associating Multiple Credit Cards when creating Commerce Server Profile using the Commerce Server Profile Adapter for BizTalk

Sanket Bakshi — Sat, 09 Dec 2006 13:44:46 GMT

When dealing with Commerce Server integration with BizTalk 2006, one of the requirements that I faced was to create multiple credit cards for a single profile and then link all those credit cards to that profile.
To achieve this, is a two step process. The first step involves creating the actual credit cards on the Commerce Server. This has to be done by independant calls to the Profile Adapter for each credit card. These calls are made with the "EndPoint Message Type" set as "Import Profiles" on the Send Port (either Static or Dynamic).

Once all the credit cards are created, then the profile create message needs to have the list of ids for all the credit cards created. These ids are stored in the Commerce Server SQL database in the following format -

2;<creditcardid1>;<creditcardid2>

Where the creditcard Ids are nothing but GUIDs corresponding to the keys of the actual credit card objects.

Now to send it from the Biztalk Adapter, it should be sent as multiple nodes of the <credit_card_list> type. So in the actual profile create request instead of having a list of credit cards separated by a semi-colon (;) (as might seem from the value stored in the database), each individual credit card Id should be sent as a separate node.

So the create profile request looks similar to this -

<ProfileDocument>

<UserObject>

<GeneralInfo>

<user_id>fc32ccb8-0672-4457-a99d-6ac01c497912</user_id>

<user_name>sanket</user_name>

<email_address>s@b.com</email_address>

......

<credit_card_list>ab3299c1-3d57-44e9-a99d-63451c49340e</credit_card_list>

<credit_card_list>e40234ed-ab7e-1a5c-a99d-6cbe1c49ba91</credit_card_list>

<GeneralInfo>

......................

</UserObject>

</ProfileDocument>

Hope this helps you save some time when dealing with this feature.

Correction: "Updates Overwrite" property of Commerce Server Profiles Adapter

Sanket Bakshi — Sat, 09 Dec 2006 13:24:00 GMT

A couple of days ago, I had blogged here about an error on the Commerce Server Profiles adapter.

When trying out an update to the Profile, the Commerce Server Adapter returns back the following error message -

<?xml version="1.0"?>
<CommerceServerProfilesUpdateResponse>
<InvalidProfileUpdateMessage>
The message body received by the send adapter is not valid for the CommerceServerProfilesUpdate API call. Message ID: 529029a6-5954-4446-bdd3-a5b4c5051cbb. Detail: There has been an optimistic locking conflict. The object could not be saved. This occurs when someone else makes a change to the object after you have loaded the object but before you have saved the object. To force overwriting of the other user's change, set the overwriteOnConflict flag to true. Profile Name = 'Address'. Primary Key Value = '{f8ec6880-9a9a-4059-9e10-1bb1712687ba}'.
</InvalidProfileUpdateMessage>
</CommerceServerProfilesUpdateResponse>

The solution that I had suggested was to set the "Updates Overwrite" property on the adapter to "true". However, on exploring the avenue further, I came to know that setting this property to "true" would actually cause the Commerce Server to update all the properties for the related profile. If it is not specified as an input, then it is marked as null. This behaviour might not be desirable in most of the cases where you would not like to pass all the information. Only information that is updatable should be updated.

The "Updates Overwrite" property allows doing that by setting it to "false". However, when this propery is set to "false", the Commerce Server Webservices checks for the update timestamp on the incoming message. If the timestamp is not available or if it's value is greater than the last updated timestamp in the database, then the adapter returns back with the above error message.

To take care of this issue, all you have to do is set the "date_last_changed" & "csadapter_date_last_changed" fields in your update request message to the Commerce Server to the current datetime. This can be done by simply using the date & time functoid in the map or by distinguishing these properties and then assigning them in expression shape as -

updateRequestMessage.UserObject.ProfileSystem.date_Last_changed = System.DateTime.Now().ToString();

updateRequestMessage.UserObject.ProfileSystem.csadapter_date_Last_changed = System.DateTime.Now().ToString();

This will now allow you to update the profile without sending the additional information that need not be updated.

One thing that I noticed with this is that even after doing this, If I go ahead to update the profile without specifying my existing credit cards, it then deletes all the credit card links in my profile. To overcome this I have to pass the credit card ids (GUIDs) along with my user profile update message.

Solving the error when updating Commerce Server using the Profile Adapter

Sanket Bakshi — Mon, 13 Nov 2006 12:50:37 GMT

When dealing with Commerce Server Adapters for BizTalk, you might face an issue where the Profile Adapter returns this message on update -

To solve this issue, simply go to the adapter configuration on your send Port and set the "Updates Overwrite" property of the adapter to true. If this property is set, the adapter is allowed to overwrite the existing properties on the profile and hence the error is taken care of.

Understanding Persistence Points in Biztalk Orchestration

Sanket Bakshi — Sun, 12 Nov 2006 12:07:14 GMT

Often ignored, but a really important point when designing orchestrations is of course the persistence points. Understanding of the persistence points becomes really an important factor if you are dealing with orchestration performance. Lets take a glance at what they are and how we can take care of them.

The SQL Server does bulk of the job at the backend to keep the BizTalk server running. BizTalk has this inherent behavior of saving the orchestration state to the SQL Server so as to quickly and easily recover from any failures. Persistence points are those points in the orchestration that save the orchestration state to the database.

However, this behavior induces latency in the execution of the orchestration because of several database trips that are needed when executing the them. Although its quite difficult to avoid the persistence points when designing the orchestration, understanding them and using appropriate shapes and patterns can ensure that you keep them to minimum. For example, the orchestration persists its state after end of every transaction, at the end of orchestration, after every send shape etc. However, if the Send shapes are enclosed within atomic transactions, they do not persist the state. Instead, the state is then persisted at the end of the atomic transaction.

In a usual scenario, the persistence in the orchestrations occur at the following points -

End of a transactional scope (atomic or long running)
At the execution of other orchestrations through the Start Orchestration shape
At the Send shape (except in an atomic transaction)
At the end of the orchestration

Apart from that, the engine might decide to persist the orchestration in the following cases -

When an Orchestration Instance is suspended
When the system shutdowns in a controlled manner
When the engine determines it wants to dehydrate
When an orchestration instance is finished

Persistence can also occur at debugging breakpoints when the orchestration debugger is being run.

Lets just illustrate with a simple example how persistence points can increase the latency during the execution of the orchestration.

Consider orchestration scenario that has 2 send shapes sending messages to two different systems.

This can be implemented in the following ways -

A. A parallel branch that has two Send shapes.

B. Two send shapes being executed sequentially in the orchestration.

C. Two send shapes being executed sequentially in the atomic shape.

Question: Now which of these do you think would work the fastest?
Of course the (B) scenario is out of question. The two Send shapes will have their persistence points. If the Send shape is the last shape in the orchestration, then the persistence point of the Send shape and the end of the orchestration will be batched together to result in a single persistence point instead of two.

With scenario (A) even we have the parallel branches in place, Biztalk by nature executes them in a sequential manner. Parallel branches do not necessarily mean they are executed on separate threads. Hence, even in this case, it results in sequential execution with two persistence points. Infact, in this case, the persistence point will not be batched with end of the orchestration and hence it results in the slowest of the three.

The scenario (C) takes the fastest position of the three. The send shapes are enclosed in the atomic shape and hence the state is persisted only at the end of the atomic shape. Again, if it is at the end of the orchestration, the persistence point will be batched with the persistence point at the end of the orchestration. Hence resulting in a single persistence point.

Hence, when designing the orchestration from the performance perspective, it is imperative to take a look at what points the orchestration will be persisting the data. Reducing the database trips helps increase the performance of the orchestration.

Supporting mixed content messages with BizTalk

Sanket Bakshi — Thu, 02 Nov 2006 06:18:04 GMT

Ocassionally, when dealing with Xml messages from third party systems, your Biztalk solution might need to deal with mixed content. A classic example of the mixed content is the HTML as in -

Biztalk allows handling these kind of messages when defining the schemas. To do this, all your have to do is set the mixed attribute of the Record to true. Setting this attribute ensures that the record can contain text as well as any child nodes within it.

Explanation about the mixed property can be found here.

Naming the lines in a Biztalk map

Sanket Bakshi — Wed, 01 Nov 2006 16:03:02 GMT

When using the Biztalk mapper, most often than not you would need to resort to functoids. When connecting the functoid inputs from the fields in the Source schema, we simply stretch a line from the field to the functiod. However, when we open the functoid to check the input parameter, it shows up as a long xpath string. When dealing with large maps this can very easily become cumbersome.

Alternatively, a better way is to label the line that you stretch to the functoid. So whenever you draw a line from any source field to the functoid. Just click the line and enter a meanigful name in the Label property in the Properties window. The next time you open the Input Parameters window for the functiod, you will see the line label instead of the cryptic xpath.

This surely will make life a bit easier when dealing with huge maps. Also, this can be one of the good coding practices.