BizTalk Architecture, High Availability and MSMQ Adapters : Scalability

MSMQ Adapter on an NT cluster in BizTalk 2006

EldarM — Fri, 08 Jul 2005 18:28:00 GMT

Some of you may already have a preview version of BizTalk 2006 from TechEd. If so, you may try using MSMQ Adapter on NT cluster. You need to make BizTalk process a cluster resource (there is a checkbox in Admin for that) and have MSMQ service on the same cluster.

In the preview bits that you have there is a trick you need to keep in mind: the clustered Biztalk resource must have dependency on Computer Name resource and enable Use Network name as computer name as check boxes to make it work. Otherwise MSMQ queue will be considered remote and you will not be able to use it transactionally.

Is BizTalk really always stateless?

EldarM — Fri, 17 Sep 2004 18:01:00 GMT

Really stateless?

Not exactly. The server architecture is stateless, however BizTalk has add-ons like custom adapters that may compromise stateless status or require additional configuration work to keep it.

Here is an example. Configure a receive location with a file adapter. Now, sender puts a lot of files into the receive folder, and BizTalk is picking them up. Notice, that because of this file receive location BizTalk machine is not stateless anymore. So if the machine with the folder is down, messages cannot be retrieved until the machine gets back online. To make it scalable and reliable you have to put additional effort into putting this folder on a network share, on RAID or SAN storage, etc. Of course, most people don’t do that because they use file adapter in cases when application level protocol can handle transport level faults.

MSMQ/T

It’s different with more reliable transports, especially MSMQ. MSMQT adapter, which is part of the BizTalk, works in a stateless manner as expected. User does not have to do anything to achieve this, it works like magic, but behind the scenes it was not so simple. Let see, what was done to make MSMQT a “good citizen.”

First, MSMQT does not have local queues, it uses BizTalk MessageBox as a persistent storage. Hence, the actual state (queue) is stored on the SQL Server machine.

So, everything is fine? Not yet. What if several BizTalk machines are behind NLB (Network Load Balancing)? MSMQT uses exactly once in order delivery. Hence, only one machine at a time can process messages for a particular receive location – otherwise the order cannot be guaranteed. However, if this machine is down, another machine in the group should be able to pick up the work. How is it done?

When MSMQ TCP connection is established, MSMQT locks an object in the database which is associated 1:1 with the receive location. This ensures that no other machine will pick up the connection for the receive location. If the machine owning the connection goes down or the connection times out, the object is released. So the next time connection is reestablished by the external machine, it may go to a different machine in a group and continue where the first machine dropped off.

Again, user does not see this, it all happens behind the scenes. What the user sees is a reliable delivery of messages from outside to the BizTalk MessageBox and back.

MSMQ/C

If you will look into SDK samples, you will find another adapter – MSMQ/C. It is delivered as a sample, but if you visit Beta place, you know that a fully supported version is coming. This adapter does not implement MSMQ server and does not talk over the wire. Instead, it uses MSMQ service available in Windows (Message Queuing component of Windows). What it means is that a message first receive by Windows MSMQ and put into a local queue on the disk. Then MSMQ/C adapter picks it up from the queue and puts into the MessageBox.

What it means for the stateless status, is that you’ve got a local persistence store – MSMQ Queue. If machine with this particular store goes down, the messages that are already there and not consumed by BizTalk are unavailable.

You still can make it scalable and fault-tolerant, MSMQ has an elaborate story for that and you will have to follow it and configure MSQM service to work under NLB and on a cluster. In particular, it may involve making MSMQ queues a cluster resource, binding MSMQ to the NLB IP address, etc. This additional administrative cost is the result of breaking stateless status of an individual BizTalk server machine.

Notice, that the second store also gives you some headaches when it comes to backup and restore story, but let touch backup and restore in a separate article.

Any others?

Any other perpetrators breaking the stateless status? Not many. You see, the problem with the local storage may only exist if it is used as an intermediate between sender and receiver. File or MSMQ queue are examples of such intermediate storage. In this case, sender delivers the message to the intermediate non-BizTalk storage and has no clue, what happens next. Once you have an intermediate storage, you have to administer it and make sure that it is scalable and fault-tolerant, at least, if you care for that.

Another example of such intermediate storage is SQL Adapter, which uses a SQL Server table to submit the data. In this case, scalability and fault-tolerance is very easy to achieve, however, it still requires extra administration. So in most cases, it is useful to stick to the stateless server concept.

“Stateless” server – the concept (complexity 2 of 5)

EldarM — Tue, 17 Aug 2004 22:32:00 GMT

This is mostly an introductory piece to the next post. It covers the concept of the stateless server, which BizTalk (its processing part) is.

BizTalk is built to be a symbiosis of two servers: processing server (send/receive, processing, applications – that’s the one you install from the BizTalk CD) and persistence server (persistence, routing – SQL Server with databases, tables, and stored procs created remotely).

The processing server – actual BizTalk node – was architected to be stateless, that is containing no application state. This way if any BizTalk node gets down, it does not matter. There is no loss of state (or data) because there is no state on BizTalk nodes. Some other BizTalk node picks up the work from where it was last persisted in the SQL Server.

Let consider few examples. If HTTP message comes and the power goes down. If message was not yet persisted in the database (message box) then the sender did not received “400 OK” and HTTP connection will time out, so the sender will know that the message did not came through. Similarly, if instead of power failure something happened during the processing of the message, the sender will get something like “500 …” or other HTTP error. These cases are exactly the same as if connectivity is lost, e.g. some roadwork cut the cable. These cases require the sender to send the message again.

When the message is received and persisted, the processing server health does not matter anymore, because the message is now in the database. Now the sender got its “400 OK” and the message is “handed off”. If BizTalk node gets down, it does not matter anymore, because the message is in the database and it will be picked up by some other node.

Similar picture is present for most other adapters and transports. It is also the case for BizTalk applications – orchestrations. When an orchestration sends or receives a message, its state is saved in the database, so even if BizTalk will go down, the orchestration will be revived (rehydrated) from the point after sending or receiving a message. And, as you can guess, putting the message for delivery into the message box and persisting the state of an orchestration is done in the same transaction. So, even if some simple steps may be repeated, no communications with an external world going through BizTalk are repeated or missed.

“Stateless server” architecture helps to build scalable and fault-tolerant messaging solutions and applications that continue to run despite individual hardware failures. Of course, it does not come for free. Stateless server is not a silver bullet, magically resolving all the problems. And anyway, there could not be a fault-tolerant systems on a single machine. However, it is the critical part allowing to build such systems – see future posts on how to do that.

What’s also important, stateless server significantly simplifies the development. It allows to create applications (orchestrations) in a usual stateful manner, but when they run they have the advantages of stateless programs. If you worked with 3-tier architectures or J2EE, you should know that the main advantage of stateless components (data access components in 3-tier designs, business logic and data access beans in J2EE) is scalability. Their disadvantage is the need for stricter development and design discipline that needs to be enforced. With BizTalk you don’t have to do that, in fact, you can write a single application (orchestration) that spans several HTTP submissions from different sessions and different people – BizTalk infrastructure transparently takes care of the state.

Now, what’s the catch? Is BizTalk really “stateless”? Yes, it is, except… That’s what the next post is about.