Technically, you don’t have to migrate to MSMQ adapter. MSMQ/T is present and fully supported in BizTalk 2006. It is deprecated, but that only means that it may be not there in some next future version. Again, in BizTalk 2006 it is present and fully supported. Many customers may prefer just to do nothing and continue to use MSMQ/T adapter and avoid migration pain at this time.
Consider also the following question: Are you sure what kind of messaging you would like to migrate to in three-five years? Have you checked the new .Net 2.0 framework already available as Beta version? If you did, you may have noticed many exciting new features in the platform in the messaging area.
On another hand, if you are considering alternatives to MSMQ/T adapter anyway, MSMQ adapter may be a great choice. In this case, here is what you need to keep in mind.
Generally, MSMQ/T is a high-fidelity implementation of MSMQ server, and within provided functionality it is identical to MSMQ on the wire. In fact, it’s build out of the very Windows Message Queuing 2.0 code. So in simple cases migration to MSMQ is just that: reconfigure ports to use MSMQ adapter, period. Except the cases listed below, nobody should notice a difference.
In fact, if you have multi-machine installation behind NLB, you may be even able to migrate non-stop. Although, unless you have a very good reason for that, I would advice against that for the mere possibility of hiccups in any IT movement. Anyway, if you must, here is what you do:
Again, unless you have a very-very strong reason to do that non-stop, I would advise against that.
MSMQ/T ensures end-to-end in-order delivery. That means that if an MSMQ application sends messages 1,2,3 to MSMQ/T port of BizTalk, then these messages are delivered to an orchestration in BizTalk in the same order 1,2,3. The cases when it is required are rare, but when it is required, it’s an absolute must-have. Examples include stock orders that stock brokers must submit and execute in the same order as they received them.
End-to-end in-order delivery seems like a simple statement, but behind the hood it includes a lot of components working together to reach this goal:
In BizTalk 2004 MSMQ/T is the only adapter capable of that. For every other adapter at least steps 3 to 5 may occasionally switch the order. Step 3 for most other adapters is done by the component called End Point Manager, which is inherently multi-threaded. MSMQ adapter for BizTalk 2004 has a feature called “Serial Processing” that will ensure in order processing for step 3, but it does not asks MessageAgent to keep the order going forward, so the messages may be routed to an orchestration in a different order.
So, rule #1 for end-to-end in-order processing is: Use BizTalk 2006 (unless you stick with MSMQ/T).
In BizTalk 2006 receive ports in orchestrations and send ports has Ordered Processing property. If it is set, it means that the send port asks MessageBox to deliver messages to it in the same order it was submitted into the MessageBox. This flag ensures steps 4 and 5.
Also, MSMQ receive ports have Ordered Processing property, which means that they should not mix up message order and they should submit the messages in the same order as they were received from MSMQ. That ensures step 3.
Also, the actual MSMQ queue must be in order, that is transactional. That will ensure steps 1 and 2.
Now, in short, to get MSMQ/T-style in order delivery in BizTalk 2006 with MSMQ adapter, you must:
An important notice to keep in mind is that because of the local storage, the order may be only guaranteed if each queue is serviced by only one machine. That sometimes represents a problem for scalability, as described below.
There is one key difference at how MSMQ/T and MSMQ adapters process a message.
For MSMQ/T, getting a message from network, processing it through pipelines, pushing it into the MessageBox and routing is a single transaction. When the sender receives ack, it means that the message is already routed to the destination.
For MSMQ Adapter, receiving message from the network and getting it into BizTalk are two different transaction. The sender receives ack whenever Windows Message Queuing server gets the message from the network and pushes it into the local MSMQ queue. There could be no BizTalk on the machine at all, the sender still will get ack. Then MSMQ Adapter tries to pick it up, process and publish. If this process fails, the message gets into the suspended queue or get stuck in the local queue (for transactional processing), and the sender has no indication that something’s got wrong.
In most cases you don’t care about this differences, but if you do, you really need to know that. In which case you have at least two options:
High availability in MSMQ/T is easy. You just put it behind NLB, and if one machine is down, another picks up the work. For MSMQ Adapter NLB does not work if you need transactional processing with no data loss, because there is an intermediate storage – local MSMQ queues. What happens is that if a message already delivered to a local MSMQ queue, but not yet consumed by BizTalk/MSMQ Adapter, and the machine crashes, then the message is lost.
You solve this problem by NT clustering. You should put two BizTalk machines in an NT cluster, configure BizTalk process as a cluster resource, configure MSMQ to keep queues on the shared cluster disk (RAID or SAN). Then if one machine crashes, another takes over. And if the physical disk crashes, RAID or SAN takes care of that.
Important thing to know is that you only can do that in BizTalk 2006. BizTalk 2004 host process cannot be registered as a cluster resource, and hence clustered MSMQ queue will be considered remote by MSMQ, in which case transactional read does not work.
If you followed me, the actual key problem is availability of a transactional remote read on MSMQ. If it would be there, you will just use it and thus eliminate the local storage, which is the root of the problem. Funny enough, there are two incarnations of MSMQ that have remote transactional read. Less funny is the fact that both of them are not supported. Here they are:
Anyway, you have an NT cluster solution which will work today.
Just as an exercise, what if you want in-order delivery AND high availability? The answer is: you do steps 1-3 above AND you use NT cluster.
If you don’t care about transactional send/receive with in order delivery the picture is much simpler. In MSMQ/T you had it transactional because that’s the only thing that MSMQ/T is capable of. But if you don’t care about that, you can use MSMQ adapter. Just configure MSMQ ports for express delivery, put MSMQ behind NLB – the instructions are pretty much the same as for MSMQ/T and generic MSMQ – and you are done.
In MSMQ/T, scalability is achieved by using NLB.
Now, here is how to scale out MSMQ Adapter.
Again, non-transactional is easy. You just put it behind NLB and configure it according to MSMQ NLB guidelines. Messages get delivered to different local queues and picked up in a completely random order, but you don’t care about. And in return you get a blinding fast delivery and good scale out.
For non-transactional story, that’s it.
Technically, there is no such thing as transactional not in-order delivery in MSMQ. However, if you configure MSMQ queues for transactional processing and scale it out with NLB, BizTalk may break the order in failover scenarios. What’s more, in failover scenarios picking messages up may break the transactional part. Imagine that a machine received messages 1,2,3 and had a disk failure. In which case, MSMQ will confirm to the sender receipt of messages, but BizTalk will never pick them up. To counteract that you will need also use NT cluster on the machines with MSMQ queues and BizTalk, which will make sure that the messages are not lost in failover.
So, to have it transactional, you will need to combine MSMQ scale out using NLB with NT cluster. Notice that you don’t get in-order delivery in this case. See below why. But if you want transactional processing and don’t care about the order of messages, that’s the way to go.
MSMQ scales all right, and the message will come to MSMQ queues in order. MSMQ adapter will pick them in order and the rest of BizTalk 06 machinery will ensure that the order is preserved. However, between MSMQ queue and MSMQ adapter pick-up there is a break point in order. If TCP/IP connection for MSMQ session is broken, then it may be restored by NLB to a different machine. Then messages 1,2,3 will come to machine A, and 4,5,6 to a machine B. Both sequences will be in order. But the instances of MSMQ adapter on the machines A and B, will not be able to synchronize them between each other. More to that, if the connection is broken because machine A went down, it may be still down when messages 4,5,6 will arrive to the machine B. Then on machine B MSMQ adapter instance will pick it up, process, and afterward machine A will complete the failover and start to push messages 1,2,3, with the resulting order of 4,5,6,1,2,3 in the MessageBox.
A partial solution would be to load balance it manually by allocating destination queues to different machines. It can be done by using separate BizTalk hosts through configuration.
If this is not acceptable then to solve this problem with MSMQ adapter you need a remote transactional read, which is the only existing alternative to MSMQ/T capabilities. And MSMQ remote transactional read is not available yet. So you may be better off sticking to the existing MSMQ/T solution until MSMQ remote read will be available and supported by BizTalk. Again, consider the question in the beginning of this article.