BizTalk Architecture, High Availability and MSMQ Adapters : BizTalk

(Reminder: this blog is DEAD) Question from mail: steps to install the jde adapter for biztalk

EldarM — Fri, 23 Mar 2007 21:44:00 GMT

Can you please help me by giving me the steps to install the jde adapter for biztalk.
thank you
----------------------------------
This message was generated from a contact form at: http://blogs.msdn.com/eldarm/default.aspx
It was submitted by bhargava (...@....in)

I assume setup.exe or whatever is written in the docs for JDE adapter would work, but I don't work on BizTalk for more than a year now, so I'd recommend going to public newsgroups/forums with this question. There is a lot of people who may already know the answer and the BizTalk team monitors them and answers the questions there. Quote:

Just for reference, they (public newsgroups) are here, and the team monitors them.

(Reminder: This blog is dead) Hello Sir, I have developed a BizTalk2004 application.

EldarM — Thu, 30 Nov 2006 19:25:00 GMT

Reminder: this blog is dead. Please, try one of the BizTalk team members listed on the left for your questions.

Sent: Thursday, November 30, 2006 5:10 AM
To: Eldar Musayev
Subject: (BizTalk Architecture, High Availability and MSMQ Adapters) : Regarding BizTalk Maps - Post Deployment
Importance: High

Hello Sir,

I have developed a BizTalk2004 application. It has one orchestration, 2 schemas and 1 map. I have deployed that application into a production machine. I have packaged it using BTSInstaller and installed it in that production machine.

After a month, I have changed the map in my local machine. Now I need to replace the map in that production machine also. How can I do that without reinstalling the whole package?

I need to replace the map only.

Is there anyway like compEif.exe?

Please suggest me.

Thanks and Regards,
Nallasivan

I am not sure about this specific question, although, I think, it may be possible. Except that you should keep in mind, that a lot of artefacts are compiled, not interpreted. As a reminder, I have moved from BizTalk team months ago, so you may be better off asking some current BizTalk team members. Or, even better, try the newsgroups support. Did you see my previous post? Quote:

*** Addded 3:13pm:
Got the response from the team:

Hi Eldar, please refer him to PSS or the public newsgroups.
--mike

Just for reference, they (public newsgroups) are here, and the team monitors them.

(Reminder: This blog is dead) Mail: How i Load balance Biztalk 2004?

EldarM — Mon, 26 Jun 2006 20:42:00 GMT

Sent: Monday, June 26, 2006 7:13 AM
To: Eldar Musayev
Subject: (BizTalk Architecture, High Availability and MSMQ Adapters) : How i Load balance Biztalk 2004?
Importance: High

I need help on configuring Biztalk 2004 load balancing.
I am not talking about NLB.
Is there any configuration in Biztalk for load balancing?
Please do revert back at the earliest.
Regards,
Nilesh Roy.

First of all, there is no single switch "load balance", you simply need to understand what you are doing. See the previous posts on this blog, it's explained there in fine details.

Very short: you don't have to load balance BizTalk server because bu itself it is stateless. You just throw more boxes in a row, and that's it. But individual adapters is a completely different story, and - don't forget! - BizTalk is supposed to connect to virtually anything. It means that adapters may have a very different set of idiosynchrazies. That's what you have to load balance, and each of them may require a different approach to load balance. It's not BizTalk that you have to load balance, it's the transports that you use and adapters for those transports. Makes sense?

So, load balancing BizTalk depends on what you connect to and how. Again, see previous posts on that. Also, you may have to load balance SQL Server with message boxes. Basically, you add more machines with message boxes, and that's it. In BizTalk 2006 you may also make primary message box dedicated to routing, that improves the throughput. See Lee's blog on that.

And, I'll forward you email to BizTalk team (remember? I am not there anymore!). They'll answer you more if they have something to add, but considering that your question is very generic, that's probably it.

*** Addded 3:13pm:
Got the response from the team:

Hi Eldar, please refer him to PSS or the public newsgroups.
--mike

Just for reference, they (public newsgroups) are here, and the team monitors them.

Q from the mail:

EldarM — Fri, 17 Feb 2006 19:44:00 GMT

I have a problem with the application that I'm developing, and I think that the approach described on your post might help me. You talk about several differences between the Large API, and the normal API of the message queue. The Large API is documented, if yes do can you send me a link to it, if it is available of course ?

Yes, see BizTalk docs on MSMQT adapter. It's actually very simple and almost identical to MSMQ API. Also, there is the sample in SDK.

One last question, if I understant what I've read when using this API the limit is the physical memory of the machine that is sending the messages ?

That's correct, but realize that memory is also used by other programs and OS, so it's rather available memory. The numbers of what we tried successfully are in the posts.

The BizTalk product team is looking for your feedback!

EldarM — Thu, 02 Feb 2006 22:36:00 GMT

Here is a little announcement that Nancy asked me to post for your attention:

The BizTalk product team is looking for your feedback! We are conducting baseline usability studies of BizTalk Server 2006 over the next several months (Feb – May) and are looking for BizTalk 2004 hands-on users (IT Pro/Developer/Business) to give feedback and help direct the vision for future versions of BizTalk! Your input and participation as we develop our products is extremely valuable to us and helps us ensure that our products address your needs. You will receive a free piece of Microsoft software (or choice of an MSPress book) for your time and feedback.

If you are in the Seattle/Redmond area and would like to participate, please contact nancyp@microsoft.com as soon as possible to get on the schedule!

Migration considerations for moving from MSMQ/T to MSMQ adapter in BizTalk 2006

EldarM — Mon, 24 Oct 2005 23:42:00 GMT

This article describes the essential points to consider when migrating your solutions from MSMQ/T adapter to MSMQ adapter.

Why migrate to MSMQ Adapter?

Technically, you don’t have to migrate to MSMQ adapter. MSMQ/T is present and fully supported in BizTalk 2006. It is deprecated, but that only means that it may be not there in some next future version. Again, in BizTalk 2006 it is present and fully supported. Many customers may prefer just to do nothing and continue to use MSMQ/T adapter and avoid migration pain at this time.

Consider also the following question: Are you sure what kind of messaging you would like to migrate to in three-five years? Have you checked the new .Net 2.0 framework already available as Beta version? If you did, you may have noticed many exciting new features in the platform in the messaging area.

On another hand, if you are considering alternatives to MSMQ/T adapter anyway, MSMQ adapter may be a great choice. In this case, here is what you need to keep in mind.

Simple migration

Generally, MSMQ/T is a high-fidelity implementation of MSMQ server, and within provided functionality it is identical to MSMQ on the wire. In fact, it’s build out of the very Windows Message Queuing 2.0 code. So in simple cases migration to MSMQ is just that: reconfigure ports to use MSMQ adapter, period. Except the cases listed below, nobody should notice a difference.

In fact, if you have multi-machine installation behind NLB, you may be even able to migrate non-stop. Although, unless you have a very good reason for that, I would advice against that for the mere possibility of hiccups in any IT movement. Anyway, if you must, here is what you do:

Add a machine to BizTalk group with MSMQ adapter and without MSMQ/T adapter.
Create a host that uses MSMQ Adapter and has the twins of MSMQT receive and send ports with the same queue names. Run this host on the new machine (don’t start it yet).
Reconfigure your apps and send ports to look for messages coming from both old MSMQ/T receive locations and new MSMQ ones.
Start the new host and the new ports.
Take one machine in the old configuration, reconfigure it for the new one, get it back online.
Repeat it for all machines in the group. In the middle of the process, before you reconfigure the last machine, reconfigure applications and receive ports to send to new MSMQ ports.
You are done. If you’ve done it correctly, your partners should not notice a thing.

Again, unless you have a very-very strong reason to do that non-stop, I would advise against that.

End-to-end In-order delivery

MSMQ/T ensures end-to-end in-order delivery. That means that if an MSMQ application sends messages 1,2,3 to MSMQ/T port of BizTalk, then these messages are delivered to an orchestration in BizTalk in the same order 1,2,3. The cases when it is required are rare, but when it is required, it’s an absolute must-have. Examples include stock orders that stock brokers must submit and execute in the same order as they received them.

End-to-end in-order delivery seems like a simple statement, but behind the hood it includes a lot of components working together to reach this goal:

MSMQ API on the remote machine gets messages 1,2,3 in order and pushes them into a local queue in the same order 1,2,3.
MSMQ Server on the remote machine takes the messages from the queue and sends them in order 1,2,3.
MSMQ/T Adapter receives messages in order 1,2,3 and pushes them into the MessageAgent in the same order 1,2,3.
MessageAgent ensures that they get into the MessageBox in order 1,2,3.
MessageBox routes them and makes sure that if they are routed to the same instance of an orchestration or a send port, they are delivered to this instance in the same order 1,2,3.

In BizTalk 2004 MSMQ/T is the only adapter capable of that. For every other adapter at least steps 3 to 5 may occasionally switch the order. Step 3 for most other adapters is done by the component called End Point Manager, which is inherently multi-threaded. MSMQ adapter for BizTalk 2004 has a feature called “Serial Processing” that will ensure in order processing for step 3, but it does not asks MessageAgent to keep the order going forward, so the messages may be routed to an orchestration in a different order.

So, rule #1 for end-to-end in-order processing is: Use BizTalk 2006 (unless you stick with MSMQ/T).

In BizTalk 2006 receive ports in orchestrations and send ports has Ordered Processing property. If it is set, it means that the send port asks MessageBox to deliver messages to it in the same order it was submitted into the MessageBox. This flag ensures steps 4 and 5.

Also, MSMQ receive ports have Ordered Processing property, which means that they should not mix up message order and they should submit the messages in the same order as they were received from MSMQ. That ensures step 3.

Also, the actual MSMQ queue must be in order, that is transactional. That will ensure steps 1 and 2.

Now, in short, to get MSMQ/T-style in order delivery in BizTalk 2006 with MSMQ adapter, you must:

Set Ordered Processing property on the destination orchestration or send port.
Set Ordered Processing property on MSMQ receive port.
Make MSMQ queue transactional.

An important notice to keep in mind is that because of the local storage, the order may be only guaranteed if each queue is serviced by only one machine. That sometimes represents a problem for scalability, as described below.

End-to-end transactional

There is one key difference at how MSMQ/T and MSMQ adapters process a message.

For MSMQ/T, getting a message from network, processing it through pipelines, pushing it into the MessageBox and routing is a single transaction. When the sender receives ack, it means that the message is already routed to the destination.

For MSMQ Adapter, receiving message from the network and getting it into BizTalk are two different transaction. The sender receives ack whenever Windows Message Queuing server gets the message from the network and pushes it into the local MSMQ queue. There could be no BizTalk on the machine at all, the sender still will get ack. Then MSMQ Adapter tries to pick it up, process and publish. If this process fails, the message gets into the suspended queue or get stuck in the local queue (for transactional processing), and the sender has no indication that something’s got wrong.

In most cases you don’t care about this differences, but if you do, you really need to know that. In which case you have at least two options:

Add application level acks. You will need to update your orchestration and sending application for that.
Consider the question in the beginning of the article. You may be better off by just waiting for the time when you will need to migrate. There could be better options available at the time.

High Availability (Transactional)

High availability in MSMQ/T is easy. You just put it behind NLB, and if one machine is down, another picks up the work. For MSMQ Adapter NLB does not work if you need transactional processing with no data loss, because there is an intermediate storage – local MSMQ queues. What happens is that if a message already delivered to a local MSMQ queue, but not yet consumed by BizTalk/MSMQ Adapter, and the machine crashes, then the message is lost.

You solve this problem by NT clustering. You should put two BizTalk machines in an NT cluster, configure BizTalk process as a cluster resource, configure MSMQ to keep queues on the shared cluster disk (RAID or SAN). Then if one machine crashes, another takes over. And if the physical disk crashes, RAID or SAN takes care of that.

Important thing to know is that you only can do that in BizTalk 2006. BizTalk 2004 host process cannot be registered as a cluster resource, and hence clustered MSMQ queue will be considered remote by MSMQ, in which case transactional read does not work.

If you followed me, the actual key problem is availability of a transactional remote read on MSMQ. If it would be there, you will just use it and thus eliminate the local storage, which is the root of the problem. Funny enough, there are two incarnations of MSMQ that have remote transactional read. Less funny is the fact that both of them are not supported. Here they are:

MSMQ Dependent client allow transactional remote read. Technically, there is nothing to prevent you from using it, it’s the same API and MSMQ Adapter and BizTalk should not be able to detect any difference. Unfortunately, it’s deprecated and phased out now, so it’s not officially supported to use with BizTalk and MSMQ Adapter.
Upcoming Windows Vista will have a new version of MSMQ (4.0) with the remote transactional read. If you tried a preview copy, you may already know that. Unfortunately, BizTalk 2006 is going to be released way before Windows Vista, so there is no way for us to test that on the final bits. Again, technically BizTalk will not notice a difference. If you configure a receive location as both remote and transactional, today “transactional” will be ignored by MSMQ, on Windows Vista it will be transactional. Both ways BizTalk and MSMQ Adapter cannot see any difference. But from the support point of view, if it is not tested, it is not supported. We may be able to do something once Windows Vista is released, but we cannot make any promises for now.

Anyway, you have an NT cluster solution which will work today.

Just as an exercise, what if you want in-order delivery AND high availability? The answer is: you do steps 1-3 above AND you use NT cluster.

High Availability (Non-Transactional)

If you don’t care about transactional send/receive with in order delivery the picture is much simpler. In MSMQ/T you had it transactional because that’s the only thing that MSMQ/T is capable of. But if you don’t care about that, you can use MSMQ adapter. Just configure MSMQ ports for express delivery, put MSMQ behind NLB – the instructions are pretty much the same as for MSMQ/T and generic MSMQ – and you are done.

Scalability (non-transactional)

In MSMQ/T, scalability is achieved by using NLB.

Now, here is how to scale out MSMQ Adapter.

Again, non-transactional is easy. You just put it behind NLB and configure it according to MSMQ NLB guidelines. Messages get delivered to different local queues and picked up in a completely random order, but you don’t care about. And in return you get a blinding fast delivery and good scale out.

For non-transactional story, that’s it.

Scalability (transactional, not in-order)

Technically, there is no such thing as transactional not in-order delivery in MSMQ. However, if you configure MSMQ queues for transactional processing and scale it out with NLB, BizTalk may break the order in failover scenarios. What’s more, in failover scenarios picking messages up may break the transactional part. Imagine that a machine received messages 1,2,3 and had a disk failure. In which case, MSMQ will confirm to the sender receipt of messages, but BizTalk will never pick them up. To counteract that you will need also use NT cluster on the machines with MSMQ queues and BizTalk, which will make sure that the messages are not lost in failover.

So, to have it transactional, you will need to combine MSMQ scale out using NLB with NT cluster. Notice that you don’t get in-order delivery in this case. See below why. But if you want transactional processing and don’t care about the order of messages, that’s the way to go.

Scalability (transactional, in-order)

MSMQ scales all right, and the message will come to MSMQ queues in order. MSMQ adapter will pick them in order and the rest of BizTalk 06 machinery will ensure that the order is preserved. However, between MSMQ queue and MSMQ adapter pick-up there is a break point in order. If TCP/IP connection for MSMQ session is broken, then it may be restored by NLB to a different machine. Then messages 1,2,3 will come to machine A, and 4,5,6 to a machine B. Both sequences will be in order. But the instances of MSMQ adapter on the machines A and B, will not be able to synchronize them between each other. More to that, if the connection is broken because machine A went down, it may be still down when messages 4,5,6 will arrive to the machine B. Then on machine B MSMQ adapter instance will pick it up, process, and afterward machine A will complete the failover and start to push messages 1,2,3, with the resulting order of 4,5,6,1,2,3 in the MessageBox.

A partial solution would be to load balance it manually by allocating destination queues to different machines. It can be done by using separate BizTalk hosts through configuration.

If this is not acceptable then to solve this problem with MSMQ adapter you need a remote transactional read, which is the only existing alternative to MSMQ/T capabilities. And MSMQ remote transactional read is not available yet. So you may be better off sticking to the existing MSMQ/T solution until MSMQ remote read will be available and supported by BizTalk. Again, consider the question in the beginning of this article.

In bullet points

Simple migration: just reconfigure the ports.
End-to-end In-order delivery: Set destination, receive location and MSMQ Queue for in-order processing
End-to-end transactional: Use application level acks or consider postponing migration.
High availability (non-transactional): Use MSMQ behind NLB.
High availability (transactional): Use BizTalk 2006 and NT cluster.
Scalability (non-transactional): Use MSMQ behind NLB.
Scalability (transactional, in order): Manually distribute queues between receiving machines or wait until MSMQ remote read will be supported.
Scalability (transactional, no order): Use BizTalk 2006, MSMQ behind NLB, and NT cluster.

Short history of MSMQ Adapters in BizTalk

EldarM — Tue, 27 Sep 2005 23:01:00 GMT

In the beginning: MSMQ AIC

In the beginning there were no adapters. In BizTalk 2000 and 2002 they were called AICs – Application Integration Component. MSMQ AIC for BizTalk 2002 used MSMQ API from Windows Message Queuing and that was it. Windows Message Queuing was doing the network related job and it also stored intermediately messages in MSMQ queues on the machine before they were read by MSMQ AIC of BizTalk or sent to the world over the network.

It was simple and straightforward, but it had a few problems on its own:

Reliability: once the message came to some machine and before the message was picked up by BizTalk, the crash of the machine would destroy the message. Not the best story.
Double administration: you had to configure MSMQ queue and separately you had to configure MSMQ AIC for BizTalk. Double storage also meant double trouble. You had to check two places with two different tools in the search of a lost message.
Double hop: an incoming message was first stored in MSMQ queue on the disk, and then had to be read and stored in BizTalk database. This did not look like a good idea from the performance point of view.
Transactional story. Suppose you send a messages to a partner application. In MSMQ you have eventually check the dead letter queue to see if it was delivered or not. That was inconvenient.

BizTalk 2004: MSMQ/T

These and few other reasons were considered when making the decision to create MSMQ/T Adapter for BizTalk 2004. MSMQ/T Adapter is a full implementation of MSMQ Server that lives in BizTalk. When BizTalk 2004 starts, it also starts a subprocess that listens to port 1801, the port for MSMQ protocol. MSMQ/T adapter is based on Windows Message Queuing server 2.0 and it is actually done by the same people who implemented Message Queuing in Windows.

Now a lot of things become much nicer:

No local storage. Hence no separate administration, no extra cracks where the message can fall.
No local storage. Hence no extra hop. Message goes straight into BizTalk database.
No local storage. Hence no need to cluster BizTalk machine. If the machine crashed before message is safely in the BizTalk database, no ack is sent back and MSMQ protocol takes care of resending the message.
No local storage. Hence no need to check the dead letter queue. You can route acks and nacks straight back to the BizTalk applications. Your orchestration sends a message and waits in a try-catch block. If something goes wrong, you catch an exception and know that the message did not make it through. Of course, it takes a bit of time to get ack/nack over asynchronous protocol, but orchestrations are stateful long-running applications, they can wait.
And, irrelevant to the local storage, but a great feature: large message support. MSMQ messages are limited to 4 Mb in size. MSMQ/T is able to break a large, say, 100Mb message, into the 4Mb chunks and reassemble them into a single message on another side. MSMQ/T large message support is fully streamed and hence, technically, there was no limit to the size of a message, although if you used XML pipeline, you could hit a problem with XML parser in .Net 1.1 around 2Gb. We successfully passed >1Gb message in out tests, but for support purposes we’ve put more reasonable lower limit.

MSMQ/T also had a few limitations:

· It was based on 2.0 code and hence it did not had most of the 3.0 features, including message queuing over HTTP and few others. When MSMQ/T work had started, MSMQ 3.0 code base was simply not yet developed.

· It implemented only transactional, exactly-once-in-order (EOIO) delivery. The thinking was more or less along the lines, “If a user does not need transactional, why does he need message queuing?”

· No remote read from MSMQ/T “queues”. The reason was that while BizTalk receive locations played the role of a logical MSMQ queue, there was no physical queues in MSMQ/T. Once the message was received, it was at the same time in the same transaction routed to destinations, hence BizTalk “queues” were by definition always empty.

The advantages of MSMQ/T had their trade offs:

Now that BizTalk become MSMQ Server listening on the port 1801, running Windows Message Queuing on the same machine become somewhat difficult. If you had Windows Message Queuing running on the machine, BizTalk would fail to start because of a conflict on the port. And once BizTalk starts, Windows Message Queuing was not available. As we found in Beta programs, many users wanted to run regular MSMQ applications on the same box, so it was a handicap. We addressed that with two solutions:
- First, we figured out the way to install BizTalk and Windows Message Queuing side by side, see here [1]. That required two IP addresses, two computer names, had a few limitations, but otherwise worked ok.
- Second, we stopped adding MSMQ/T adapter by default. It was still an integral part of the engine, but BizTalk admin had to explicitly “Add Adapter” in the admin console, for BizTalk starting the process. Hence, until you added MSMQ/T adapter, you were able to run Windows Message Queuing. The other side of the story was that once you added MSMQ/T you could not get rid of it other than by reinstalling BizTalk 2004 – a problem we addressed in BizTalk 2006.
MSMQ/T looked slower. When you send a message from MSMQ to MSMQ, it is delivered to the destination machine, dropped into the local queue, and that’s it. The interesting part of the message’s life just starts, but the queue on the sender machine is already empty, and it feels like the message was already delivered. With MSMQ/T the message have to go through BizTalk pipelines and then transactionally stored in BizTalk database along within a batch of other messages. In fact, while being stored in the BizTalk database, it was also transactionally routed to BizTalk applications and send ports interested in this message. It takes longer than just dropping a message into the local storage, hence users occasionally observed messages piling up in their outgoing queues on MSMQ machines.
In fact, MSMQ/T was slower even in the end-to-end scenarios. The reason was that combining two or three transactions in one is always more complex and slower than doing these transactions independently. MSMQ/T combined in a single act picking the message from the network, passing it through pipelines, submitting it into the MessageBox including routing it to internal BizTalk destinations all in order all the time. Combined into a single transaction with the network protocol designed for reliability, not low latency, resulted in a substantial performance toll.
Because MSMQ/T used transactional messages only, once you send from a single MSMQ machine to a single MSMQ/T queue, you could not scale. That is the same limitation as in regular MSMQ, but regular MSMQ allows non-transactional messages, which are not affected by this problem.
You still can scale if you have multiple senders or multiple destination queues, because you will have different TCP/IP connections that could be load balanced.
Users were confused. “Hey, I’ve send a message to MSMQ/T, I check the MSMQ queue on the machine, and I see no message! In fact, there is no queue either!” Well, sure, there is no Windows MSMQ queue, it’s in BizTalk, and it’s already empty, because your message is already routed. Anyway, that was a minor problem.
Granted, the previous problem was also rooted in the fact that BizTalk HAT is not a match fro MSMQ MMC administration console when it comes to message queuing. That made administration and troubleshooting of MSMQ/T complicated.
Last, but not the least, the only way to include MSMQ 3.0 functionality was pretty much rewriting MSMQ/T using the new code from Windows, which would be a major effort. By that time MSMQ team was already busy working on Indigo, and BizTalk team was busy on a lot of other things in BizTalk, so maintaining MSMQ/T up to the date became a major problem.

MQRTLarge.dll

As already mentioned, a serious advantage of MSMQ/T was the ability to pass large messages. However, while passing large messages between two BizTalk machines was an attractive feature, it did not looked like a really huge winner. What would be great is passing large messages between BizTalk and MSMQ applications.

The problem was solved by providing MQRTLarge.dll that exposed an API similar to regular MSMQ and allowed regular MSMQ applications to send/receive large messages to communicate with BizTalk.

MQRTLarge.dll was supposed to be dropped on a non-BizTalk machine and used by MSMQ applications, and hence it is located in BizTalk 2004 SDK utilities. Because the API was slightly different to accommodate for large messages, the users had to update the MSMQ application code and rebuild it to take advantage of this feature, but other than that it was really nice. It still is.

The only deficiency of MQRTLarge.dll is that it accumulates the whole message in memory and hence available memory was an effective limit of how large message you can pass with it. We were able to pass messages up to 2Gb in size, but it required a good machine. Also, if two applications would send large messages at the same time, their memory demands will combine, which was another problem. But anyway, to have this problem we talk about really large messages several hundred megabytes in size. Most users, we know of, needed to push 4Mb limit of MSMQ just a little to 5-20 Mb, and so they never noticed this problem.

MSMQ/C Sample

BizTalk 2004 SDK also contained an MSMQ/C adapter sample that used standard Windows Message Queuing just like MSMQ AIC for BizTalk 2002. It was just a sample with limited functionality, it was never tested extensively for performance, stress or load, and it never was supposed to be used in the production environment, although we’ve heard of customers who did that. Again, it was just a sample.

MSMQ Adapter for BizTalk 2004 (see *here**[2]*)

Late in BizTalk 2004 development cycle we realized that customers will also need a traditional MSMQ adapter, and so we started to build it. At this time it was already customary to refer to “the other MSMQ adapter” as MSMQ/C, that’s why this adapter also sometimes informally called MSMQ/C Adapter. I actually did that in this blog before. However, it is not the same thing as the sample. We’ve put roughly a year of development and extensive testing into creating it.

MSMQ Adapter for BizTalk is available as a free download from Microsoft.com Download Center [3].

It has roughly the same advantages and the same problems as MSMQ AIC for BizTalk 2002, but provides richer functionality. In particular, it supports large messages using MQRTLarge.dll. The 2Gb message mentioned above was successfully passed through MQRTLarge.dll when testing MSMQ Adapter. The adapter has the same limitations as MQRTLarge.dll, and to avoid processing of two large messages at the same time, there is a “Serial Processing” flag on MSMQ port properties.

MSMQ Adapter is the recommended way to use MSMQ transport with BizTalk.

Here are some advantages of MSMQ Adapter over MSMQ/T:

Standard Windows Message Queuing is used with all features of MSMQ 3.0.
Other applications using MSMQ will run on the same machine concurrently with BizTalk.
Availability of MSMQ administration and troubleshooting tools.
Most operations are essentially local, and hence much faster. The adapter just drops a message to a local queue or gets one from there. For remote read Windows Message Queuing API uses an RPC without all complexities and delays of MSMQ protocol. It still takes time to deliver the message, but it happens asynchronously to BizTalk, while the visible processing time is comparable to a File adapter.
The adapter is compact, lightweight and functionally rich.

Things you need to keep in mind when selecting MSMQ Adapter for your project:

High availability will require an NT cluster. Even so, you will need BizTalk 2006 to utilize it.
Ordered delivery in BizTalk 2004 is not achievable with MSMQ adapter (standard feature in MSMQ/T). It’s available in BizTalk 2006 with some configuration work.
If migrating from MSMQT, see the detailed upcoming article on possible caveats.

Going forward – BizTalk 2006

In BizTalk 2006, MSMQ Adapter is the main recommended way to handle MSMQ transport. BizTalk 2006 supports NT cluster, hence there is a solution for high availability. Also, in BizTalk 2006 you can setup MSMQ port for the end-to-end ordered delivery (you need to set correctly MSMQ port, MSMQ queue, and the receiving application/send port).

MSMQ/T adapter is deprecated in BizTalk 2006. It means that it is still there and fully supported, but it’s a warning that it may disappear in the next version of BizTalk beyond BizTalk 2006. So, it may be not a good idea to consider it for new projects, unless you have a really strong reason.

MQRTLarge becomes a part of MSMQ Adapter and hence supported the same way as the adapter itself. Notice, that although you technically can make two MSMQ applications exchange large messages, the only officially supported scenarios are where one or both communicating sides are BizTalk MSMQ or MSMQ/T Adapters.

[1] http://blogs.msdn.com/eldarm/archive/2004/10/13/241799.aspx

[2] http://www.microsoft.com/biztalk/evaluation/adapter/adapters/msmq/2004/default.mspx

[3] http://www.microsoft.com/downloads/details.aspx?FamilyID=CBA87D07-7F50-4D7B-A888-388D123F736E&displaylang=en

How MSMQT behind NLB works

EldarM — Thu, 08 Sep 2005 23:03:00 GMT

MSMQT supports only one mode for MSMQ protocol -- transactional exactly once in order. Now, how MSMQ does that. The sender sends a TCP request for the destination queue to the port 1801 and establishes the connection. Then all the messages to this queue go through this already established connection. If the TCP connection breaks, the sender establishes the connection again.

If the receive side is actually a set of machines behind NLB, then when the sender sends the request for TCP connection, it is randomly assigned to one of the the machines behind NLB, and then it is a connection between the sender and this particular machine, at least until the connection breaks for some reason. Hence, while a single sender sends to a single queue, only one machine in the group services it, other machines cannot participate. But if the machine goes down, another machine can pick up the work. Because for BizTalk MSMQT the storage is common, if a connection breaks, another machine can get it when the senders reestablishes the connection.

Now, if different sender or different destiation queues are involved, new TCP connection is established for each, and NLB will assign new TCP connections to different machines, so in this case there will be load balancing. Sender A may be connected to the receiving machine 5, while sender B to machine 3, and sender C to the machine 4. So, that would provide scaling if enough senders and destination queues are involved. Still if one of these connections breaks, another machine may be used to reestablish it by NLB when the sender retries.

Again, so far there is pretty much nothing BizTalk-specific. BizTalk just implements that with fidelity, nothing more.

See http://blogs.msdn.com/eldarm/archive/2004/09/17/230951.aspx on what BizTalk does for that to work.

A better way to get Beta1 of BizTalk 2006

EldarM — Wed, 27 Jul 2005 00:44:00 GMT

Go to http://www.microsoft.com/biztalk/evaluation/bts2006beta.mspx

BizTalk 2006 Beta1 is available...

EldarM — Fri, 22 Jul 2005 17:24:00 GMT

... from https://beta.microsoft.com

If you are not registered there, you'll have to pass through sign-off approval process first, because that's the generic place where Microsoft products betas are available.

Notice that once you logged in with your Passport ID, you need to use (on the second screen) GuestID BizTalkBetaTeam to get straight to BizTalk 2006 Beta 1 form. If you use the Guest ID published on the passport login screen of the Beta place, it will take much more steps to get there. More complete instructions were posted in the Beta newsgroup.

BizTalk 2006 Beta 1: You can remove MSMQT Adapter now

EldarM — Thu, 21 Jul 2005 22:03:00 GMT

Just wanted everybody to know, that in the upcoming Beta1 of BizTalk 2006 it will be at last possible to remove MSMQT adapter just like any other. You still will have the code on the machine (it's actually part of the core engine and cannot be separated), but you will not see it in confgiuration, it will not lock the port 1801, and you can run MSMQ (a.k.a. MSMQ/C) adapter without configuring it side by side.

You still should be careful doing so. If you remove MSMQT adapter with some messages still waiting for the delivery, and later change your mind, you must remove those messages manually before you decide to add the adapter back.

MSMQ Adapter on an NT cluster in BizTalk 2006

EldarM — Fri, 08 Jul 2005 18:28:00 GMT

Some of you may already have a preview version of BizTalk 2006 from TechEd. If so, you may try using MSMQ Adapter on NT cluster. You need to make BizTalk process a cluster resource (there is a checkbox in Admin for that) and have MSMQ service on the same cluster.

In the preview bits that you have there is a trick you need to keep in mind: the clustered Biztalk resource must have dependency on Computer Name resource and enable Use Network name as computer name as check boxes to make it work. Otherwise MSMQ queue will be considered remote and you will not be able to use it transactionally.

Really large messages with MSMQT and MSMQ (MSMQ/C) Adapter

EldarM — Fri, 03 Jun 2005 21:26:00 GMT

When it comes to really large messages – and I don’t mean merely 50Mb – there is a difference between the old MSMQT adapter and the new MSMQ (a.k.a. MSMQ/C) Adapter for BizTalk.

The new MSMQ Adapter is using MQRTLarge.dll that was shipped originally with BizTalk 2004 to enable communicating large messages (exceeding standard 4Mb limit of MSMQ) to BizTalk. It works like this: you put MQRTLarge.dll on the machine with regular MSMQ, and you change your application to work with MQRTLarge API instead of the regular MSMQ API. They are very much alike, almost identical, except that you link against a different DLL and now you magically can pass messages more than 4Mb. Everything else is done under the covers. The message is broken into parts, the parts are delivered in order, and assembled on another end. Physically, you can even make it work between two MSMQ-only machines without BizTalk. It’s just not an officially supported way, because we shipped MQRTLarge just to enable communicating with large messages with BizTalk, that’s it.

There is one caveat though. Because MQRTLarge.dll API is basically the same as MSMQ API, you have to accumulate the whole message in memory, and if you want a very large message (say, 500Mb), you need a lot of memory. That puts effective limits on the size of the message you can send with MQRTLarge.dll. It’s a way higher than the old one of 4Mb and it can be improved even more with an extra memory, but it is still there.

So, if you use the new MSMQ Adapter (MSMQ/C), which is based on general MSMQ and MQRTLarge, you get this limit. Again, it’s really high, I successfully passed 100Mb messages on a machine with less than 800Mb of memory, but the limit is there.

In contrast to that, with MSMQT all message processing is streamed, so technically there is no limit. Of course, we put a maximum supported size into the docs (somewhere around 200Mb), which simply means that we extensively tested our ability to work with messages up to this size and hence we feel that this is what we can provide the support for. However, technically, we were able to pass 1Gb messages through MSMQT.

Now, an interesting question: do we really need 1Gb messages today?

As I mentioned above, it’s not “officially supported”, but people do non-supported things when they see them work. On another hand, a system that passes 1Gb messages will need a serious bandwidth, huge storage and a lot of processing power. See for yourself, an average small server with 200-300 Gb disk will be able to hold about a hundred of such messages or less (keep in mind: if it is supposed to really process them, extra copies will be created.) That’s not too much.

Also, what could you possibly have of such size? In most cases, it looks like a really wasteful design to send such large messages. Of course, I can imagine movies routed to subscribers this way, but why would you use BizTalk (or any other integration product) for such routing – there is no data conversion, right? Similarly, why would you use MSMQ? MSMQ as any asynchronous transport is designed to primarily decouple send and receive for unreliable networks. If you send a high-definition movie, you need a reliable network.

Another scenario I’ve heard of, is moving the whole project documentation through the system as a huge zip-file. This also does not sound like a great idea, because the same thing may be done with SharePoint Server holding the documentation, and only small work items (like request, review, approval) floating through the system.

So I wonder, if anybody really uses BizTalk for processing of 1Gb messages? If you do, can you let me know?

From the mail: BRE (Rules Engine) question

EldarM — Fri, 08 Apr 2005 01:47:00 GMT

I am not Rules Engine guy, but I've checked with them, and here it is:

Hi, how could I implement a rule like this in the BRE of BiztalkServer2004 (not splitting it, into separate rules):

1)
IF (condition1)
THEN (action1)
ELSEIF (condition2)
THEN (action2)

Answer: Has to be separate rules:
IF (condition1)
THEN (action1)
IF NOT(condition1) AND (condition2)
THEN (action2)

2) IF (condition1) THEN
         IF (condition2) THEN (action2)
         IF (condition3) THEN (action3)
   END IF

Is there any way to call a rule within another one?

Answer: No, one rule cannot directly call another rule.

IF (condition1) AND (condition2)
THEN (action2)
IF (condition1) AND (condition3)
THEN (action3)

OR

IF (condition1)
THEN Assert/Update(some object that is used in conditions 2 and 3)
IF (condition2)
THEN (action2)
IF (condition3)
THEN (action3)

New links...

EldarM — Wed, 30 Mar 2005 15:24:00 GMT

Added "BizTalk Gurus" category today for people with BizTalk blogs. Not too many links so far, I certainly missed some great blogs that I've seen and now cannot recall where, so if you have a blog on BizTalk and wnat me to place a link to it, let me kow.

Here is what I added there for now (in alphabetical order):

Alan Smith from Stockholm is maintaining the growing Bloggers Guide to BizTalk
Andy Morrison: just a good blog with interesting notes.
Arch Hacker: another good blog from UK.
El Grego: good blog, really catched my eye with posts on WMI writing and MSMQT ACKs.
Stas Kondratiev: a BizTalk blog from Moscow. BTW, there are very interesting BizTalk deployments in Russia.