Motley Queüe

Combining LINQ with System.Messaging

Patrick Toolis of the MSMQ test team has written a great whitepaper on using LINQ (Language-Integrated Query) with MSMQ via system.messaging. It's twelve pages long with a number of screenshots and diagrams, so I'm not going to try to post it directly into the blog. Instead, here is a summary and a link to the paper.

Language-Integrated Query (LINQ) provides a way to embed SQL-like query statements into code which iterates over enumerable data. The library System.Linq provides an extension to C# and Visual Basic which allows traditional query syntax to be used in source code. Because the query is embedded in the source, it gets the advantage of compiler facilities such as type-checking, unlike queries in strings which are just text data to the compiler. MSMQ's system.messaging provides enumerators for a number of objects, allowing application writers to use LINQ when dealing with them, particularly messages in queues.

Another New Feature: Transactional Remote Receive

MSMQ has supported receiving messages from queues hosted on other machines ("remote receive") since the beginning. However, remote receives within a transaction have not been supported. That's a significant detail, and has generated a lot of customer requests, because for many applications transaction support is important. Or, to be more specific, the ability to put a message back into the queue if something fails while processing it is important. If the receive was done in a transaction, then the application can abort the transaction, the message reappears in the queue, and the application can try again later; if the application has failed entirely and the process exits, then the transaction is aborted automatically, and the message is not lost and can be retried when the application is restarted. If the receive was not done in a transaction, then at best the application is left unable to either process it or put it back, and at worse the message simply disappears when the application process exits.

If the application and the queue can be hosted on the same machine, that all works fine. However, in some larger customer scenarios, the actual processing of messages requires a farm of machines running the application to provide sufficient capacity, and the queue is hosted on another machine entirely. The application is now doing remote receives, which cannot be transactional, and that presents the serious application architecture problem I just described. Prior to MSMQ version 4, the best workaround was to make the farm machines MSMQ dependent clients whose supporting server was the machine hosting the queue.

With MSMQ version 4, dependent clients are being phased out for a variety of reasons, so the workaround will no longer be available after a certain point. However, it was a workaround rather than the desirable solution, and the MSMQ team has addressed the impending problem by providing the desirable solution: remote receives using transactions are now available, allowing you to implement your application architecture in the straightforward way. Strictly speaking, this feature isn't quite new, since it has been present since Windows Vista was released a year ago. However, the scenarios it is most useful for are more likely to run on Windows Server 2008, so it will probably attract more attention now.

Note that performing a remote transactional receive requires both the receiver and the queue host to be running MSMQ 4, which means either Windows Vista or Server 2008. It also requires MSDTC (Microsoft Distributed Transactional Coordinator) on both machines so that the transaction can be flowed between them. MSDTC itself is present and running by default, but some additional setup is required to enable it to coordinate cross-machine transactions.

Upcoming change in NACK behavior

Hello and welcome back from the holidays!

I will get around to describing how to use acknowledgements and deadlettering soon, but right now, those of you who are already using acknowledgements should know that there is a slight behavior change coming up in Windows Server 2008 and Vista SP1 for certain classes of negative acknowledgements (NACKs).

The classes involved are MQMSG_CLASS_NACK_ACCESS_DENIED (0x8004), MQMSG_CLASS_NACK_BAD_DST_Q (0x8000), MQMSG_CLASS_NACK_BAD_ENCRYPTION (0x8007), MQMSG_CLASS_NACK_BAD_SIGNATURE (0x8006), and MQMSG_CLASS_NACK_UNSUPPORTED_CRYPTO_PROVIDER (0x800B). The thing they have in common is that it is relatively easy for an attacker to craft a MSMQ message which will cause the destination machine to generate one of these NACKs. Depending on how the attacker sets the admin queue formatname in the message, these NACKs could pile up in an outgoing queue on the destination machine or they could spam some legitimate queue on that machine or elsewhere. It's not a very disastrous attack, but it is relatively easy, so in Windows Server 2008 and Vista SP1 these classes of NACKs are suppressed by default. The previous behavior, which was to generate these NACKs, can be restored by creating a DWORD registry value named PermitInsecureNacks under key HKLM\SOFTWARE\Microsoft\MSMQ\Parameters\Security and assigning the value 1 to it.

This behavior change can interact with the MSMQ authentication changes introduced in Windows Vista and described in an earlier posting, since the BAD_SIGNATURE NACK which characterizes some of the issues described there is now suppressed by default.

MSMQ 4.0 Performance Counters and the NetNameForPerfCounters Registry Key

MSMQ provides several sets of performance counters which query the state of the service and other objects (sessions, queues, etc.) and can be used in various monitoring systems. There are a few KB articles and blog entries explaining things you need to know when using the performance counters. This document explains what is new in MSMQ 4.0 performance counters and what you need to know when using the performance counters in cluster setup.

Before MSMQ 4.0 (Vista and Windows Server 2008), MSMQ performance counters were implemented based on PerfLib v1. In this model, a shared memory is created for data exchange between the MSMQ service and the Messaging Queuing Performance Monitoring DLL (mqperf.dll). The shared memory is named by the name of the computer under whose context MSMQ is running. Because PerfLib v1 is not cluster aware, remote query of the virtual server performance counters may not work as expected (http://support.microsoft.com/kb/267316). To solve this issue, a registry key NetNameForPerfCounters was introduced.

In MSMQ 4.0, MSMQ performance counter implementation is based on PerfLib v2, which doesn’t require a shared memory and a performance monitoring DLL. This migration enables local and remote query of performance counters by just specifying the server name; it is expected to work without using the registry key. However, two cluster-related issues were found during the RC testing period:

1.       If two or more service instances are running on one node at the same time, performance counter query does not work for either of them.

2.       Remote query of clustered MSMQ service counters from a down-level OS does not work as expected.

A decision was made to add back the registry key to partially solve the first issue. Because of the change in PerfLib v2, this registry key doesn’t retain the same behavior as before. The behavior difference is explained under different scenarios. In all scenarios, the performances counters are for services that are running. Some of the terminologies used in the explanation are listed below:

1.       Service – the MSMQ service

2.       Non-clustered computer – computer that doesn’t have cluster service installed.

3.       Clustered node - computer that hosts a cluster.

4.       Consumer – applications that call PerfLib APIs to query performance counters.

5.       Local – the consumer is running on the same physical machine as the provider (can be under different virtual server context).

6.       Remote – the consumer is running on a different physical machine than the provider.

7.       Down-level OS – Windows 2000, XP, 2003

8.       Virtual server – the Network Name resource an MSMQ resource belongs to.

Scenario 1: non-clustered computer, service is running.

In this scenario, the registry key is not needed. The performance counters are available for both local and remote (including down-level OS) consumers.

Scenario 2: clustered node, node’s service is running and all clustered MSMQ resources are offline.

Assume the node name is XYZ and the query is for XYZ. If the registry key is not set or set to XYZ, the behavior is the same as in scenario 1. Otherwise, no performance counters of any service are available to any consumer. In this scenario, WS08 has the same behavior as WS03.

Table 1: XYZ Counters - Only Node’s Service Running on WS08/WS03 Cluster

Consumer Key	Local	Remote, Vista/WS08	Remote, Down-level
Not set	Working	Working	Working
Set to XYZ	Working	Working	Working
Set to non-XYZ	Not working	Not working	Not working

Scenario 3: clustered node, node’s service is stopped and one clustered MSMQ resource is online (the corresponding service is running).

In the following discussion, counters are assumed for the running clustered service and the name of the corresponding virtual server is XYZ.

If the registry key is not set or set to XYZ, local query and remote query from Vista/WS08 machines are working. Local consumer doesn’t have to be running under the virtual server context. This is an improvement in MSMQ 4.0. Remote query from down-level OS is not working. If the registry key is set but not XYZ, no counters are available to any consumer. The following tables list the results for WS08 and WS03 with differences highlighted.

Table 2.1: XYZ Counters - One Virtual Server Running on WS08 Cluster

Consumer Key	Local	Remote, Vista/WS08	Remote, Down-level
Not set	Working	Working	Not working
Set to XYZ	Working	Working	Not working
Set to non-XYZ	Not working	Not working	Not working

 

Table 2.2: XYZ Counters - One Virtual Server Running on WS03 Cluster

Consumer Key	Local	Remote, Vista/WS08	Remote, Down-level
Not set	Working only when under XYZ context	Not working	Not working
Set to XYZ	Working	Working	Working
Set to non-XYZ	Not working	Not working	Not working

Scenario 4: clustered node, two or more services are running.

The services running can belong to the node or to virtual servers. Assume the counter that the consumer wants to query is for computer/virtual server XYZ.

If the registry key is not set, performance counter query for any running service is not working for any consumer. This is a regression from Windows 2003, where counter query is working when consumer is running under the same computer context as the service. If the registry key is set to XYZ, the result depends on whether XYZ is the node or a virtual server. If XYZ is a node, the result is the same as in scenario 2. If XYZ is a virtual server, local query and remote query form Vista/WS08 machines are working, but remote query from down-level OS is not working. This is another regression from Windows 2003. If the registry key is set but not XYZ, no counters for XYZ are available for any consumer.

Table 3.1: XYZ Counters - Two or More Services Running on WS08 Cluster

Consumer Key	Local	Remote, Vista/WS08	Remote, Down-level
Not set	Not working	Not working	Not working
Set to XYZ	Working	Working	Working only when XYZ is the node
Set to non-XYZ	Not working	Not working	Not working

 

Table 3.2: XYZ Counters - Two or More Services Running on WS03 Cluster

Consumer Key	Local	Remote, Vista/WS08	Remote, Down-level
Not set	Working only when under XYZ context	Working only when XYZ is the node	Working only when XYZ is the node
Set to XYZ	Working	Working	Working
Set to non-XYZ	Not working	Not working	Not working

 

Summary

The results can be summarized as:

1.       MSMQ 4.0 performance counters improved for single service scenario. If the consumer is running on a Vista/WS08 machine, no key is needed and the counter query is working even for clustered service.

2.       There are some regressions in multiple services scenario. No counters of any service are available when the key is not set. Setting the key will bring MSMQ 4.0 on parity with MSMQ 3.0 if the consumer is running on Vista/WS08 machine.

3.       Performance counters of a clustered MSMQ service are not available to any down-level consumers, regardless setting the key or not. This may become a serious issue for customer, and we are actively seeking solutions to it.

Another difference worth mentioning is that, on WS03 cluster, changing the registry key will change the performance counter provider without restarting all services, provided all consumers are shutdown beforehand. However, on WS08 cluster, all services have to be restarted for the registry key to take effect.

On WS03 cluster, the value of the NetNameForPerCounters registry key needs to be in capital letters. This is not required on WS08 cluster.

—Xin Chen
MSMQ Development Team

Determining Whether a Remote Queue Exists, Part 2A

Timeouts, acknowledgements, and deadlettering, working together, will help you deal with not just the situation where a remote queue doesn't exist, but all the other things which could go wrong along a message's way from sender to destination.

MSMQ has two timeouts, which can be set individually for each message: "time to reach queue" (TTRQ) and "time to be received" (TTBR). TTRQ controls how long a message can be in transit before arriving in the destination queue. TTBR controls how long a message can sit in the destination queue before it is "discarded". Both timers start as soon as the message is sent, so TTBR actually includes the transit time as well. TTBR also has priority: the message is "discarded" when the TTBR timer runs out, regardless of how much time may be left in TTRQ. (The act of "discarding" a message may include sending a negative acknowledgment (NACK) and/or deadlettering the message, which I'll talk about later.)

When would you set these timers? Well, suppose you wanted to send a stock price update every minute to an app which would display the current price. If there is a transmission problem, updates could build up on the sender. However, since the receiving app only cares about the latest price, all those intermediate updates building up on the sender are meaningless. Any message which hasn't reached the destination queue in one minute is about to be followed by a message bearing more recent information, so it can be safely thrown away. In this case, you would set TTRQ to 60 seconds. Now, once the message arrives in the destination queue, you don't know exactly when it may be received. If the application only checks the queue once per minute, the message could sit in the queue for another 59 seconds before it is actually received. If it sits in the queue for more than 59 seconds, then there should be another message arriving with more recent information, so you could set TTBR to 119 seconds (or maybe 125 just to allow a little extra margin). On the other hand, if there is a transmission problem, setting TTBR like that could leave the app in a situation where it has no information at all — perhaps you'd like to display an old price rather than nothing. Setting TTBR to 1800 (half an hour), for example, would let updates stay in the queue for a while, but if the receiving app is down, having a TTBR set means you will never have more than a known number of updates sitting in the queue, instead of having them build up indefinitely.

What do timeouts have to do with detecting errors? Timeout settings which are appropriate for your business requirements mean that each message has a definite lifetime, at the end of which you will either get an error indication or you will know that the message has been received, if you have applied the right acknowledgement and/or deadletter settings to the message. Knowing that you will always get a definite answer makes writing your application much easier.

Acknowledgements and deadlettering are how your application gets that answer, and I'll talk about them in part 2B.

Message timers documentation: http://msdn2.microsoft.com/en-us/library/ms705726.aspx

Be Prepared: Known Issues for Upgrading to Windows Server 2008

There are a few differences in MSMQ and IIS functionality between Windows Server 2003 (MSMQ v3.0) and Windows Server 2008 (MSMQ v4.0). This means that there are a few configuration changes you may have to make to your MSMQ system post-upgrade to make everything function the way it did on Windows Server 2003.

Issues with HTTP and transactional messages:

MSMQ’s transactional message support uses internal messages called “order ACKs” to coordinate between sender and receiver so that no user messages are lost. When transactional messages are sent via HTTP, the URLs used to send the order ACKs back to the sender sometimes contain the special character “+”. IIS 7 has a security feature in request filtering to disallow all double escaped characters, and this interferes with the return of order ACKs to the sender if the URL contains a “+”. This results in the transactional messages accumulating in the outgoing queue of the sender in those cases.  In order to prevent this issue, which may arise any time transactional messages are sent via HTTP, you can disable the double escaped character filtering in IIS7 for MSMQ’s virtual directory. This can be done using the appcmd tool like so:

%windir%\system32\inetsrv\appcmd.exe set config "Default Web Site/msmq" -section: system.webServer/security/requestFiltering -allowDoubleEscaping:true -commitpath:apphost

To Keep Multicast working:

On Windows Server 2003, MSMQ’s multicast support was part of the MSMQ core optional component: if MSMQ was installed, support for multicast messages was also installed, as was the PGM Multicast transport which it depends on, and the firewall did not filter PGM traffic. In order to tighten security for users who aren’t using this feature of MSMQ, multicast support is now a separate optional component, and it is not installed by default during an upgrade. If you intend to use multicast messages after an upgrade, you must explicitly install it the same way you would install any other MSMQ subcomponent. You can either do this through the UI (in Server Manager), or on the command line via ServerManagerCmd. See my previous post on Unattended Installation for Windows Server 2008 for more information on unattended installations.

—Jolie Boushey

 

This Post Intentionally Left Blank

It's a holiday weekend here. Happy Thanksgiving to all our US readers! Tune in next week for another exciting adventure.

Please Stand By

Justin is out of the country and I'm on the far side of the continent, so we're experiencing a small break in our posting schedule.

For now, a quick tip: MSMQ is a transport. It supports a few mechanisms for scanning through the messages in a queue, but they were never intended for high performance and there is no built-in way to search for a message by property values. If you are thinking about leaving messages in the queue and picking them out according to some selection criteria, you are probably setting yourself up for problems in the future. A better architecture to achieve the same thing is to write a little application that receives messages from the queue and stuffs them into a database, then select messages out of the database with the full power of SQL available to you.

Determining Whether a Remote Queue Exists, Part 1

One of the basic principles of MSMQ's design is that sender and receiver are decoupled. The sending application does not need to know whether the receiving application is running or even reachable under current network conditions: it can just send, and let MSMQ take care of getting the message to the destination at whatever time that's possible. (Note that the decoupling occurs within MSMQ, and that both of the applications are tightly coupled to MSMQ, requiring the local MSMQ service to be running in order to perform MSMQ operations.) This design principle determines how MSMQ behaves in various circumstances when a queue doesn't exist.

When sending, the local MSMQ service knows what local queues exist, so it immediately returns MQ_ERROR_QUEUE_NOT_FOUND, hexadecimal value C00E0003, if you attempt to open a nonexistent queue for send. However, opening a remote queue for send will always succeed, regardless of whether the queue or even the machine exists. This is where and how the decoupling between sender and receiver occurs — MSMQ has no way to tell the difference between a remote machine which is temporarily unreachable, one which used to exist but is never coming back, and one which just plain doesn't exist, so it makes no attempt to do so. It just allows all sends and goes through the same retry procedures to attempt to get the messages to where the sending application said they should go.

Receiving is a little different. If you open a queue for receive and the queue doesn't exist, you will always get error code MQ_ERROR_QUEUE_NOT_FOUND back regardless of whether the queue is local or remote. If the queue you're trying to open is local, then the MSMQ service knows whether the queue exists; if it's remote, then the local MSMQ service must contact the MSMQ service on the remote machine. As described above, decoupling occurs on the sending side, and the original designers chose not to include a second point of decoupling on the receiving side, so even remote receives are tightly coupled.

Getting back to the title of this post — in summary, there is only one place where this question can occur: when sending to a remote queue. In all other situations, the presence or absence of the queue is immediately obvious in the return code of the API call. When sending to a remote queue, this is a question that you probably shouldn't be asking, to be honest. If your application determines whether a remote queue exists before sending to it, then you have given up the decoupling which is a major reason to be using MSMQ in the first place. In part 2, I will cover timeouts, acknowledgements, and dead lettering, at least some of which you should probably be using anyway, since they not only handle the case where the remote queue doesn't exist, they also handle all the other things which could go wrong even if it does. And if you really do need to know whether a remote queue exists, I'll cover how to find out in part 3.

A Preview of Windows Server 2008 MSMQ Clustering

A very user friendly administrator UI tool for creating, configuring and managing MSMQ Clusters

While this isn’t really a new MSMQ clustering feature, it’s worth being mentioned first. If you have used Win2k3 Cluster’s Cluster Admin UI, you will find the new and improved UI tool, which can be launched from Start->Administrative Tools->Failover Cluster Manager, a no-brainer.

 

When creating a MSMQ cluster virtual server, the UI will create the virtual server with the resource group, all the necessary resources in the group including IP address, disk, network name, MSMQ and MSMQ Triggers, and their default properties and dependencies automatically. It saves you all the steps you would have to do yourself in Win2k3 Cluster. Another cool thing is that the UI first checks if MSMQ and MSMQTriggers are installed on all the nodes before creating the virtual server; if not, it will stop and pop up an error.

 

No more downloading MMCV.exe for managing MSMQ virtual server

Instead, you will find a link on the Failover Cluster Management UI called “Manage MSMQ”, as shown the picture below.

 

 

 

If you click the link on the active node, you will get the MMC plugin with MSMQ folder under “Services and Applications”. You can view public, private, and system queues.

 

MSDTC goes Active/Active

What does this have to do with MSMQ clustering, you may ask? How about this: each MSMQ resource group can have its own DTC resource and you can make your MSMQ resource depend on the DTC resource. That means you can guarantee that the DTC resource will come online before the MSMQ resource when a failover occurs.

 

—Jenny Zhou

MSMQ + System.Messaging + WCF + Free T-Shirts = One Incredible Conference

TechEd Developers 2007 is just a couple days away now and we're getting really excited to hear what you have to say.  If you use MSMQ, System.Messaging, or WCF Queues, please be sure to stop by our Ask the Experts booth so we can see what's on your mind and tell you about what's on ours. 

Also be sure to check out the new Hands-On-Lab designed specifically for MSMQ and System.Messaging users looking to learn more about WCF.  We've got a great step-by-step tutorial on building your first WCF Queued application as well how to use the prototype of a never-before seen channel that will change the way you think about MSMQ.

As if that weren't enough, we're also giving away FREE T-Shirts to the first 25 people to finish our lab and then stop by our Ask the Experts booth and tell us what they thought.  These shirts are part of a very limited edition run and are sure to be instant classics; make sure you get it on your todo list. 

So bring your questions, your comments, and your insight to the WCF Ask the Experts booth at TechEd Developers and we'll see you there!

Justin Wilcox
Microsoft Message Queuing (MSMQ) Test Lead

The Legacy of an Old Bug

Internally, MSMQ uses the Windows-provided CryptoAPI to do all of its encryption and hashing. Starting with Windows NT 4.0 Service Pack 2, CryptoAPI had a bug in its implementation of RC2 encryption in the enhanced (128-bit) provider: the keys were generated with an effective length of only 40 bits, not 128. This directly affected the ENHANCED privacy level of MSMQ's encryption feature, especially since those versions of MSMQ used RC2 by default, so most customers were probably using that rather than the unaffected RC4.

This CryptoAPI bug was fixed in Windows 2000 Service Pack 4, Windows XP Service Pack 1, and Windows Server 2003. At the time the fix came out, many MSMQ clients and servers were still running on unfixed version of Windows, presenting a compatibility problem: without some kind of special measures, fixed and unfixed versions of MSMQ would be unable to exchange messages using enhanced RC2 encryption because each side would see the key length generated by the other as incorrect and the decryption would fail.

For MSMQ on Windows 2000 SP4, a pair of registry values were added. One value controlled the effective length of keys used when encrypting messages with enhanced RC2, and the other value controlled whether to accept incoming messages encrypted with enhanced RC2 but having "short" (40-bit) keys. The defaults were to continue using 40-bit keys when encrypting and to accept them on incoming messages, thereby allowing transparent compatibility with MSMQ installations on unfixed machines.

For MSMQ on Windows XP SP1 and Windows Server 2003, a similar pair of registry values were added. On these operating systems, the defaults were to use 128-bit keys when encrypting but to accept 40-bit keys on incoming messages. Thus, these installations can transparently accept messages from unfixed machines but require configuration to send messages to them using enhanced RC2 encryption. These registry values and defaults are also present on Windows Vista and Windows Server 2008 but are less important, because MSMQ4 will always use RC4 when sending to earlier versions of MSMQ unless other registry values have been set to force the use of RC2.

It has been many years now since this CryptoAPI bug was fixed and hopefully all of your MSMQ installations are on fixed versions of the OS. If that's true, what can you do to ensure that MSMQ throughout your enterprise is really using 128-bit keys, thereby tightening up your security? (All of the registry values listed below are created under HKLM\SOFTWARE\Microsoft\MSMQ\Parameters\Security. The MSMQ service must be restarted for these values to take effect.)

• On Windows 2000 SP4 machines, create the registry value SendEnhRC2With128 and set it to 1. This will cause outgoing messages encrypted with enhanced RC2 to actually use 128-bit keys.
• On Windows XP SP1, Server 2003, Vista, or Server 2008 machines, look for the registry value SendEnhRC2With40 and remove it if present. If it is present and nonzero, it causes outgoing messages encrypted with enhanced RC2 to use 40-bit keys instead of 128.
• On all fixed machines, create the registry value RejectEnhRC2IfLen40 and set it to 1. This causes incoming messages encrypted with enhanced RC2 to be rejected if they have 40-bit keys.

One additional thought: before touching anything, search for the SendEnhRC2With40 registry value on Windows XP and above. If it is present with a nonzero value, that means that at one time that machine needed to send messages encrypted with enhanced RC2 to an unfixed machine. Thus, any machines which that one communicates with have an increased chance of still being unfixed and candidates for double-checking.

The original documentation of this issue can be found in item 4.20 of the MSMQ FAQ.

Automate Your MSMQ Installation

Two New Operating Systems Bring Two New, Exciting Ways to Install (or Uninstall) MSMQ

This week we talk about automated installation (and new in Vista and Windows Server 2008, uninstallation) of MSMQ via unattend files. The Optional Component setup story has changed a bit since Windows Server 2003, but have no fear, we’ve got the info you need to be ready for installing Message Queuing on Vista and Windows Server 2008, including unattend file samples!

The biggest change in Optional Component unattended installation from Windows Server 2003 is the file format. Windows Server 2003 used .ini files, while Vista and Windows Server 2008 use XML files. The other big change is that you can now uninstall MSMQ components via unattend uninstall, which was not possible with sysocmgr.exe on pre-Vista operating systems.

Operating System	Supported OC Installer	Unattend File Format
Windows Server 2003