MSMQ from the plumber's mate

Authenticating MSMQ messages between forests

If you try to send authenticated messages between machines in different forests, you will see them end up in the Transactional Dead Letter Queue (assuming you enabled source journaling). This is because authentication uses certificates that are stored in Active Directory but forests - even those with two-way trusts - cannot share this information. So when a message arrives at a destination, the recipient tries to use the certificate to look the corresponding user account up in its local Directory Services but is unable to find a match. MSMQ then sends a negative acknowledgement back to the sender saying "signature invalid".

As discussed below, the solution is to use external certificates generated by a CA.

Message Authentication
“Message authentication provides two services. It provides a way to ensure message integrity and a way to verify who sent the message. To request authentication, the sending application must set the authentication level of the message to be authenticated. When authentication is requested, a digital signature and a user certificate are attached to the message in addition to the sender's SID, which is attached to all messages in the sender identifier property (PROPID_M_SENDERID or MSMQMessage.SenderId) by default. An internal certificate generated by Message Queuing or an external certificate supplied by a certification authority can be used. By default, Message Queuing attaches the internal certificate for the user to the message. An external certificate can be attached directly by the sending application, or an internal or external certificate can be attached by Message Queuing at the request of the sending application using a security context structure specified in the message.”

This does require some extra coding to make use of the external certificate as MSMQ looks after internal certificates for you.

Take care copying commands from emails

Windows isn't always WYSIWYG and sometimes it will pretend.

I'm talking here about how different characters are displayed on the screen to make the layout look nice. If you are typing away in Word then you can sometimes spot hyphens, for example, being changed in width on the fly. The text reads fine but you need to remember that the underlying character has a completely different ASCII/Unicode number. This becomes very important when copying text from an Office document or email into a script or the command line.

Examples include:

Single-quotes

The single-quote character - actually the apostrophe - (' 0x0027) can be exchanged with the left single-quote character (‘ 0x2018) and the right single-quote character (’ 0x2019)

Double-quotes

The double-quote character (" 0x0022) can be exchanged with the left double-quote character (“ 0x201C) and the right double-quote character (” 0x201D)

Hyphens

The hyphen-minus character (- 0x002D) can be exchanged with soft hyphens (-­ 0x00AD), hyphens (‐ 0x2010), en dashes (– 0x2013) or em-dashes (— 0x2014).

So if you have a command that won't run without errors and it looks syntactically correct then just retype it with the keyboard - good chance it will start working.

0xC00E006A - Queued Components and MSMQ

If you are trying to get a queued component working and see an event like:

Event Type:        Error
Event Source:    COM+
Event Category:                (104)
Event ID:              4811
Date:                     19/05/2007
Time:                     11:12:13
User:                     N/A
Computer:          COMSERVER1
Description:

Server Application ID: {some GUID}
Server Application Instance ID: {some other GUID}
Server Application Name: Your_application_name
The serious nature of this error has caused the process to terminate.
Error Code = 0xc00e006a :
COM+ Services Internals Information:
File: d:\nt\com\complus\src\comsvcs\qc\listener\listener2.cpp, Line: 532
Comsvcs.dll file version: ENU 2001.12.4720.3959 shp

or you are trying to create your queued component class on the client (via the queue:/new: moniker) but BindToMoniker fails with 0xC00E006A (MQ_ERROR_UNSUPPORTED_OPERATION - The operation is not supported for a WORKGROUP installation computer).

The cause is the difference between how MSMQ is configured on the server and the client. Assuming the Queued COM+ Application was installed on a server where MSMQ was in "Active Directory Integrated Mode" then it uses public queues. If the client machine has MSMQ installed in "Workgroup Mode" then it can only use private queues. This error is expected when trying to make use of a public queue from a client in "Workgroup Mode".

One workaround is to reconfigure the Queued COM+ Application to use a private queue:

1. Change the MSMQ installation to "workgroup mode" on the server. To do this on windows 2003, uncheck the "Active Directory Integration" option under "Application Server | Message Queuing" in the Add/Remove Windows Components dialogue. 
2. Restart the COM+ system application. To do this on Windows XP/Windows 2003, run "net stop comsysapp".
3. Delete the existing MSMQ queues for the QC application. 
4. Uncheck the "Queued" option for the COM+ application, click apply.
5. Check the "Queued" option for the COM+ application, click apply.
   Now the queues will be recreated and this time the main queue will be private.
6. Re-export the application proxy .msi file, and reinstall this on the client.
   Now the client knows it should send messages to the PRIVATE queue.
7. If you want, you can now add back the "Active Directory Integration" option on the server.

Another workaround would be to set the client to run MSMQ in "Active Directory Integrated Mode" (essentially the reverse of step 1)

[[Thanks to Todd Foust]]

Monitoring empty queues in MSMQ

From the MSMQ FAQ (link on the left):

5.10      ....And why is it that I cannot see all my queues in System Monitor?

... a queue is only monitored if it contains messages or if an application explicitly opened it. Otherwise, the Message Queuing service does not keep any state information for a queue, and it is not available in System Monitor.

Note: The above information also applies to MQMgmtGetInfo API. This API can return state information only regarding active queues. If a queue is not opened by an application and it doesn't contain messages then this API cannot be used to get information on that queue. In that case, MSMQ on Windows Server 2003 will return the error “MQ_ERROR_QUEUE_NOT_ACTIVE” (0xc00e0004). On MSMQ versions that are older than Windows Server 2003 the error returned is “MQ_ERROR” (0xc00e0001).

The main reason for this is that creating data structures for queues that are empty is not an efficient use of resources - and some customers can have hundreds of queues which are empty most of the time.

Common workarounds are:

1. Run an application whose sole purpose is to keep all the queues open, forcing MSMQ to maintain performance data for them
2. Place a dummy message in the queues that cannot be read. This only works if the main application analyses the messages in the queue to ensure they are what it is looking for. If the application simply reads the first message off the top of the queue then this workaround can't be used.

Looking for HResults values, Win32 error codes and NTStatus values?

Reading my colleague's blog I noticed links to useful pages on MSDN:

• HRESULT Values
• Win32 Error Codes
• NTSTATUS Values

As you probably know, HRESULT values from MSMQ have the format 0xC00Exxxx.

In binary the first 4 bytes that is 1100 0000 0000 1110 which we can translate using the HRESULT link as follows:

Bit 0 - 1 means failure
Bit 1 - reserved (AKA I'm not sure)
Bit 2 - 0 means Microsoft-defined value (rather then customer-defined
Bit 3 - 0 means error code not an NTSTATUS result (so product-specific)
Bit 4 - not used if Bit 3 is zero
Bits 5-15 - 000 0000 1110 means the error code is for MSMQ only (FACILITY_MSMQ)

So that's my curiosity satisfied about where the numbers come from anyway.

Clearing up MSMQ certificates from Active Directory

Some people have found a problem where Active Directory contains too many MSMQ certificates for a particular user account, usually if that's the one they use for installing all their MSMQ machines.

As you know, MSMQ uses certificates for authentication and encryption of messages.

MSMQ generates a certificate for each machine on the first login by the user. The private part of this certificate is stored locally in the secure cert store. The public part of the certificate is stored in Active Directory as an attribute of the user object. Since we need a certificate per machine, this attribute will have as many certificates as the number of machines on which the user previously logged in. The digest is a hash of an individual certificate, and lets MSMQ quickly identify the user for a given certificate.

MSMQ stores all the certificates in a single binary stream (BLOB), and it may appear as a single entity but in fact it contains many individual certificates.
The digests are stored as multi-valued attributes, and you will see multiple entries if there are multiple digests. Typically, there are as many certificates as there are digests.

On the sender, if a message is authenticated, MSMQ signs the message with the private part of the certificate and provides the identity of the sender in the message along with the certificate. On the receiving side, MSMQ looks at the incoming message, picks out the certificate, generates the digest, and looks up the user object that contains this digest. It then matches the user’s identity in the message with the one found in AD, and verifies the signature of the message before finally deciding that the message is indeed authentic or not.

There are two attributes, mSMQDigests and mSMGSignCertificates.

MSMQ-Digests
An array of digests of the corresponding certificates in attribute mSMQ-Sign-Certificates. They are used for mapping a digest into a certificate.

MSMQ-Sign-Certificates
This attribute contains a number of certificates. A user can generate a certificate per computer. For each certificate we also keep a digest.

It is possible for a domain account to be used to log on to so many different machines (about 820 seems to be the magic number) that the blob gets too big and exceeds the JET limitation for non-linked multi valued attributes. Once you hit the limit you'll start seeing errors - 0x80072024 / ERROR_DS_ADMIN_LIMIT_EXCEEDED.

If you are in this situation then there is no programmable solution exposed through MSMQ.

There are, though, some workarounds with varying levels of "recommended".

Note - if you delete these certificates and digests, MSMQ will no longer find these digests/certificates/users in the AD and can no longer verify the authenticity of messages sent from these accounts. Queues that are marked as requiring authentication will not be able to accept incoming messages.

The slow and laborious but supported workarounds:

1)  in AD Users and Computers, find the User Object
    - right-click, bring up Properties
    - select the "Message Queuing User Certificates" tab
    - highlight the first certificate row
    - press Remove
    - repeat for all certificates.
 
An administrator can use this method to clear out certificates for many users, during the one login session.

2) when logged on as the affected user
 
    - Open Computer Management, Services and Applications, Message Queuing
    - Click Properties
    - In the Message Queuing Properties dialog box, click the User Certificate tab
    - under User certificates, click Remove.
    - In the Personal Certificates dialog box, click the applicable user certificate, and then click Remove. 
 
I recommend that, as far as possible, you use one of these methods to maintain MSMQ User Certificates.

The "quick and dirty, use at your own risk" workaround:

3) use Windows' LDFIDE command line tool. LDIFDE reads in an input file and makes changes to AD according to the instructions in the file. You need to create an LDIF input file using the correct LDIF syntax, to delete the msMQDigests and msMQDigests attributes for the affected users. LDIF files follow the standard syntax described in RFC 2849. The syntax as used for Microsoft Active Directory is exactly the same as used for any LDAP directory, such as Sun One Directory, OpenLDAP and Novell eDirectory (ie, it isn't some weird Microsoft-proprietary thing)
 
For example, to clear out all MSMQ Certificates for user "TestUser" in the "BigDomain.Org" domain, create a plain text file:
 
1) DelCerts.ldf
 
dn: CN=TestUser,CN=Users,DC=BigDomain,DC=org
changetype: modify
delete: mSMQSignCertificates
-
 
dn: CN=TestUser,CN=Users,DC=BigDomain,DC=org
changetype: modify
delete: mSMQDigests
-
 
Note especially the line with the hyphen "-"; this is required and must be followed by one blank line before the end of the file.
 
This file is then used as input to LDIFDE, like this:
 
C:\>ldifde -i -f DelCerts.ldf
Connecting to "PrimaryDC.BigDomain.org"
Logging in as current user using SSPI
Importing directory from file "input.ldf"
Loading entries..
2 entry modified successfully.
The command has completed successfully
C:\>
 
You could potentially create one file for ALL user accounts you want to clean up but it's probably safer to begin with doing one account at a time. As a batch-style command-line tool, it could potentially make many changes before you have the chance to pause it.
 
For your environment, you will need to change the Distinguished Name (DN) to the correct value for your users.

In AD, if an Object's attribute has no value, then there is no instance of that attribute assigned to the Object. So, a user who has never logged in to a MSMQ machine has no mSMQSignCertificates or mSMQDigests at all. The next time they log in to an MSMQ machine, then an instance of the attribute will be created on the user Object, and populated with the right value.

CAUTION: This procedure has not been formally tested and is not supported by Microsoft. You should make a full backup of all your AD data before you try this procedure and be ready to recover if necessary.

[[Thanks to Uday Hegde, Yoel Arnon]]

LSASS process can get into a spin with too many MSMQ messages

There's a new KB article out (although the hotfix was produced last year):

FIX: The Lsass.exe process may use a high percentage of CPU resources when you run Message Queuing on a domain controller with global catalog

The following is taken from the KB: 

SYMPTOMS
Consider the following scenario. You are running Microsoft Message Queuing on a local Windows Server 2003-based domain controller with global catalog (GC). Too many messages are sent to a public queue of Message Queuing. In this scenario, the Lsass.exe process may use a high percentage of CPU resources.

CAUSE
This problem occurs because sending messages to the public queue involves a GC query. For ordinary domain controller connections, the result of the DsGetDCName function is cached. However, for GC connections, the result is not cached. Therefore, all serverless calls that connect to a domain controller over the GC port incur the expense of the DsGetDCName function. This behavior causes high CPU utilization in the lsass.exe process.

So I had to look up what serverless meant. Apparantly:

"A serverless bind string is one that does not contain the name or IP address of the target server. Serverless binds are used to locate the best domain controller to handle the request."  

This is for convenience when some application needs to locate a domain controller occasionally. Unfortunately it doesn't work too well with performance-centric systems like MSMQ where minimising DC location activity is important. So the solution is to do a "server bind" if the domain controller is also a GC and then fall back to a "serverless bind" upon failure.

Sending encrypted MSMQ messages

MSMQ has moved out of private corporate LANs and now companies send messages over the Internet. Data security should therefore be top of the to-do list for anyone wanting to follow this route.

The first consideration is what do you actually need to encrypt? Are you only interested in ensuring that the message payload cannot be sniffed on the wire or must the content also be secure while in storage in the server's queues? This is important as MSMQ only natively provides the former.

Over-the-wire encryption

Using SSL
With MSMQ 3.0 and above came the ability to send messages using HTTP and HTTPS protocols. If you have an existing application then it will need to be changed so that the message addresses confirm to the different format (For example, DIRECT=HTTPS://URL_Address/msmq/PublicQueueName instead of DIRECT=ComputerAddress\PublicQueueName). 

The following are good reference material for configuring the application and webserver:

How to use Message Queuing 3.0 to perform secure Internet messaging over HTTPS in IIS

Configuring HTTPS Messaging for MSMQ 3.0

Using IPSEC

IPSec exists below the Transport layer so its encryption services are transparent to applications. This means you don't need to change how your MSMQ application works - if it can send messages successfully over the unsecured network it should perform as well when IPSEC has been enabled.

Here's the link to the TechNet resource for all things IPsec

Using MSMQ encryption

MSMQ has the functionality built-in to perform encryption using keys stored in directory services. Important note here - this means only domain members can send encrypted messages to each other. Cross-forest or workgroup-mode communication will not be able to take advantage of this feature.

The privacy level for the queue determines whether it accepts messages that are encrypted ("Body"), unencrypted ("None") or both ("Optional").

Setting this level, though, is only half the story as the application needs to ensure that messages are sent encrypted. Any unencrypted messages that arrive at a queue with body-level privacy will simply be discarded.

From the Motley Queüe blog:

"Let’s look first at what MSMQ encryption actually does. MSMQ messages can be encrypted by setting the Message.UseEncryption property if using System.Messaging, or PROPID_M_PRIV_LEVEL message property if using the native API.  When this property is set, the message body is encrypted using a key from the directory service when it is removed from the outgoing queue to be transmitted on the network.  At the destination machine, the message is decrypted and placed in the destination queue.  This type of encryption is a part of transport security — if the message were to be intercepted in transport, the contents would be safe."

Further reference material is here:

Message Encryption

MSMQ overhead when you use message security features

Using Application Encryption on the Message

To manually encrypt messages when there is no access to the directory service, you can choose to have the application encrypt the message body. 

From the MSDN article on Application-Encrypted Messages:

To encrypt a message body, the sending application must have an RC2 or RC4 symmetric key to encrypt the message body, as well as the public key of the receiving computer to encrypt the symmetric key. On the receiving side, the destination queue manager can decrypt the message only if the receiving computer is operating in domain mode.

Using Application Encryption on the Data

All of the above focus on protecting messages while they are on the wire. If you wish to ensure that the data carried by the message is secure end-to-end - even in the outgoing and destination queues - then you need to encrypt the data BEFORE giving it to MSMQ. An MSMQ message can carry any type of data so the fact that the information is an encrypted block of bytes won't matter - the queue manager will treat the message body just like any other unencrypted message.

This means that responsibility for encryption and key exchange lie fully with the application and its developer, not MSMQ. How to do that is a vast subject area in its own right and I'll leave that to the experts.

How big is an MSMQ message really?

Just a quick note about how MSMQ reports message size.

If you look in the Computer Management snap-in you will see a column marked "size". This is not the message size; it is instead the message BODY size. 

I have already populated the queue with 10 messages containing a 10-character payload in my example screenshot. As the 10 characters are in Unicode (double-byte), they take up 20 bytes instead of 10.

To get an idea of the actual message size, have a look in Performance Monitor:

As you can see, there are 10 messages taking up 2,440 bytes. So each message must be averaging 244 bytes - 20 bytes of which is the 10 character payload, 20-22 bytes for the label (again double-byte Unicode) and the rest is overhead.

To get an idea of what sort of things constitute this overhead you can look through the MSMQ API on MSDN or, more simply, have a look at what other column headings are available in Computer Management:

For example, I found after adding the various queue name lengths that both "recipient queue name length" and "destination queue format name length" were 44 bytes long (for the PUBLIC={GUID} queue name used by my application to send the messages).

If the actual message size is critical to your system then you could, if you really wanted to, track down and account for all the bytes in the message.

It is not possible to generalise about how big the overhead will be, though, without benchmarking it yourself with a wide range of representative messages.

Rule #1 for setting up the MSMQ-MQSeries Bridge

Every few months (or longer) I need to set up a test MSMQ-MQSeries Bridge environment.

And every time I forget the cardinal rule.

"Remember that IBM MQSeries is Case-Sensitive".

Yet again, last night, I had to go through the various management screens to correct all the references to the queue manager and modify the name until finally all the pipes went green.

Gah!

MSMQ is asynchronous on the inside too

There is a new Knowledgebase article out - although I use the term "new" with some reservation. The article refers to a hotfix that has since been included in service pack 2 which gives you an idea how old the content really is.

FIX: Message Queuing 3.0 still accepts incoming messages from the network after you successfully call the MSMQApplication.Disconnect method on a Windows Server 2003-based computer

So I'm not mentioning the KB article because you may need the fix as you will already have service pack 2 in place so would be a moot point (or not a moot point, depending how you use the English language).

What is instead interesting is why there is a problem. MSMQ is designed to shift messages as fast as possible between locations and everything else is a secondary function. So, in this article's case, a call is made to disconnect MSMQ. The application making the call immediately gets a "success" back as the queue manager accepts the request. Unfortunately that request will not necessarily be acted upon right away, depending on how busy the server is with incoming messages at the time.

An analogy would be with switching off a hose - it takes a while to turn the tap and stop the flow of water. In most cases that's not a problem but if we are talking about a fire hose then a lot of water can still pass through before the flow finally stops.

If, in the message queuing world, you are monitoring the health of the MSMQ server and are calling Disconnect to stop the volume of incoming messages reaching a particular level then that delay before disconnecting may be critical. There could be so many messages arriving per second that the server fills up and stops working (See this insufficient resources post for more detail on why) before the disconnection completes.

The resolutions are:

1. Call Disconnect earlier than you need (although this is a bit hit and miss)
2. Add the FastDisconnect registry value introduced in the KB article

Note that FastDisconnect is still not immediate (to allow any outstanding acknowledgements to arrive) but it is an improvement.

ComputerWeekly.com IT Blog Awards

	Thanks for everyone that voted for me in the ComputerWeekly.com IT Blog Awards this year. Unfortunately I was in a tough category - the winner was not only the best in the Company category but chosen as best blog overall. So congratulations to Steve Clayton.
Check out Steve's blog "Geek In Disguise" to see what the fuss is all about.

0x800288c5 trying to install MSMQ on Windows Vista

Understanding How You Use This Blog

Greetings Blog Readers,

My name is Ed Jolly, and I am a director in the Commercial Technical Support (CTS) organization at Microsoft. I am here to request a few minutes of your time.

We would like to learn more about blog readership through a brief survey. This is an opportunity for us to better understand what is valuable to you and what you would like to see in the future.

Below is a link that will take you to another website to complete the survey. Based on what we learn, we may request more feedback in future surveys like this.  When you open the survey, you will see a list of blogs that CTS engineers contribute to across many different products. We have not posted a listing of these blogs in the past, and I hope it helps you find other blogs that are helpful to you.

The blog survey is completely anonymous.

The survey period has now ended.

Thank you in advance for your time, participation and assistance.

Ed Jolly (edjolly@microsoft.com)

MSMQ and IPv6

As IPv6 becomes more prevalent in your environments, you'll need to make sure that you are running MSMQ systems that can actually cope with the new IP addresses.

There are (at least) three issues to watch out for.

1. The addressing of messages is not as simple.
   For example, an old school address of
   
         DIRECT=TCP:10.32.55.123\PRIVATE$\queuename
   
   would become
   
         DIRECT=TCP:[2001:4898:2a:1005:230:48ff:fe73:989d]\PRIVATE$\queuename
   
   Notice the use of square brackets.
   

2. Not all operating systems can work with IPv6 out of the box. Windows Vista and Windows Server 2008 are both fine but Windows Server 2003 has IPv6 disabled by default. Although you can enable IPv6, MSMQ 3.0 (that comes with Windows Server 2003) will still only be listening on the IPv4 IP address. In that situation, any MSMQ 4.0 machines will not be able to send messages to an MSMQ 3.0 destination using IPv6 addresses.
   

3. On a Windows Server 2008 cluster you may find MSMQ is only listening on an IPv6 address even though there is an additional IPv4 address configured. This is down to the slightly different times taken for each IP address to become registered in DNS. MSMQ will only listen on the IP addresses that are returned when it queries DNS for the network name being used. IPv6 addresses are slightly faster than IPv4 addresses to initialise and so are more likely to be in DNS when MSMQ comes looking.
   
   Workarounds:

• Don't use IPv6 addresses if you don't need them yet

• Recycle the MSMQ resource to force DNS to be queried (all addresses will have been registered in DNS by then)

• Change the Dependencies for the Network Name used by MSMQ from "<IPv6 address> OR <IPv4 address>" to "<IPv6 address> AND <IPv4 address>". The "OR" logic means the network name resource - and therefore MSMQ - comes on line when either of the addresses has initialised; the "AND" logic requires both addresses to be ready before the network name resource can come on line, giving more time for the DNS updates to complete.

Do check the event log first, though, as there may be some other explanation:

• MSMQ Event 2197 "Message Queuing failed to listen on the IPv6 protocol. Messages will not be accepted on IPv6. <error code>"

• MSMQ Event 2198 "Message Queuing failed to listen on the IPv4 protocol. Messages will not be accepted on IPv4. <error code>"

Reference

Windows Server 2003 IPv6