MSMQ Information and Troubleshooting Blog

MSMQ Service Reports "Insufficient Resources to perform this operation" when trying to create a Private Queue.

2012-10-30T21:08:00Z

Problem:

Customer was trying to create a private queue on a Windows 2008 R2 Server and getting back an "insufficient resources to perform this operation" error when trying to create a private queue.

BackGround:

It has been 5+years since I have worked on a MSMQ Insufficient Resource to perform this Operation error. On Windows 2000 and 2003 these errors were pretty frequent and the root cause was due to storage size. So when I got this customer reporting this on Windows 2008 R2 Server I was a little surprised since storage size is not a really that big of a deal.

Cause:

After a little digging around, I found that the security on the server had been hardened. As a result of this, the MSMQ storage directory default security had been modified and had locked the msmq service out and the service was unable to access the storage folder.

Resolution:

We gave everyone group full access to the msmq storage and that resolved the issue.

MSMQ Service Hangs when trying to stop or start it on Windows 2008 or Windows 2008 R2 Server.

2012-06-04T18:34:59Z

Problem:

If you are running Microsoft Message Queue (MSMQ) on Windows 2008 or Windows 2008 R2 Server and the service hangs on startup or when stopping it and logs the following error to event log:

Event ID: 2084

The Message Queuing Service cannot start because a queue is in an inconsistent state. For More Information See Microsoft Knowledge Base Article 827493 at support.microsoft.com

Resolution:

After troubleshooting we found that the machine name for this server was greater than 15 characters. Netbios names can only be 15 characters or less. This was causing problems with the MSMQ Service as a result it was ending up in the inconsistent state as specified in the error. To fix this issue, we uninstalled MSMQ, renamed the machine to something smaller than 15 characters and then reinstalled MSMQ. This resolved the issue.

Biztalk 2010 Admin Server unable to Manage WCF Ports.

2012-06-04T17:58:07Z

Problem:

If we have a Dedicated Admin Server for Managing the Biztalk Environment then you may receive the following error when trying to Manage WCF ports:

Windows Communication Foundation Administration Tools

You will be able to manage the file Adapter and other Standard Adapters without any problems. Only the WCF adapters will give you the error above.

Cause:

The following Component was not installed.
Windows Communication Foundation Administration Tools

Resolution:

We were able to fix the issue by doing the following.

1. Close all instance of bts admin console
2. Go to Control Panel > Uninstall and choose BizTalk Server and in there modify the install to add Windows Communication Foundation Administration Tools
3. Once that was added, open a new instance of bts admin console and it works

Biztalk 2010 Receive location intermittently stops reading files from FTP server.

2012-05-17T22:50:52Z

Recently had a very interesting case. The biztalk 2010 FTP adapter would read messages but if there is a gap of 10-15min where the Server did not have files to give to the Biztalk FTP Receive Location, the Biztalk Service would report Error 500 - Failed to connect to FTP Server.

A network monitor trace basically showed that Biztalk was trying to connect to the FTP server using the old session information. But the server had already decided to discard the old session and did so without notifying the Biztalk Server. As an easy workaround we executed the QUIT command in the After Put Property of the FTP Receive Location. Though not the most efficient method but did the job.

MSMQ Management console on Win2k8 & WIn2k8 R2 Cluster not Visible

2012-02-09T19:26:47Z

Symptoms

The customer has a Win2k8 R2 Cluster with MSMQ Resource. They right click the MSMQ resource but MSMQ does not show up in the Computer Management console. We see the same thing with MMCV.

Resolution

Changed the MSMQ physical node service acct to run under Network Service and that resolved the cluster issue.

MSMQ 4 & 5 not receiving messages from Large Number of Clients

2012-02-09T18:41:00Z

Symptoms

Had a very interesting case this week. Customer had few MSMQ 2008 Servers forming the NLB. They had few thousand clients sending non transactional messages to MSMQ Queues using Direct Format Names. Everytime you restart the MSMQ on the server the client would be able to send messages and then stop. The messages would start building in the out going queue with state Waiting to connect. A netmon trace would show the Server sending resets to the clients and some times the reset would come from the server.

Cause

MSMQ service on the server caches the client certificate. If the number of clients exceeds the default cached size then the server tries to delete half the
number of certs based on age. Some problem here causes the threads to hang.

Resolution

There is an easy fix to this problem. All we do is increase the size of the cache so that we do not have to delete the certificates to make way for other
certificates.

HKLM\SOFTWARE\Microsoft\MSMQ\Parameters\UserCacheSize Dword Value to Number of client +100(for safety)

You can learn more about the registry key here.

MSMQ Performance degrades over time & MSMQ LQS folder is 100s of MB.

2012-02-09T17:54:09Z

Symptoms

MSMQ Performance degrades over a period of time. The MSMQ LQS folder is 100s of MB in Size. You have to delete
the Queue queue and recreated them to make this work.

Cause

There is some problem with the security descriptor as a result MSMQ keeps writing it over and and over again to the files in LQS folder.
This results in the file size growing large and the performance of the service goes down.

Resolution

The simple workaround to this problem is to set the queue security at the time of creation. When we create a
new queue, disable inheritance from parent and then give permissions to the appropriate account.

MSMQ Performance degrades over time & MSMQ LQS folder is 100s of MB.

2012-02-09T17:54:03Z

Symptoms

MSMQ Performance degrades over a period of time. The MSMQ LQS folder is 100s of MB in Size. You have to delete
the Queue queue and recreated them to make this work.

Cause

Resolution

MSMQ Installation on Windows 2008 R2 Fails with 0x80070643 - Fatal Installation Error.

2011-04-08T00:18:00Z

Recently I got a case where the MSMQ Service Installation failed with 0x80070643 - Fatal Installation Error. We reviewed the c:\windows\cbs\cbs.log

The CBS.log file did not have much to go with. It just repeated the same error 0x80070643 - Fatal Installation Error. I cleared the Event Logs and retried installation.

We saw several failures to start DHCP client Service. We found the service set as not started and was disabled. We set the service properties as automatic and started the service. Now when we try to reinstall MSMQ we got the real error message -2147024891 Access Denied Error.

Cause:

The Problem here is that the MSMQ Service run under the Network Service Account. During Installation if we are unable to start the service then the Installation Rolls Back. The Network Service Account permissions were restricted and hence the service was not starting up and thus rolling back the installation every single time.

Resolution:

To Resolve this issue add the local network service account in the Administrators group on the local machine. Once the service is installed and startup you can try taking it out of the Admin group then retry restarting the service and see if it works.

This may not work for all installation issues but if definitely something that is easy to try.

Installing HTTP Component with MSMQ 3.0 Fails With error 0x80070422

2011-02-25T20:47:05Z

On Windows 2003 Server if you are trying to install MSMQ with HTTP Component then the installation may fail with the following error "The application for the IIS extension with the path /LM/W3Svc/1/Root/MSMQ could not be created. hr = 0x80070422 "

To Resolve this all you need to do is make sure that COM+ System Application Service is started. If it is not started, then start it and also make sure that the startup type is set to Automatic.

HTTPS Messaging with Client Side Certificate fails with IIS error 403

2010-10-26T21:16:00Z

Symptoms

We have a Win2k3/Win2k8 Server. We are trying to send HTTPS messages to this Win2k3 Server. The Server Requires Client Side Certificates. The IIS log shows error 403.7 - Client Side Certs Reqd. Also if you enable deadlettering on the messages the messages end up in deadletter queue with HTTP error 403.

Cause

If a Win2k8 Server has this problem then the problem is that the Network Service Account under which MSMQ runs does not access to the Private Key in MSMQ Certificate Store. Here is the error logged in the System log on Win2k8 Server.

Log Name:      System
Source:        Schannel
Date:          8/18/2010 3:15:10 PM
Event ID:      36870
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      ComputerName

Description:
A fatal error occurred when attempting to access the SSL client credential private key. The error code returned from the cryptographic module is 0x8009030d.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
    <Provider Name="Schannel" />
    <EventID Qualifiers="49152">36870</EventID>
    <Level>2</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2010-08-18T19:15:10.000Z" />
    <EventRecordID>34419</EventRecordID>
    <Channel>System</Channel>
    <Computer>Computer Name</Computer>
    <Security />
</System>
<EventData>
    <Data>client</Data>
    <Data>8009030d</Data>
</EventData>
</Event>

Resolution

On Windows 2008 Server you need to execute the command at command prompt. The command below gives network service account access to the Private Keys for the Certificates.

certutil.exe -service -service -repairstore MSMQ\My "" D:PAI(A;;GAGR;;;BA)(A;;GAGR;;;SY)(A;;GAGR;;;NS) MSMQ\My

Recycle MSMQ and confirm that the error above is no longer logged in the event log.

The Step Below may or may not be required.

On the Win2k3 Server that is running IIS we execute the following command.

C:\Inetpub\AdminScripts>

Cscript.exe adsutil.vbs SET w3svc/1/SSLAlwaysNegoClientCert "true"

Note: You need to execute this from AdminScripts Directory.

MSMQ 3.0 on WinXP and Win2k3 Unable to Send WCF messages to Win2k8 and higher - Error The Signature is Invalid.

2010-05-12T17:40:00Z

Customer was running a WCF application on WinXP. His WCF app messages were ending up in the deadletter queue with error "The Signature is Invalid."

We changed the security Configuration to

Also,

useActiveDirectory="false"

Now we get access is denied error

More Information

Thanks to James Birdsall, we were able to resolve the problem. The problem was caused due the fact that Win2k8 rejects a MD5 Signed document from XP Sp2. Inorder for the Win2k8 and higher to accept messages From Win2k, Win2k3 and WinXp you will have to add the following Registry Key (NOT VALUE)

HKLM\SOFTWARE\Microsoft\MSMQ\Parameters\Security\WeakHashAlgorithms HKLM\SOFTWARE\Microsoft\MSMQ\ClusteredQMs\MSMQ$MSMQResourceName\Parameters\Security\WeakHashAlgorithms

NOTE: If you are receiving messages on a cluster then you need to create the key on both the physical nodes and the cluster node.

This registry key allows the Win2k8 Box to Receive MD2, MD4, MD5 Messages. If you want to be selective and say do not want to accept MD5 messages but accept MD2 and MD4 messages then you can add the following registry value

DisableMD5 dword value 0x8003

For more details refer to article http://msdn.microsoft.com/en-us/library/ms706984(VS.85).aspx and James Birdsall's (developer) blog at http://blogs.msdn.com/motleyqueue/archive/2007/09/29/authentication-and-msmq4.aspx

Error When trying to Open a Biztalk Orchestration using Visual Studio 2008

2010-04-29T22:16:00Z

I had a very interesting case last week. Whenever we tried to open an Orchestration in Visual Studio 2008 we get the following error:

Package 'ODPackage' has failed to load properly (GUID = {2F926337-2BFB-46AB-BBC4-A955CE25FF6F}).)

What was interesting is that we were able to open some orchestrations in certain projects. Also the projects that were throwing the error, if we added them to a new solution then the files would open but as soon as we saved the solution and tried to open the orchestration file, we would get the error again.

While debugging the solution, we found that the problem was related to certain project files that did not have an file extension. In VS 2008, when the project is opened we check for file types by looking at file extensions. If the file extension is not present then we error out.

The easy workaround to this is to add an file extension to the files that are missing them or exclude them for the project.

Customer also mentioned that folders with no files in them may also cause this problem.

Option to launch MSMQ 4.0 management console not available for MSMQ 4.0 Cluster Resource

2010-04-23T18:39:00Z

I had an very interesting case last week. Customer has a Win2k8 Cluster. He has Clustered MSMQ and the MSMQ resource is online and fully functional. The Problem he was facing is how to manage MSMQ? On Win2000 and Win2003 Clusters we have the MMCV tool that can be used to manage the cluster resource. On Win2008 when you create a MSMQ resource you have the capability to manage by lauching the MSMQ Management Console from the Cluster Failover Admin. All you need to do is select the MSMQ resource and then select Computer Management. In this particular case, the management functionality was not visible.

During the course of troubleshooting, we recreated the MSMQ resource a couple of times. But that did not help. After a lot of research, I found that the Management Functionality is not available if you add the resource to a pre-existing cluster group. Obviously in a real world scenario this is not feasible since we cannot dictate the order in which resources are created. But as a possible workaround, I found that the resource can still be managed using the MMCV Utility just as in Win2k and Win2k3 clusters.

MSMQ 4.0 Messages stay stuck in outgoing Queue if and only if a Message Time Out is Specified

2010-04-16T20:47:00Z

A very interesting case came in this week. Customer is sending HTTPS messages from Win2k3 machine to another Win2k3 Server with an Win2k8 Server acting as an intermediate server. The HTTPS remained stuck in outgoing queue and would eventually timeout. What made the case interesting was the fact that the problem only reproduces when you specify a timeout value. If no TTRQ value is specified, then the messages make it through to the destination server. This is what was mind boggling.

At first I thought, this may be a bug with the HTTPS code. But we were able to reproduce the problem with HTTP messages as well. After some digging we realized that the problem was related to time sync issues between the sender and receiver. The Sender was about 2 mins behind the Receiver. As a result with a 30sec time to reach queue value specified, the messages would have expired even before the message arrived in the queue. As a result the messages remained stuck in the outgoing queue on the sender.

To fix the problem, we ensured that the system clocks on the two machines were in sync and the messages started to flow between the two boxes.

Biztalk MQSC Send Port Unable to Send Messages to MQSeries.

2010-04-12T23:38:00Z

I was investigating a problem where I was unable to Send Messages to Send Messages to MQSeries using the MQSC adapter. Everytime I tried, the following event would be logged in the event log.

Event Type: Warning
Event Source: BizTalk Server 2009
Event Category: (1)
Event ID: 5740
Date: 4/6/2010
Time: 4:18:37 PM
User: N/A
Computer: abcxyz
Description:
The adapter "MQSC" raised an error message. Details "The type initializer for 'Microsoft.BizTalk.Adapter.Mqsc.MqscAsyncBatch' threw an exception.".

I spent hour trying to figure out the problem. Also Reinstalled the Adapter but that did not help. Finally found that the problem was related to using a 64Bit host as the Send Handler. Switched it to 32bit Host and that resolved the issue. Kb article 939202 also mentions that the 64bit host is not supported with MQSC adapter.

Biztalk 2009 and Team Foundation Server - Change in Deployment Properties.

2010-03-16T21:14:00Z

I had an interesting case recently. Customer was using Team Foundation Server with Biztalk 2009. He mentioned that in the earlier versions of Biztalk he was able to specify the deployment properties just once and share it across multiple developers.

After some research here is what we found. Earlier we had our own project type and had everything in a single file. In BizTalk 2009 onwards we have two files for biztalk projects(which is the correct behaviour): one is btproj and other is .btproj.user file.

All build related information required at the time of build and expected to be shared between developers are kept in *.btporj file and all user specific properties are kept in *.btproj.user file which is not supposed to be shared. For example in Schema properties like :

Input Instance File Name , it varies from one user to other ConfigurationDatabase: it again varies from one user to other. Same with other properties.

I don't think there is any direct workaround for this, I can't ask them to checkin user file ( which ideally they should not do) because in 2009 release we save full path for any artifact in user file which is used to map properties between *.btproj and *.btproj.user file. Copying whole project directory/files from one place to another will break mapping and you will be back to the same problem, no properties for an artifact in btproj.

Note: *.btproj always save relative path , so we don't have any problem there.

If users really want a workaround then they have to first change full path to relative path( relative path to btproj file) of that artifact in user file for ex: <File Path="Schema2.xsd">. Now they have to checkin this user file in their TFS server which will be used by all users.

Thanks to Saurabh Sharda for his valuable input.

MSMQ HTTP Message Send Problems.

2010-02-11T20:44:00Z

Recently I handled an issue where customer was unable to send MSMQ HTTP messages from one Win2k8 machine to another. The messages would remain stuck in the outgoing queue. No matter what we tried the problem would not go away. What we found on the network trace was very interesting:

We get an HTTP 500 but it gives a small hint as seen below - The Specified Module was not found.

Netmon Trace Output

14 2010-02-03 14:34:00.300000 10.10.10.1 10.10.10.2 HTTP HTTP/1.1 500 Internal Server Error (text/html)

Content-Length: 101
Content-Type: text/html
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
Date: Wed, 03 Feb 2010 22:33:59 GMT

<html><head><title>Error</title></head><body>The specified module could not be found.
</body></html>

Another clue to this problem is when you try to suft to: http://remoteservername/msmq we got an HTTP 502/504 error and not an http 404.

Now we started a Procmon trace on the MSMQ Server to see what is going on. We found something very interesting. The Process Monitor log showed that we are unable to find 32bit version of mqise.dll.

To work around this problem, copy a 32-bit version of the Mqise.dll file to the %WINDIR%\SysWOW64 directory on the computer that is running a 64-bit version of Windows.

A second option is to run IIS in 64Bit mode.

Kb Article 936497 talks more about this in more detail.

Biztalk FTP adapter Append Functionality.

2009-11-10T00:01:00Z

Recently I got an very interesting problem. A customer wanted to implement the File Append Functionality that is present in the File Adapater using the File Adapater. There are a business requirement that would not allow him to create a new file on the ftp server.

As an workaround we tried to Use the Before Put and After Put to execute the Append Command. But that did not work. The FTP adapter however would have none of this. It would log an error stating Command not Understood.

However when I ran same commands from the Command prompt I was able to append to the file on the ftp server. After some digging and some help from the Biztalk Dev Team, I found that the FTP protocol mandates that a PASV/PORT command be sent before any data transfer( RETR / STOR/ APPE ). In this case before the append ( APPE ) command is sent we need to open a data connection by passing a PASV or PORT command specifically.

Further, the issue here is not the PASV command because After Put can take multiple ftp raw commands separated with ";" and this way user can place a PASV command in the AfterPut field before the APPE command. The main issue here is that we don't have any way of sending the local file data over the data connection once setup because AfterPut can handle only control channel based ftp commands.

Currently there is no way to implement this is the FTP adapater.

WorkAround

A suggested workaround would be to read the file using FTP, append to it and then post it back using FTP.

MSMQ 4.0 Installation on Windows 2008 Server.

2009-09-19T00:48:00Z

MSMQ Installation on Windows 2008 Server is a little different when compared to Windows 2003 Server. To install MSMQ 4.0 you need to execute the following Steps.

Open the Server Manager Console
Select Features and then click Add Features
Go Down and select Message Queuing.
Expand the Message Queuing Tree and then Select the necessary options that you want to install.
Then Click Next and follow the Prompts to Install the Service.

Note: To complete the installation, you will be requested to reboot the server.

After the reboot is complete and you log back in, the MSMQ Service installation will now run through completion.

On some occassions, you may run into installation problems. In such case you may want to refer the install log. The MSMQ installation log is C:\Windows\Logs\CBS\cbs.log. Within CBS log, look for the first occurence of the text "Failure will not be ignored: A rollback will be initiated after all the operations in the installer queue are completed"somewhere before this error message, the actual error will be listed.

The Application event log may also point you in the right direction to resolve any installation related issues.

Checklist when installing MSMQ

If installing MSMQ 4.0 with Active Directory Integration, make sure that the Network Service account has the permission to create the MSMQ configuration Object on the computer account. If you do not have access the installation will fail with an error:

0xc00e0025 unable to create MSMQ configuration Object in Active Directory. Access is Denied.

To resolve the above error execute the following steps

Open Active Directory Users and Computers Snap-in
Go down to the computer account on which you are going to install MSMQ.
Right click and then Select Properties.
Go to the Security Tab and click the Advanced button
Click the Add button
Add <Domain>\Network Sevrice account
Click to enable Allow for the Create MSMQ Configuration Objects permissions
Then Click Ok and close the dialog box.