Had a very interesting case this week. Customer had few MSMQ 2008 Servers forming the NLB. They had few thousand clients sending non transactional messages to MSMQ Queues using Direct Format Names. Everytime you restart the MSMQ on the server the client would be able to send messages and then stop. The messages would start building in the out going queue with state Waiting to connect. A netmon trace would show the Server sending resets to the clients and some times the reset would come from the server.

MSMQ service on the server caches the client certificate. If the number of clients exceeds the default cached size then the server tries to delete half the
number of certs based on age. Some problem here causes the threads to hang.

There is an easy fix to this problem. All we do is increase the size of the cache so that we do not have to delete the certificates to make way for other
certificates.

HKLM\SOFTWARE\Microsoft\MSMQ\Parameters\UserCacheSize  Dword Value to Number of client +100(for safety)

You can learn more about the registry key here.