One thing that developers find frustrating is wasting time working out where messages have gone and having a mismatch between the transactional types of the messages and queues is a common cause.

Sending to remote queues always succeeds because the sender cannot determine if the destination is transactional or not. The remote queue manager will reject the message if the transactional type is different.

For local local queues, the process depends on the version of Windows in use. In the past (MSMQ 1.0 and 2.0), an exception would have been returned which a Try-Catch could intercept. This was changed in MSMQ 3.0 so that you needed only one failure handling mechanism (for local and remote queues) instead of two (one for local queues using exceptions and another for remote queues using acknowledgements).

To see the rejection from a transactional mismatch, you need to request a negative acknowledgement (NACK) or use the Deadletter queues (DLQs). 

Here's a chunk of C# code to demonstrate switching on acknowledgements, journaling (positive source journaling) and dead lettering (negative source journaling) for a message:

var msg = new Message();
msg.Body = "Hello world";
msg.Label = "Can you see me";
msg.UseDeadLetterQueue =
true;
msg.UseJournalQueue = true;
msg.AcknowledgeType = AcknowledgeTypes.FullReachQueue | AcknowledgeTypes.FullReceive;
msg.AdministrationQueue =
new MessageQueue(@".\private$\Ack");

And here are four different scenarios with C# code for sending the message to a local queue. 
"Test" is a non-transactional queue and "TestTX" is a transactional queue, both pre-created.
The screenshots show examples of what messages you can expect to see as a result of the properties set on the original message.

Sending a transactional message to a transactional queue

var mq = new MessageQueue(@"FormatName:DIRECT=OS:.\private$\testtx");
mq.Send(msg, MessageQueueTransactionType.Single); 
 

The message arrives in the "TestTX" queue and a confirming acknowledgement appears in the "Ack" queue (notice the priority of zero that transactional messages have):


Sending a non-transactional message to a non-transactional queue

var mq = new MessageQueue(@"FormatName:DIRECT=OS:.\private$\test");
mq.Send(msg);

The message arrives in the "Test" queue and a confirming acknowledgement appears in the "Ack" queue:

 


Sending a non-transactional message to a transactional queue

var mq = new MessageQueue(@"FormatName:DIRECT=OS:.\private$\testtx");
mq.Send(msg);

In the Ack queue is a negative acknowledgement showing that the message is not the right type for the queue; there will be an almost-identical message in the "Dead-letter messages" queue (different Message and Lookup ID):


Sending a transactional message to a non-transactional queue

var mq = new MessageQueue(@"FormatName:DIRECT=OS:.\private$\test");
mq.Send(msg, MessageQueueTransactionType.Single);

In the Ack queue is a negative acknowledgement showing that the queue is not the right type for the message; there will be an almost-identical message in the "Transactional dead-letter messages" queue (different Message and Lookup ID): 

 

Note - this approach can be applied to other mismatches, like having queues that require authentication or encryption.