[[Corrected December 17th, 2009, with input from Jolie Boushey]]
This isn't going to be easy. There are two major obstacles to cope with
System.Messaging in .NET does include some useful looking options but unfortunately they fall short.
If you cannot check the properties then the next step is to use the queue and see what happens.
For example, this blog post shows how to determine if a queue exists:
You could change it to send a transactional test message; if the message goes to the Transactional Dead Letter Queue with a class of "Nontransactional queue" then you know the queue isn't transactional. Downside is what to do with the successfully delivered message if the queue IS transactional...
An alternative could be to do a transactional remote receive (assuming MSMQ 4.0 at both ends). If you get 0xC00E0050 (MQ_ERROR_TRANSACTION_USAGE) then the queue isn't transactional.If you get receive a message because the queue was transactional then just abort the transaction to put it back in the queue.
If anyone has any more efficient methods to share then please post them here.