By default, distributed transactions will not work when at least one of the computers involved (client, middle-tier or database) is part of a Windows workgroup. This is due to the fact that Windows workgroups do not provide the necessary security infrastructure (provided by a directory service like Active Directory available on Windows domains) to allow mutual authentication and packet privacy for MSDTC. If you need distributed transactions, my recommendation is to move your computers to a Windows domain. If a Windows domain is not an option available to you, before using the workaround provided here, you should make sure that you put the systems in a well isolated environment (behind a firewall for instance). The workaround is to disable the security for MSDTC and can be done by adding a DWORD registry value called TurnOffRpcSecurity at HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSDTC and by setting this value to be 1. An MSDTC service restart is required for the change to take effect.
The same workaround can be used when the computers are in different Windows domains that don't trust each other.