The DCOM and Firewall teams have done some great work in Windows XP SP2 at making things more secure. In my opinion, this was a great thing. Keep in mind that these changes have cost me two months of dev work, and already many support issues, so I think this is a rather bold statement.
Anyway, the big change is that if the client and server are running on different machines, and the Client is running on XP SP2, then this will no longer work:
hr = CoImpersonateClient();
if (hr == S_OK)
DCOM will cause BackwardCall to fail with E_ACCESSDENIED.
The firewall will cause 'CServer::ForwardCall' to not execute because pCallback will fail to unmarshal (E_OUTOFMEMORY).
XP SP2 introduces the concept of machine wide security limits for DCOM. The default limits require that all remote requests must from an authenticated user. The way this used to work is that the server would impersonate the client (CoImpersonateClient), this would give the server an impersonation token that represents the client user. This token is very useful as long as the server only tries to use it on the server’s machine. However, when the server tries to use it on another machine, it won’t work. If it did work, it would be called delegation. Delegation is very dangerous because it means that if I connect to a remote server, that server now has the right to connect as me to any other machine. So it could go and talk to the payroll server and change what bank account my paycheck is direct deposited to. Anyway, since we don’t have delegation, that request will come in as 'ANONYMOUS USER'. With XP SP2, these callbacks don’t work.
To make this work, you need to stop impersonating the client in the server, and instead find a way to make the callbacks work using the server process’s identity. If you are on the same domain (or domains with two-way trust), the easy way to do this is to run the server as Local System or Network Service. These accounts can make calls off the box using credentials from the machine account. You could also use a domain account, or a local account which exists on both machines with the same user name and password. If the machines aren’t on the same domain (or domains with two-way trust), then you need to have accounts with matching user names and passwords.
There is some more information on Don Box’s blog and on TechNet.
For the callback to work, the main DCOM port (TCP 135) must be opened. Unfortunately, since this port is not part of your application, the firewall can’t detect that it needs to be opened, so the application writer is forced to find a way to warn users to open the port. You will also want to unblock your application because DCOM will open ports in your own process.