One of the problems with remote debugging is the lack of consistency with symbol loading. For managed debugging, we load symbols on the debuggee machine. For native debugging, we load symbols on the debugger machine. This means that if you are, for example, interop debugging a C++ binary with managed and native code, you will need symbol for the same binary on both machines. I think the ideal solution would be to load symbols on the debugger machine, but also suck symbols from the debuggee machine if they are there. This model works really well unless you happen to be remote debugging to a computer which is far away, and where the debuggee was also built on that far away computer. Having symbols on the debugger machine works especially well with symbol server. You don’t really want to pull binaries from a symbol server onto the debuggee machine. Especially because the remote debugging components might be running as a different user.
We will fix this someday.