Even when I switch to "Release builds"? We recently got a connect bug on this so I thought I'd share my thoughts.
In the debugger, we always try to minimize the impact we introduce on the debuggee, but there is always some work that we do in a synchronous way. We alway deal with breakpoints, even when they are in "tracepoint mode", we always do work when receiving a module load or thread create event. As I say, we try to minimize the impact of this work, and are constantly looking to improve the performance (and therefor usually the perturbance) on the target application. However the biggest contributor I see to app performance degradation is the fact that when launching under the debugger, the operating system uses the debug heap.
The VS debugger uses the standard Windows CreateProcess call to create the debuggee. See docs for that here: http://msdn.microsoft.com/en-us/library/ms682425(VS.85).aspx. We pass in the create flag DEBUG_PROCESS as documented here: http://msdn.microsoft.com/en-us/library/ms684863(VS.85).aspx
When we do that, Windows enables three heap debugging features:
1. and 2. have a sizable performance impact when freeing memory. For more information on this subject, read Mark Russinovich and David Solomon's book "Windows Internals" Chapter 7.
There are two workarounds for this issue.
Remember however that you are giving up a lot of validation that the OS does on your behalf. Catching memory corruption issues early is very important, so don't take this step blindly.