Here's a list of things that may slow down execution under a debugger. I've seen a few threads go by about ways that running under the debugger significantly slows down normal execution by more than 2x, and this is a collection of those items. 

Background:
Stepping through each statement (F10) runs significantly slower than just running the code under a debugger (F5) (some perf numbers here). But just running code F5 under a debugger can run at full speed. For example running pure jitted code that doesn't do anything a debugger needs to know about (like loading types or throwing an exception) ought to be run at full speed. So this loop should not slow down merely because it's under the debugger:

    for(int i =0; i < 10000; i++);

Certain operations become slower under a debugger. For example, if the app generates debug events (like exceptions or loading types), it will slow down to communicate with the debugger. The debugger can also twiddle knobs to slow the debuggee down (such as disabling optimizations). So the debugger slowdown depends on what the code can do, and what the debugger does.

List:
Here's a list of things that may significantly slow down normal F5 execution under a debugger. This is mainly targetted at managed-debugging, but much applies to native as well:

1. Pinvokes: This is the big one for managed / interop-debugging: Pinvokes can be 100x slower.  Fix is here.
2. Symbol loading. VS2005 prints the symbol loading in the status bar at the bottom. If you have a lot of dlls, or a slow network share on your symbol path, this could slow you down. Check your symbol path (especially the _NT_SYMBOL_PATH environment variable)  to make sure you don't have some lame network path on there. Consider using a symbol cache. VS conveniently lets you set this in its symbol dialog (Tools : options : debugging : symbols).
3. Conditional Breakpoints / tracepoints in a tight loop. Bottom line: Hitting a breakpoints slows down the debuggee's execution. If the debugger hits a breakpoint, it stops the debuggee. Conditional breakpoints will stop the debuggee, evaluate the condition, and then resume it. Tracepoints will stop the debuggee, log, and resume. So both of these stop the debuggee and resuming it without bothering the user. If this happens alot (such as in a tight loop), that could be a slowdown. Check your Breakpoints window for such breakpoints.
4. Lots of 1st-chance exceptions.  Exceptions are slower under a debugger. VS will log these to the output window. Check the output window for other suspicious activity.
5. Maybe it's from non-optimized code. This is a long-shot because in general, debuggable code should only be a reasonable slowdown. In some cases, (inlining in a tight-loop), optimizations can make a huge difference. If you think this may be the problem, you can toggle this in VS in Tools |Options | Debugging/General, uncheck "Suppress JIT optimizations on module load (Managed only)". Ngen is another optimization and debuggers may do things that disable ngen. See debugging and ngen  if the lack of ngen images is causing you grief.
6. Maybe func-eval. The debugger usually only evals when the debuggee is stopped. This could happen at conditional breakpoints.
7. Debugger configuration. Interop-debugging is slower than managed-only. Remote-debugging is slower than local debugging. Hopefully, this shouldn't be signficant in an F5-scenario, but mentioning for completeness sake.

I'll come back and update this list as I think of more thing.