If you're finding that the VS debugging seems to get slower and slower (especially when you first crank up a process), one thing to look for is unresolved breakpoints. That is, breakpoints that the debugger can't set just yet, as the module/assembly hasn't loaded yet.

In my particular scenario, I had several dozen breakpoints from an old debugging session with a particular DLL. I had since switched my attention to a completely different process, but the debugger was still valiantly examining each DLL as it loaded to see if it should set any breakpoints in it. When a process load dozens of DLLs, and you have dozens of breakpoints to check for each DLL, it's no wonder things aren't as speedy as you might once remember.

The moral of the story: Go to the breakpoint window every once in awhile to clean out old unneeded breakpoints.