From the mailbag: "I notice that running and attaching the debugger takes on average now 8-10 seconds for even a simple console application."
 I'm not a VS expert, but here are some trouble-shooting tips:
 1. Try a managed-only attach instead of interop (mixed-mode) attach.  As the dev responsible for the CLR's Interop-debugging support, I can vouch that interop-attach is significantly more complicated and may slow things down.
 2. If it's VS, try removing any addins. (this suggestion was from the mailbag)
 3. Make sure your symbol server path is set properly. If you have a lot dlls, the debugger may be hitting the symbol server for each one. If the symbol server then sends it over a slow network connection, attach will take a while. Keep an eye on the status bar for any hints here. I once had a bug assigned to me that complained that attaching to VS with VS was hung. It turned out that VS had 150 dlls loaded and it was just taking forever to get through all the symbols.
3a. [Update] Use a local cache always, Beta2 has a bug where we would hit the symbol servers even when the binary was in the local cache but in RTM we will not. 
3b. [Update] If you do not care about loading off the symbol servers, check the option in tools |options |debugging "Search the above locations only when symbols are loaded manually", You can alwasy right click on the callstack and do a load symbols if you need to.
 4. Try attaching with Mdbg instead for a comparison. If VS attaches very slowly, and Mdbg attaches very quickly, then at least we've narrowed down the problem to something that VS is doing. If Mdbg attaches very slowly, then there's not much left you can do (except the standard shot-in-the-wild machine-wide stuff life reboot to reclaim leaked resources or defragment your hard drive)

If people have other tips, let me know and I can add them to this list.

[Update] added Sections 3a, 3b per suggestions from Raj (VS dev on the native-debugger).