You may have noticed that debugging optimized builds (eg, what you commonly get when you attach to a end-user instance of your app that wasn't launched from your debugging environment) is usually a degraded experience.

At an architectural level, there's a fundamental tension between being debuggable and optimizing. The debugger generally requires the code to behave to a certain contract established between the compiler and the debugger (and recorded in the PDB). That contract mainly requires the code to maintain a close relationship to the original source code. The Optimizer makes aggressive transforms to the code for the sole purpose of making it execute more efficiently, with no regards to the original source code. 

Here are some of the things that break and why. This is not a complete list. This is general, so isn't managed or native code specific:

1. Callstack frames may be missing.  This is often Inlining: (This is one reason that you shouldn't programmaticaly rely on the results on System.Diagnostics.StackTrace). The CLR's inliner in Whidbey is pretty wimpy, so this is actually less of an issue for managed code compared to native code.
2. You callstack may show you functions that you're not actually executing. Code folding. If function_1() and function_2() both compile to the same code, the compiler/linker could share that code and just use 1 copy. So calls to function_2() would appear to have called function_1().  But the code all executes the same. The CLR does code-folding for generics.
3. Local variables may be missing: Locals may get optimized out. The local may not be used, or maybe the compiler can store it in a more efficient fashion than the current PDB contract can describe to a debugger.
4. Locals may be reported, but have the wrong value. This is bad. This often happens when the debugger doesn't realize the code has breached the PDB contract. IMO, I would rather a debugger just tell you it can't find a value, or tell you that it's guessing.
5. You may not be able to set breakpoints on certain lines. The lines may be optimized out. This may be code folding.
6. Stepping through a function may get random: The optimizer may rearrange the ordering of basic blocks. This could be to eliminate jump instructions or to rearrange the code for better hot/cold locality.
7. Breakpoints you do set may not be hit even when you think the code executes.  This is another manifestation of the previous issues. In this case, the optimizer has changed the wrong of source to code executed, without properly informing the debugger (perhaps the existing PDB contract can't describe the optimizer's actual mapping).

In theory, the you can compensate for many of these problems by making the PDB contracts richer so that the optimizer can describe to the debugger what it's actually doing.