Aaron helped me figure this out as it puzzled me for a couple of days :)
It's also a topic at: http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=3377742&SiteID=1
If you create an isolated shell and then drop your VS packages into it, you won't be able to debug into any managed package until you change the debug settings to mixed mode. The debugger defaults to "Auto" but it only loads the native symbols.
To enable debugging into your managed packages, try this:
This will now load the symbols for your managed packages and allow you to debug into it.
Note - your F5 experience will be a little slower.