Jackson, recently posted a new sample for how to write a Visual Studio debug engine. As Joc put it: "in our architecture, the debug engine is the part that does all of the communication with the debugging/diagnostic APIs for a given platform, and communicates those back to the UI through a common set of interfaces. The current debugger ships debug engines for managed code, native code, script, native devices and t-sql in the box. The debugger engine extension point gives people writing new platforms the ability to provide an excellent VS debugging experience."
Please note that writing a debug engine is a complex undertaking. If you have a language that's targetting the .NET runtime and the platforms it supports, then you probably don't need to go this route (a new Expression Evaluator, Language Service and project system are more appropriate targets). But if you have a new platform (CPU, runtime, etc) that needs debugging support, this sample should provide good guidance.