greggm's WebLog

Writing a script debugger without using the MDM

Today’s blog is about writing your own script debugger.

Visual Studio has provided ‘Script’ debugging for a long time. In this context, ‘Script’ refers to script run by jscript.dll or vbscript.dll such as the script code in Internet Explorer, classic ASP scripting, or the cscript/wscript programs. Long long ago, someone provided a sample for creating your own script debugging utilizing the active script debugging interfaces (ex: IRemoteDebugApplication).

However, this sample relied on a Microsoft executable called mdm.exe (IMachineDebugManager/CLSID_MachineDebugManager). After Visual Studio 2003, Visual Studio stopped shipping mdm.exe. So today, I wanted to provide a brief summary of how one could port an mdm.exe-based script debugger to the newer interfaces. Mdm.exe provided essentially two related features -- it allowed a debugger determine what script applications where currently running, and there was an eventing interface that let a debugger be informed when a new script application started. These features worked because the debugger component inside of the target process (pdm.dll) would connect to mdm.exe whenever a new script application was created.

Today, these features have been moved to pdm.dll. So now, pdm.dll is loaded in both the debugger and in the target process. In the target process pdm.dll still continues to do the same work as it has always done. However, now one can also load pdm.dll into the debugger process to obtain the old MDM features.

PDM now has a new CLSID for the work that it does in the debugger process – CLSID_MsProgramProvider. It exposes an AD7 interface for obtaining this information: IDebugProgramProvider2.

There is sample code attached to this blog which demonstrates how to launch or attach to a program using these new interfaces.

Reporting a Visual Studio crash to Microsoft

Background

Sometimes Visual Studio crashes. It would be nice if this wasn’t the case, but unfortunately it happens. However, like many Microsoft applications, Visual Studio has Error Reporting so that when a problem happens, you can tell us about it in one click. However, sometimes you would like to actually tell a live human about your crash. There are a number of advantages to doing so:

1. A human might be able to tell you about a work around that Windows Error reporting doesn’t know about.
2. A human can tell you if this crash was something that we could diagnose.
3. It helps to make sure your issue is really fixed. When a developer at Microsoft looks at a Windows Error report, all we see is where the crash happened and some local variables. Sometimes we can figure out what was happening and create a repro to confirm our fix. This is the best case, but it often doesn’t happen this way. More often we need to either guess what a fix is but not test the fix, or we are unable to even guess what a correct fix might be.

For these reasons you might want to also report the issue through the Visual Studio product feedback center.

What to do

Step #1: Go to http://connect.microsoft.com/site/sitehome.aspx?SiteID=210 and enter a bug.

Step #2: In the bug, include the ‘bucketing information’. You can find this information by:

• Open the event viewer (Right click on Computer and go to ‘Manage’; click on the ‘Event Viewer’ tree item).
• Open the ‘Application’ section
• Look for the entry that happened about the correct time and had the source set to ‘Application Error’
• Copy the content and paste it into the bug, it should look something like:

Faulting application devenv.exe, version 9.0.30428.1, time stamp 0x4815597f, faulting module scriptle2.dll, version 9.0.21022.8, time stamp 0x47317e18, exception code 0xc0000005, fault offset 0x0000328c, process id 0xfbc, application start time 0x01c8bc39887cac69.
 

Visual Studio 2008 SP1 Beta

In case you missed it, Visual Studio 2008 SP1 Beta has been released:
http://www.microsoft.com/downloads/details.aspx?familyid=CF99C752-1391-4BC3-BABC-86BC0B9E8E5A&displaylang=en

For the debugger, Visual Studio 2008 SP1 contains:

1. As with all service packs, fixes for all the significant bugs that we found since Visual Studio 2008 RTM.
2. Improved support for downloading symbols and PDBs for the .NET Framework and other Microsoft products.
3. Improvements for using Visual Studio with non-local symbol stores (such as the Microsoft public symbol servers)
4. Step into specific for managed. We have had this feature with native for years, with SP1 we added this support for managed as well. If a line contains multiple function calls, you can right click on the line in the editor and select which function you would like to step into.
5. Step filtering for managed. There is now an option to stop stepping into property get routines.

We want to try and get the RTM version into customer’s hands as soon as we can, so this means there isn’t too long for us to act on customer feedback. So if you are interested, give it a whirl soon.

One very important note: If you have installed any Visual Studio 2008 hot fix, make sure that you run this tool _before_ installing Visual Studio 2008 SP1 Beta:
http://code.msdn.microsoft.com/RemoveKB944899

If you would rather not run a tool, manual steps can be found here:
http://blogs.msdn.com/heaths/archive/2008/05/16/kb944899-should-be-removed-before-installing-visual-studio-2008-sp1.aspx

Visual Studio Remote Debugger Service user account requirements

I was asked today -- Why does the Visual Studio Remote Debugger Service need to be run as an administrator?

Since it doesn't appear that this information is documented, I figured I would provide an answer. Running the remote debugger service as an administrator is really a recommendation rather than a requirement. The actual requirement is:

1. Account must have the 'logon as service' privilege
2. Account must be able to connect 'backwards' to the Visual Studio computer over the network. For this reason, on a domain, its easiest if the service is running under Local System, Network Service, or a domain account.  If you want to run it as a local account see: http://blogs.msdn.com/greggm/archive/2004/10/04/237519.aspx
3. Account must have rights to debug the target process. This means the service needs to either run under the same account as the process to be debugged, or the service needs to run as an administrator. 

We recommend running the service as an administrator because:

1. The service only allows connections from administrators, so there is no security reason why running it as an administrator is bad.
2. Running it without administrative privileges may put the user in a situation where the Visual Studio user has access to debug a particular process, but since the remote debugging service lacks these rights the Visual Studio user is not allowed to debug the process.
3. It’s a  lot easier to setup this way, and there is enough that can go wrong with remote debugging already.

Exception Filter Inject

Recently I have found the need to have an exception filter in some C# code that I have written. Since there is no language support for this, I wrote an IL instrumentation tool:

http://code.msdn.microsoft.com/ExceptionFilterInjct

For debugging scenarios, the value of this tool is to make it easier to save a minidump of exceptions. Unfortunately, there is not yet a rich Visual Studio experience for managed minidumps, so you will need to use sos.dll (see http://msdn2.microsoft.com/en-us/magazine/cc164138.aspx).

 

Advice on creating a code generator

I have been spending some time recently creating a custom code generator that outputs C++ and C# code from a custom XML format. This blog is about some of what I have learned while working on this.

Advice #1: Use '#line'

There is a really cheap way to get a debugging experience for your custom language: #line. With #line, you get to change the debug info that ilasm/csc/vbc/cl writes into the PDB so that the debug info points back to the original source file instead of the one that your code generator creates. This is quite handy if you custom language has at least parts where there is a strong connection between the generated code and the custom language. Notes:

1. There is not a similar technique to allow you to re-associate variables, so I would advise using the original names of variables if possible.
2. If a line of the original source expands to multiple lines of the generated code, just repeat the #line before each line of the generated code.
3. There is an issue with this technique if your source language is XML-based in VS 2008 RTM. Visual Studio hopes to correct this problem in SP1.
4. In C#, you can use this technique with #pragma checksum to set the checksum of the source file. This is useful if you expect to have a bunch of source files with the same name (ex: default.aspx) or if you are already grabbing the checksum for some other reason. Otherwise, it is probably overkill for a custom code generator. Note that the GUID with #pragma checksum identifies the hash algorithm (ex: MD5 is { 406ea660-64cf-4c82-b6f0-42d48172a799}).

Example:

Generated file:

    static void Main(string[] args)

    {

#line 1 "HelloWorld.ExampleLanguage"

        Console.WriteLine("Hello World");

#line default
    }

HelloWorld.ExampleLanguage (custom language source file):

Hello World

Advise #2: Use XML

XML is a great way to do a custom language these days because at no cost, you get a lexer, a parser, syntax validation, a language sevice, and your compiler gets to work with deserialized classes instead of with text. Here is the procedure that I would recommend:

1. Figure out what you would like your language to look like by writing a bunch of examples.
2. Run xsd.exe over your examples to create the start of a schema.
3. Open up the generated schema and start making changes. You might need to make changes because the generated schema wasn’t specific enough (ex: you have an enumerator that it has represented as a string), or because your examples had bugs in it, or because you want to add documentation.
4. Once you have the schema vaguely correct, it is time to use xsd.exe to create class files from your schema. This allows you to use XmlSerializer.Deserialize to create classes from the input XML with little work. You want to create a build step to do this as you will be changing your schema often.
5. Hookup the schema file so that you can run it as part of compilation. This provides you with a pretty good set of validation without effort. I did this by embedding the schema as a resource in my compiler, but obviously you could also leave it as a file that your compiler opens.
6. (Optional) Use sgen.exe to generate the serialization assembly -- XmlSerializer depends on a generated assembly to perform the serialization/deserialization. By default, this assembly is generated dynamically, but you can also use sgen.exe to generate this assembly up front.
7. When you edit your language, edit in Visual Studio and make sure that the XML editor has your schema open. The XML editor will pick up schema items that are in your project, and it will also pick up schemas that are in the Visual Studio 'schema directory'.

Example target for running xsd.exe: 
  <!--Generate Example.cs using xsd.exe -->
  <Target Name="GenerateXSDClasses"
    Inputs="Example.xsd"
    Outputs="$(IntermediateOutputPath)\Example.cs">
    <Exec Command="$(RunManagedToolPath) xsd.exe Example.xsd /classes /fields /namespace:ExampleCompiler /out:$(IntermediateOutputPath)"/>
  </Target>

  <ItemGroup>

    <Compile Include="$(IntermediateOutputPath)\Example.cs"/>

  </ItemGroup>

You would also need to wire the target into a property group that runs before compiling

Example of embedding the schema as a resource:

<EmbeddedResource Include="Example.xsd" />

Example of using the schema:

    public static void InitializeSchema()

    {

        if (s_schemaSet != null)

        {

            throw new InvalidOperationException();

        }

       

        System.Reflection.Assembly ThisAssembly = typeof(MyType).Assembly;

        Stream stream = ThisAssembly.GetManifestResourceStream("ExampleCompiler.Example.xsd");

        XmlReader schemaDocument = XmlReader.Create(stream);

        s_schemaSet = new System.Xml.Schema.XmlSchemaSet();

        s_schemaSet.Add("http://schemas.microsoft.com/vstudio/Example/2008", schemaDocument);

        s_schemaSet.Compile();

    }

 

    XmlReaderSettings settings = new XmlReaderSettings();

    settings.ConformanceLevel = ConformanceLevel.Document;

    settings.Schemas = s_schemaSet;

    settings.ValidationEventHandler += MyValidationEventHandler;

    settings.ValidationType = ValidationType.Schema;

 

    using (XmlReader reader = XmlReader.Create(filename, settings))

Advise #3: Check in the custom compiler output

If you are writing a very high quality compiler that you are trying to productize then, when people use your compiler, you would have them wire it into their build process such that they would input your custom language and get back a dll or exe.
But this is not necessarily the correct bar for a custom code generator. In my case, I am creating a truly custom compiler. Very few people are going to author the input language, and it doesn’t make sense to expend valuable QA resources directly testing the compiler (rather they would test the generated code). So rather than taking the output of my compiler and directly building these files, I instead checkin the compiler output as a baseline, and the build process runs the custom compiler and compares the output to the baseline. If they differ, it issues a build error.

There are a number of valuable properties that I get out of this 'baseline' approach:

1. If I edit the compiler, testing the compiler becomes very simple. I just do a build and see if I got the same result that I expected.
2. If I edit the input, I get confirmation that the compiler did what I expected. I need to diff the baseline output against the new output and validate that I got expected changes.
3. Compiler bugs turn into build breaks. If I had a bug in my custom compiler where the output was incorrect, rather than going silently unnoticed instead the bug turns into a build break.

Automatically reading source files from TFS

I recently stumbled upon  a blog that explained how to use the Source Server technology to enable the debugger to automatically download source files from TFS. This is useful when you work on a software project that is large enough that you debug against modules that are built by your build lab instead of being built locally.

Here are instructions for setting up Team Build: http://jelle.druyts.net/2007/12/09/SettingUpSourceServerForTFSBuilds.aspx

After doing this, you will also need to configure the debugger. John Robins has instructions for doing that in MSDN Magazine. Go down to the 'Debugging in Visual Studio 2005' section.

The debugger team is expanding

The Visual Studio Debugger team is expanding, and is looking for candidates. If you are interested in applying for a position, we have openings in Dev, Test and PM.

Some great things about working on the debugger:

1. You are helping to improve a tool which millions of developers use every day. There are few software development tools which impact more people.
2. You get to use what you sell. There aren’t too many places where you can work on software that you also get to use. But you will have this experience on the debugger team.
3. You get to play with lots of cool technology. People are always inventing new platforms, and they are always going to need debugged.
4. Great team -- you will get to work with a team of talented and fun people. You will also get to work with me :)
   

Debugging WCF Services in Visual Studio 2008

One of the new features that we added in Visual Studio 2008 was debugger enhancement for Windows Communication Foundation (WCF) services.

There are four features that we support:

1. You can step from a service client into the implementation. Just step into ('F11') a web service proxy and the debugger will automatically go and attach to the server process and stop when execution reaches the service implementation.
2. You can step out of the implementation of a service back into the client.
3. You can get a logical call stack which shows the client and server (see below).
4. The debugger can automatically attach to services in your solution whenever the debugger sees a service request going to the service. This allows you to hit breakpoints in your service without having to go and explicitly stepping into it.

There is more information about these features in our documentation: http://msdn2.microsoft.com/en-us/library/bb514135.aspx. The first three of these features also exist for ASP.NET web services and have been included in Visual Studio for several releases. The fourth feature is new.

There is one troubleshooting item that I wanted to mention because it is not included in our documentation. If you are finding that none of these features seem to be working, it is possible that something has gone wrong with the installation and that the necessary machine.config changes did not happen. To resolve this, we shipped a tool to manually install/uninstall the necessary changes:

1. Start a command prompt (Start->Run, cmd.exe). Note that this will get you a 64-bit command prompt on a 64-bit OS, which is what you want. If you are running on Vista, you need to start this command prompt as an Administrator.
2. Change directory to the root of your VS installation (if you installed to the default location: cd /d "%ProgramFiles%\Microsoft Visual Studio 9.0\Common7\IDE")
3. Run: vsdiag_regwcf.exe -i

You could also use the '-u' flag if you some reason you want to disable this integration.

Hints on using Just-In-Time debugging with Windows Vista

Just-In-Time debugging is trickier on Windows Vista than on previous operating systems. Today I am going to talk about the various knobs that you can use to make this better.

Hint #1: Use Visual Studio 2008 Beta2. I expect Beta2 to be out soon. If you are developing on Windows Vista, I definitely recommend trying Beta2 over previous versions of Visual Studio. While it is still a beta, Visual Studio 2008 greatly improves the Vista experience (at least for the debugger). As far as Just-In-Time debugging goes, the fact that you can't debug elevated processes was a big problem for Visual Studio 2005.

Hint #2: If you get a Windows Error reporting dialog for your application and there is no 'Debug' button, then click 'Send Report' and then you will get a second dialog with a debug button.

Hint #3: Windows Error reporting has several configuration settings. If you are finding that Windows Error Reporting is preventing you from debugging your applications, there are a few settings that you can tweak.
First, you can define the 'AeDebug\Auto' string:

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug]
"Auto"="1"

Note that this is a String value, NOT a DWORD.

If this doesn't work, Error reporting can also be disabled at various levels through control panel:

1. Start->Control Panel
2. Click 'System and Maintenance'
3. Click 'Problem Reports and Solutions'
4. Click the 'Change Settings' link on the left side
5. Click the 'Advanced settings' link
6. Configure at will

Hope this helps.

[Updated on 5/23/2008] In Vista SP1 and Windows Server 2008, there is now a registry key to save the generated .dmp file: http://msdn.microsoft.com/en-us/library/bb787181(VS.85).aspx

 

Debugging Windows Error Reporting

If you’re a software developer, chances are that you have written an application, and this application has crashed. When this happened, it probably put up a dialog that looks something like this:

 How do you figure out what went wrong?

Strategy #1: Extract the minidump from Windows Error Reporting

Windows Error reporting has already created a minidump of the crash. So one way to find out what went wrong is just to look at the minidump. This works really well if your application is written in native code and there is no debugger on the machine where the crash occurs. You can use it for managed code too, but you will need to use sos.dll to analyze the dump (see MSDN). Open a command prompt (Start->Run, cmd.exe), and switch to your temp directory:

C:\Documents and Settings\greggm>cd /d %tmp%

Look for the dump file that Windows Error Reporting produced. It will have a '.dmp' or '.mdmp' extension and the date should be shortly after the crash happened:

C:\DOCUME~1\greggm\LOCALS~1\Temp>dir *.*dmp
 Volume in drive C has no label.
 Volume Serial Number is 70E3-6676

 Directory of C:\DOCUME~1\greggm\LOCALS~1\Temp

05/24/2007  08:54 AM           109,570 4A88835.dmp
               1 File(s)        109,570 bytes
               0 Dir(s)  25,379,823,616 bytes free

Mark the file as read-only. This will prevent Windows Error Reporting from deleting the file after you dismiss the dialog:

C:\DOCUME~1\greggm\LOCALS~1\Temp>attrib +r 4A88835.dmp

Now dismiss windows error reporting and copy the dump file wherever you want. You can analyze the dump in Visual Studio by opening the dump file as a project (File->Open Project), and start debugging (F5).

 

Strategy #2: Access the dump from Online Crash Analysis

Microsoft has a program to allow ISVs to access the crashes in their applications that have been submitted by users. If the person experiencing the crash is a customer, this is a great way to find out what happened. See the winqual site for more information.

 

Strategy #3: Debug the crash through Just-In-Time Debugging

Probably everyone already knows about this, so I won’t spend much time discussing it. However, as long as the application is crashing on a computer that has a debugger installed, this is the easiest option. The only thing tricky that I will mention is that in Windows Vista, depending on your computers Windows Error Reporting settings, you might actually need to click the 'Send information' button before Windows Vista will present you with a second dialog that allows you to debug.

Analyzing Tracepoint Output

Last week I had this displeasure of tracking down a fairly unpleasant reference counting bug. I wound up having to solve the problem by brute force:

1. Set a breakpoint in the constructor of leaking object
2. Set a data breakpoint on the object's reference count variable
3. Change the data breakpoint to a tracepoint that prints the callstack (see dialog)
4. Match up the callstacks.

However, my tracepoint was hit so often that matching up callstacks by hand proved to be impossible. To help, I wrote a little tool that I decided to post. The tool does some basic analysis of the tracepoint output - compute a database of all callstacks with their hit count, and compute a calltree with hit count. While matching up the callstacks was still not easy, it did help.
Basically, the tool gave me two things:

1. In places where an AddRef and Release where matched with the same callstack, I could bucket all of these callstacks together and eliminate them quickly
2. If CFoo::CFoo called AddRef and CFoo::~CFoo called release, I could easily filter out the callstacks that contained neither CFoo::CFoo or CFoo::~CFoo. If the resulting callstacks balanced out (just as many CFoo::CFoo as CFoo::~CFoo) then I could throw out all these callstacks quickly.

The code for my tool is in the attached ‘TracepointAnalysis.cs’ file. Here is some example code to call it:

 

    static void Main(string[] args)

    {

        List<Callstack> stacks;

        List<CallTreeNode> callTreeRoots;

 

        CallstackFilter filter = delegate(CallstackKey key) {

            return key.Contains("CFoo::CFoo") || key.Contains("CFoo::~CFoo");

        };

 

        TracepointAnalysis.ReadFile(@"c:\log.txt", filter, out stacks,

            out callTreeRoots);

 

        DumpStacks(stacks);

        DumpCallTree(callTreeRoots);

    }

 

    private static void DumpCallTree(List<CallTreeNode> roots)

    {

        foreach (CallTreeNode root in roots)

        {

            root.Dump(0);

        }

    }

 

    private static void DumpStacks(List<Callstack> list)

    {

        // Sort the stacks by their hit count

        Comparison<Callstack> comparison = delegate(Callstack a, Callstack b)

        {

            return a.HitCount - b.HitCount;

        };

        list.Sort(comparison);

 

        foreach (Callstack item in list)

        {

            item.Dump();

        }

    }

Setting conditional breakpoints using object ids

In native code, its sometimes useful to set a breakpoint condition so that a breakpoint will only stop for a particular instance of an object. To do this, I simply use the address of the object:

this == (CMyObject*)0x10fc10

In managed code, the garbage collector moves objects around, so even if you were writing in a language that had great support for pointers, setting a conditional breakpoint this way is probably useless. Instead, you can do this using object ids.

Step #1: Make an Object ID. To do this, get the object that you want to use in your condition into the watch window. Then right click on this object and invoke the ‘Make Object ID’ context menu. The watch window output will now include ‘{1#}’:

+  this {cswin.Form1, Text: Form1} {1#} cswin.Form1

The ‘1#’ indicates that this is the first object ID. Each object ID created will get a new object ID (2#, 3#, etc). There is more documentation under 'Object Identity' on MSDN.

Step #2: Set conditional breakpoint. Right click on your breakpoint and click ‘Condition…’, and set it to what you want. For example:

this == 1#

Happy debugging.

Tracking down native leaks

I have been spending some time looking at native memory leaks recently, and I decided to blog about some of the techniques which worked well for me.

First, find out what objects are leaking. If your code doesn’t already have some sort of leak detection scheme, you can take advantage of the fact that the operating system’s heap does. KB 268343 explains how to use UMDH to get a text file of all the leaking callstacks. Keep in mind that to get useful output, _NT_SYMBOL_PATH must point to symbols for both the operating system and your application.

So now you know which objects are leaking, but for ref-counted objects, an allocation callstack probably doesn’t get you to a solution. You need to find the missing ‘Release’ call.

Here is what I did that worked pretty well:

Step #1: Get a good repro. Without a repro, you don’t have a chance for leak bugs.

Step #2: Look at all the reported leaks and find a leaf object that is leaked. Often times leaks chain where one object is leaked, but that object contains a reference to another object. So the other object must also leak. Before spending ages looking at all the AddRef/Release calls to a leaking object, make sure that you are looking at a leaf object. For an example, if all 'apple' objects contain references to their 'tree' objects and trees and apples are leaking, look at the apple leaks instead of the tree leaks.

Step #3: Find an instance of a particular object that leaks. Set a tracepoint on the destructor and contructor of the leaking class. On the constructor, I would recommend a When Hit message like ‘New Object: {(void*)this} $CALLSTACK’. On the destructor, try a When Hit message of ‘Delete Object: {(void*)this}’. Run your scenario and do some time text processing to find out which callstacks leaked and didn’t. In batch script:

for /f "tokens=3" %d in ('findstr /c:"New Object" debug.txt') do findstr %d debug.txt

Where debug.txt is a text file that I saved the content of the output window to. Now I can look at all the allocation callstacks for the object – both those that leaked and those that didn’t. You will need to find some pattern that will allow you to predict one leaking object instance. If your lucky, the pattern is something simple like ‘the first instance created’.

Step #4: Trace all the AddRef/Release calls. For this I would recommend setting a data tracepoint on the field that holds the ref count. To do this: Debug->New Breakpoint->New Databreakpoint and set the location to the address where the ref count is stored (ex: 0x10E89934). Then change the ‘When Hit…’ property to output a callstack and the current ref count -- ‘{*((DWORD*)0x10E89934) } $CALLSTACK’.
When your done, hopefully you have a reasonable list of AddRef/Release calls to your object. So its easy enough to match them up and see what went wrong.

One cause for 'The debugger is not properly installed' in Visual Studio 2002/3

Way back in 2004 I posted an article talking about the 'The debugger is not properly installed' in Visual Studio 2002/3. Today I wanted to briefly talk about another reason for seeing this problem that an extremely helpful user pointed out (thanks Joel!).

If running debugger diagnostics shows:

Result of 'CoCreateMachineDebugManager(&spMachine)' is '800401f9'

Then you may have bad registry entries under HKEY_CLASSES_ROOT\CLSID\{73b25ffd-f501-437b-8b11-7f0de383964f}. If you have an 'InprocHandler32' key under there, then this is your problem.

To fix it:

1. Open a command prompt
2. cd /d "%CommonProgramFiles%\Microsoft Shared\vs7debug"
3. mdm.exe /unregserver
4. Open regedit.exe and check if HKEY_CLASSES_ROOT\CLSID\{73b25ffd-f501-437b-8b11-7f0de383964f} still exists. I think it will. If so, delete it.
5. mdm.exe /regserver
6. net stop mdm
7. net start mdm
8. Relaunch Visual Studio