Rick Byers : PDBs

My school assignment required parsing PDB files

rmbyers — Tue, 09 Jun 2009 20:15:00 GMT

I'm just finishing up my Masters in Computer Science, and was surprised when I recently got an assignment that involved parsing PDB files. Over the years I've been involved in many discussions about why the Microsoft PDB format isn't public. John Robbins recently went as far as to say "The actual file format of a PDB file is a closely guarded secret", but that's probably and overstatement given that the CCI code which contains a C# managed-PDB reader has now been released on CodePlex. So needless to say, I didn't expect to be having to parse PDB files for school ... especially in a computational biology class.

Ok, so if you do a web-search now for "PDB file format", the first hit you get isn't for Microsoft's Program Database file format, but for the Protein Data Bank file format. Who ever came up with the idea that file types should be uniquely determined by a 3-letter extension that isn't coordinated by any central registry anyway? Now I've got to decide which way I want windows to treat PDB files (and both have been pretty important to me lately):

Customizing PDB lookup for source information in StackTrace

rmbyers — Fri, 22 Jun 2007 08:49:00 GMT

The System.Diagnostics.StackTrace class in .NET can be used to generate a textual representation of the current callstack. This is used, for example, by Exception.ToString(). If requested by the caller, StackTrace can include source file locations (file names and line numbers, etc.) for each frame whose module has a PDB file available to the CLR. PDB files are designed to be used primarily in development-time scenarios, and so the idea here is that when you're developing or testing your application and it spits out an exception (eg. to a log file, or an unhandled exception to the console), it will help you to debug the issue if you can see exactly where in the leaf method the exception was thrown, and where exactly each child function was called (actually it's not technically "exactly" - if JIT optimizations are enabled the results may be approximate, and frames may be missing completely due to inlining). If you've done much .NET programming, you probably knew all this already.

The more interesting (and less well documented) question I want to address is where exactly you must place your PDB files for this to work. The CLR will look next to the corresponding module (DLL or EXE), and also check a few other standard locations (those local paths specified by the _NT_SYMBOL_PATH environment variable for example, and I believe the Windows system directory). In fact, it's not really the CLR controlling any of this, but the ISymUnmanagedBinder::GetReaderForFile API from diasymreader.dll, which itself is implemented on top of the IDiaDataSource::loadDataForExe API. Since this support for source locations is designed for development time, when you're generally running binaries you've just built - the PDBs are almost always next to the binaries and this works great.

Occasionally we get requests from people who like this feature but complain that the CLR isn't flexible enough to find their PDB files where they want to put them. Sometimes this stems from wanting to use this feature for something it wasn't designed, such as shipping PDBs with your product and logging/reporting errors from the field. For that scenario you're usually MUCH better off using Windows Error Reporting and minidumps. You generally do not want to ship your PDBs to your customers (they're big, and can make it easier to reverse engineer your code - although this is a much bigger concern for unmanaged C++ code than .NET code). In other cases, you may want to generate machine-readable stack traces (with module names, method tokens and IL offsets), and then post-process them using PDB files at your location to get source location information.

But, there are a few scenarios where it does really make sense to want more flexibility in how PDBs are located for the StackTraces generated at runtime. For example, I recently got a request through product support from a customer with a large test environment where they were deploying their actual product. They keep all their PDBs (1TB+ of them!) on a symbol server, and they would like to be able to use them to generate stack traces with source info without having to deploy PDBs to all their test machines (in the proper directories). Although the CLR doesn't support this directly, there isn't any reason you can't implement this yourself. The CLR StackTrace class exposes StackFrame objects which have all the information you need to map back to source addresses given an ISymbolReader instance. ISymbolReader instances can be created directly (controlling PDB location policy manaually with ISymbolUnmanagedBinder2::GetReaderForFile2) by calling into diasymreader.dll through COM interop. I've posted sample code for a StackTraceSymbolProvider class that does this here (using MDbg's COM interop wrappers for diasymreader.dll).

Here's an example of how this code can be used to print out a StackTrace from an Exception while explicitly controlling the directories searched (searchPath is a semi-colon separated list of directories including SRV* entries for symbol servers), and whether things like a symbol server will be checked:

            catch (System.Exception e)
            {
                st = new StackTrace(e, true);
                StackTraceSymbolProvider stsp = new StackTraceSymbolProvider(searchPath,
                    SymSearchPolicies.AllowSymbolServerAccess |
                    SymSearchPolicies.AllowOriginalPathAccess |
                    SymSearchPolicies.AllowReferencePathAccess |
                    SymSearchPolicies.AllowRegistryAccess);

                Console.WriteLine("Custom stack trace:");
                Console.WriteLine(stsp.StackTraceToStringWithSourceInfo(st));
            }

To get this flexibility you have to re-implement some of the formatting done by StackTrace.ToString(), but you might want the flexibility to control this anyway (for example, it's easy to include column numbers in addition to line numbers). It's non-trivial to wire this all up (especially if you're not familiar with COM-interop), but it's all plumbing really. Hopefully this sample code will save some of you the hassle of figuring out this plumbing yourself.

DebuggingModes.IgnoreSymbolStoreSequencePoints

rmbyers — Fri, 09 Sep 2005 03:01:00 GMT

In my last post I gave an overview of the DebuggableAttribute, what values the C# compiler gives it, and how the CLR uses those values. I mentioned that with /debug+, the C# compiler sets the IgnoreSymbolStoreSequencePoints DebuggingModes bit, but I didn't describe what this bit does. Understanding sequence points and ths IgnoreSymbolStoreSequencePoints bit is important for anyone writing a compiler for .NET (including one that uses Reflection.Emit).

A sequence point is used to mark a spot in the IL code that corresponds to a specific location in the original source. For Reflection.Emit, they are emitted by the ILGenerator.MarkSequencePoint method. This serves two purposes. First to allow debuggers (and the StackTrace class) to map IL offsets back to source and line information. Secondly to tell the JIT compiler that it must preserve this location uniquely in the generated native code so that a user could, for example, set a breakpoint on it (see Mike's blog entry for more details).

Since they're only needed for debugging scenarios, the location of sequence points, and their mapping back to source locations, are stored in the PDB file. Unfortunately this means that the JIT compiler has to open the PDB file and read this data any time it compiles a method with optimizations disabled (if JIT optimizations are enabled, then it doesn't care about sequence points in order to get the best codegen possible). The PDB file format is quite complicated, and has a long legacy. So as you might imagine, there is an unfortunate performance penalty here.

Now in addition to the sequence points specified in the PDB file ("explicit sequence points"), the JIT also infers additional sequence points based on the IL code ("implicit sequence points"). For example, it adds an implicit sequence at every point where the IL evaluation stack is empty, or on any "nop" instruction. So in Whidbey we added the IgnoreSymbolStoreSequencePoints bit to allow compilers to indicate to the JIT that they are content with the JITs rules for implicit sequence points, and there is no need for the JIT to look in the PDB file for more.

So our official recommendation is that all compilers (including those using Reflection.Emit) set the IgnoreSymbolStoreSequencePoints bit and rely on "nop" instructions to ensure the JIT places sequence points at the right place. This can make a nice JIT performance improvement, and also has some other benefits related to not needing to read the PDB file at JIT time (eg. it may not be available). The Whidbey C# and VB compilers do this, but unfortunately the managed C++ compiler does not.