Peter Huene's Blog

Programmatic Coverage Analysis in Visual Studio 2010

phuene — Tue, 01 Dec 2009 19:31:34 GMT

As hinted upon in my last post, today’s entry will be on how to programmatically perform analysis on a Visual Studio coverage file in Visual Studio 2010.

The first step is to reference the coverage analysis assembly, Microsoft.VisualStudio.Coverage.Analysis.dll, in your project. This managed assembly can be found in “%ProgramFiles%\Microsoft Visual Studio 10.0\Common7\IDE\PrivateAssemblies” (use %ProgramFiles(x86)% on a 64-bit OS). The project system should copy the assembly locally for your application to use. This assembly has native x86 dependencies, so it can only be used from an x86 process. It is therefore marked as 32-bit required and if you plan on loading it on a 64-bit OS, ensure your entry point assembly is also marked as 32-bit required (i.e. built for the “x86” platform).

To run your application, you will also need to copy Microsoft.VisualStudio.Coverage.Symbols.dll to your output directory. This is actually a native module, so you won’t be able to add it as a reference. The easiest way I’ve found to do this is to “Add Existing Item” to the project, browse to the file (it’s also in PrivateAssemblies), click the little down arrow next to the “Add” button, select “Add As Link”, select the file in Solution Explorer, bring up the Properties Window, and change “Copy to Output Directory” to “Copy if newer”. Note that this module has a dependency on the Microsoft Debug Interface Access library (msdia100.dll), which is a native COM component that is registered when you install Visual Studio. You will need to register msdia100.dll using regsvr32.exe on a machine that doesn’t have Visual Studio installed if you plan on using the coverage analysis API.

Now that we’ve setup a project to use the analysis API, let me remind you about John Cunningham’s old post about using the Visual Studio 2005/2008 coverage analysis API. There has been a minor breaking change with the old API. The CoverageInfoManager class no longer exists. Instead, CoverageInfo objects are created via static methods on CoverageInfo itself. Here’s a quick comparison:

The old way to analyze a file:

        1: CoverageInfoManager.ExePath = "<executable_search_paths>";

       2: CoverageInfoManager.SymPath = "<symbol_search_paths>";

       3: CoverageInfo info = CoverageInfoManager.CreateInfoFromFile("<path_to_coverage_file>");

       4:  

       5: CoverageDS dataSet = info.BuildDataSet(null);

       6:  

       7: CoverageInfoManager.Shutdown();

And the new way:

       1: using (CoverageInfo info = CoverageInfo.CreateFromFile(

       2:         "<path_to_coverage_file>", 

       3:         new string[] { "<exe_path1>", "<exe_path2>" }, 

       4:         new string[] { "<sym_path1>", "<sym_path2>" }))

       5: {

       6:     CoverageDS dataSet = info.BuildDataSet();

       7: }

The executable search paths are locations where the instrumented modules can be found and the symbol search paths are where the instrumented symbols can be found. They are optional and there is an overload of CoverageInfo.CreateFromFile that takes only the path to the coverage file. The analysis engine will always check the same location as the coverage file to locate the instrumented modules and symbols in addition to the paths supplied.

The examples above use the CoverageDS type that is unchanged from Visual Studio 2008. This type is a typed data set containing the following tables: Module, Namespace, Class, Method, Lines, and SourceFileNames. It also has methods for exporting/importing the dataset’s data called ExportXml and ImportXml. The exported XML is the same format that Visual Studio uses when it exports a coverage file to XML in the coverage results tool window.

The downside to using CoverageDS is that it will load all of the coverage data into memory at the same time. This is simply not scalable when dealing with large numbers (i.e. many millions) of basic blocks, which usually results in 500 MB or more of data. Therefore, the Visual Studio 2010 coverage analysis API has an alternative method for enumerating the coverage data on demand. This also allows for easier filtering before method statistics are rolled up in the statistics of their classes, namespaces, and modules.

How to dump out the coverage statistics for each method:

       1: using (CoverageInfo info = CoverageInfo.CreateFromFile("foo.coverage"))

       2: {

       3:     List<BlockLineRange> lines = new List<BlockLineRange>();

       4:  

       5:     foreach (ICoverageModule module in info.Modules)

       6:     {

       7:         byte[] coverageBuffer = module.GetCoverageBuffer(null);

       8:  

       9:         using (ISymbolReader reader = module.Symbols.CreateReader())

        {

            uint methodId;

            string methodName;

            string undecoratedMethodName;

            string className;

            string namespaceName;

  

            lines.Clear();

            while (reader.GetNextMethod(

                out methodId,

                out methodName,

                out undecoratedMethodName,

                out className,

                out namespaceName,

                lines))

            {

                CoverageStatistics stats = CoverageInfo.GetMethodStatistics(coverageBuffer, lines);

  

                Console.WriteLine("Method {0}{1}{2}{3}{4} has:",

                    namespaceName == null ? "" : namespaceName,

                    string.IsNullOrEmpty(namespaceName) ? "" : ".",

                    className == null ? "" : className,

                    string.IsNullOrEmpty(className) ? "" : ".",

                    methodName

                    );

                Console.WriteLine("    {0} blocks covered", stats.BlocksCovered);

                Console.WriteLine("    {0} blocks not covered", stats.BlocksNotCovered);

                Console.WriteLine("    {0} lines covered", stats.LinesCovered);

                Console.WriteLine("    {0} lines partially covered", stats.LinesPartiallyCovered);

                Console.WriteLine("    {0} lines not covered", stats.LinesNotCovered);

            }

        }

    }

}

This example creates a CoverageInfo in the same manner as above, but instead of calling BuildDataSet, it enumerates through each module in the coverage file and then enumerates each method in the module to dump out its statistics. You’ll notice that we call GetCoverageBuffer on the module which returns a byte[]. There is a byte in this array for each basic block in the module. If the byte is zero, it means the basic block was not covered. If it is non-zero, it means the basic block was covered. So a simple method for counting the “raw” basic blocks covered/not covered would be to count the zero vs. non-zero bytes in this array. However, the total number of basic blocks reported in coverage statistics is usually less than the total number of basic blocks in the module because certain basic blocks are discarded by GetNextMethod (usually these basic blocks are for compiler-generated code). So keep that in mind if you want to analyze the coverage buffer directly.

Note that each method identifier is unique for a particular build of a module. Each module has two properties that uniquely identify it: Signature (a Guid) and SignatureAge (a uint). These properties actually correspond go the debug information’s signature information in the module and change whenever a module is re-linked. For incremental links (VC++), the signature may remain the same while the signature’s age counter will increment, so these two values need to be taken together to version a particular build of a module.

Also, with modules built by VC++ that were linked with COMDAT folding enabled (an optimization that usually comes into play when using templates), you may see multiple functions returned by GetNextMethod that map to the same basic blocks because the functions contained duplicate code and were folded into a single copy. To get around this, we only roll up a method’s statistics to the module’s statistics as long as we haven’t seen the starting basic block before (lines[0].BasicBlockIndex). That way, the module’s totals are always accurate, although this means you may not always sum up the namespace numbers in a CoverageDS to arrive at the module’s numbers. I generally recommend disabling identical COMDAT folding when collecting native code coverage data, as it gives you a better idea of what was actually executed by your tests.

I’ll leave filtering and doing more complicated analysis (i.e. rollup statistics) as an exercise for the reader. Hopefully this will serve as a good starting point for those interested in programmatically analyzing code coverage data using Visual Studio 2010.

Code Coverage in Visual Studio 2010

phuene — Tue, 01 Dec 2009 01:00:00 GMT

It’s been far, far too long since I’ve updated my blog, but I wanted to quickly cover what’s new in Visual Studio 2010 for code coverage (Premium and higher SKUs). Almost all of the changes are under the covers, but some are noteworthy:

Support for x86-64 instrumentation (native, mixed, and platform-specific managed code) and collection on a 64-bit Windows OS. To utilize this support in VS, set the host platform to “x64” in the test settings. From the command line, use the tools in “%ProgramFiles(x86)%\Microsoft Visual Studio 10.0\Team Tools\Performance Tools\x64”. NOTE: the vsperfcmd/vsperfmon you use must match the architecture of the process you’re collecting data from. Hopefully this will be fixed in the future.
Support for ASP.NET collection in IIS under Vista+ from within Visual Studio. Due to Session 0 Isolation (a security feature of Vista+), code coverage collection was prevented from working within VS. Collection had to be performed manually using the command line tools. In an elevated Visual Studio 2010 instance, Session 0 collection now works correctly.
Support for concurrent code coverage sessions in Visual Studio 2010 or Team Foundation Build 2010. Previously, code coverage collection was restricted to a single test run in a Windows session, so if multiple test runs were taking place at the same time with code coverage collection enabled, one could possibly fail saying the code coverage monitor was already running. This was occasionally observed in previous versions of Team Foundation Build when a test run had code coverage enabled. Now, each test run’s collection is isolated from one another, so this limitation has been removed.
Eliminated the common cause of “out of memory” errors from the TFS Coverage Analysis Service (CoverAn) that was responsible for merging code coverage data published against a Team Foundation Build. The service has been replaced with a background job that is capable of efficiently merging very large sets of coverage data. This change also eliminates the requirement that the service identity has to be granted access to the build drop location to perform its work.
Added the System.Diagnostics.CodeAnalysis.ExcludeFromCodeCoverageAttribute class to System.dll. This attribute can now be applied to classes, methods, and properties to exclude the collection of code coverage data. Instrumentation still respects the DebuggerNonUserCode and DebuggerHidden attributes to prevent instrumentation. However, as these attributes are also used by the debugger when stepping, consider switching to the ExcludeFromCodeCoverage attribute instead.
Revamped the analysis API to be more efficient and consistent. I will try to touch on this bullet point in a future blog post describing how to analyze Visual Studio code coverage data programmatically.

Additionally, a bunch of analysis fixes went in to improve the code coverage results and the merging of code coverage files. Unfortunately, the overall code coverage results UI (something I did not own when I owned code coverage) has remained unchanged from the previous version.

That’s it for now. I hope to blog a little bit more about code coverage before going in-depth into the test impact analysis features of Visual Studio 2010. Following that, I hope to discuss IntelliTrace since that is my current feature area.

Code Coverage Collection

phuene — Mon, 14 May 2007 23:52:37 GMT

By the end of my last post about code coverage instrumentation, we have instrumented our executable image and it's now ready for code coverage collection. Code coverage collection is the process of collecting the runtime data of what functions were covered during test execution. The end result of coverage collection will be a coverage data file which can be analyzed to show the coverage results to us in a meaningful way.

Visual Studio uses a separate tool to monitor, coordinate, and flush the coverage data to the coverage file. This tool is VSPerfMon.exe - called the "Visual Studio Performance Monitor" (named so because it was first used by the profiler). When the monitor is started, it listens for incoming connections from executables that have been instrumented for coverage. One of the details I didn't mention about instrumentation is that when an executable image is instrumented, the instrumentation process creates a dependency on VSCover90.dll (9.0 for Orcas, 8.0 for 2005) for the image being instrumented. This is accomplished either though the PE import table for native images or through an injected P/Invoke for managed images. VSCover90.dll is essentially the code coverage runtime and it contains the logic for communicating with the monitor. When VSCover90.dll is loaded and initialized, it will attempt a connection to the monitor. If the monitor is not running, the connection will silently fail and coverage collection will be disabled for the lifetime of the process. An instrumented image will execute normally without the monitor running. If, however, the monitor is running and the connection attempt succeeds, VSCover90.dll will register the hosting process and will begin coverage collection.

VSPerfMon.exe can be found with the other profiler and code coverage tools in "%ProgramFiles%\Microsoft Visual Studio 9.0\Team Tools\Performance Tools". To run the monitor as a foreground process, execute the following command from a command prompt:

vsperfmon /coverage /output:<path_to_coverage_file>

This will run the monitor in "stand-alone mode". In this mode you can see the processes that have connected to and disconnected from the monitor. The monitor traps Control-C, so to shutdown the monitor you will need to use another tool called VSPerfCmd ("Visual Studio Performance Command"). This tool communicates with the running monitor process to perform a variety of tasks. You would execute the following command line to shutdown the monitor:

vsperfcmd /shutdown

The command tool can also be used to start the monitor in the background. To do so, execute the following command line:

vsperfcmd /start:coverage /output:<path_to_coverage_file>

This command will cause the monitor to be spawned as a background process. You will use the same command as the one given above to shutdown the monitor.

There are a few important things to note about the monitor that can prevent successful coverage collection:

By default, the monitor only accepts connections from processes running under an identity that is a local administrator. The most common problem experienced with this is that no data is collected for ASP.NET web sites. This is due to ASP.NET web sites running under the Network Service identity by default, which is a restrictive identity meant for services that accept network connections. To solve this, there is a /USER switch on VSPerfMon/VSPerfCmd to allow the specified user access to the monitor. When you perform code coverage on a web site inside Visual Studio, a request is made by Visual Studio to discover what the identity of the worker process is. This identity is then passed to the monitor to allow the connection. This happens automatically before test execution when there is a web site checked in the code coverage section of the test run configuration.
By default, the monitor creates the objects and communication channels needed by the coverage runtime in a local session scope. Therefore, if you are running the monitor in a different session from the process that will connect to the monitor (e.g. if you are not logged on from the console session and you want a service to connect to the monitor), then you need to specify the /CrossSession switch to VsPerfMon/VsPerfCmd. This will instruct the monitor to create those objects in a global scope so that they are accessible from other sessions.

Let's recap what we have so far:

Instrumentation is performed using vsinstr.exe for native, mixed mode, and managed images. To instrument an image, execute the following: "vsinstr.exe /coverage <path_to_image>".
To start the monitor process that will collect the coverage data, execute the following: "vsperfcmd /start:coverage <path_to_coverage_file>".
Run the test(s) you want to collect coverage data for using the instrumented images.
To stop collection, execute the following: "vsperfcmd /shutdown".

We now have a coverage file which we can use to view coverage results. The coverage file can be imported into the "Code Coverage Results" tool window inside of Visual Studio. This will show you per-method statistics and double clicking on a method will result in opening the source file in the editor with coverage coloring turned on. I will present a walk-through of how to accomplish everything I've been talking about so far inside of Visual Studio in a future blog entry.

I will discuss how to use the coverage analysis API to read the coverage file programmatically in my next blog entry.

Code Coverage Instrumentation

phuene — Fri, 04 May 2007 06:02:00 GMT

For my first on-topic blog post, I would like to give an overview of how code coverage instrumentation works in Visual Studio (it seems like a good a place to start as any). For basic information about what code coverage is, check out wikipedia's Code Coverage topic. Code coverage is a feature available in Visual Studio Team Edition for Developers, Visual Studio Team Edition for Testers, and Visual Studio Team Suite.

One of the most common questions I get by people that know about code coverage is what kind of code coverage does Visual Studio support. Visual Studio uses a block-based statement (also known as C1 coverage) and condition coverage methodology. A block is commonly defined as a sequence of instructions (in this case x86 or CIL instructions) that have a single entry point and a single exit point. We consider an exit point to be a branch instruction, a function call, a return instruction, or, in the case of managed code, a throw instruction.

I think it'll be useful to to relate this to source code, so take this (rather silly) example in C++:

   1:  int foo(bool condition)

   2:  {

   3:      int i = 0;         /* block 0 */

   4:      if (condition)     /* block 0 */

   5:      {

   6:          i = 5;         /* block 1 */

   7:      }

   8:      else

   9:      {

  10:          i = bar();     /* block 2 and 3 */

  11:      }

  12:      return i;          /* block 4 */

  13:  }

Here is the generated debug x86 of the code above (I've replaced the real addresses with easier to read "pseudo-addresses" that erroneously assumes x86 is a one byte fixed-length instruction set):

Address:	Instruction:	Arguments:	Block:
0000	push	ebp	0
0001	mov	ebp,esp	0
0002	push	ecx	0
0003	mov	dword ptr [i],0	0
0004	movzx	eax,byte ptr [condition]	0
0005	test	eax,eax	0
0006	je	0009h	0
0007	mov	dword ptr [i],5	1 (due to branch)
0008	jmp	000Bh	1
0009	call	bar	2 (due to branch)
000A	mov	dword ptr [i],eax	3 (due to call)
000B	mov	eax,dword ptr[i]	4 (multiple entry points)
000C	mov	esp,ebp	4
000D	pop	ebp	4
000E	ret		4

Notice that everything up to and including the first branch instruction (je) is considered to be part of the first block. Everything up to and including the next branch instruction (jmp) is part of the second block. The third block is comprised of only the call instruction for the bar function. The fourth block is comprised of storing the return value from the call to bar in the variable i. You'll notice that per the definition of a block as a single entry and single exit sequence of instructions, the instruction at 0x000B has two entry points: as the next instruction from 0x000A and also from the unconditional jump at 0x0008. Therefore, it is also considered the start of a new block and everything up to the final ret instruction is the fifth block. So as you can see, a single line of source code can actually be more than one block.

The job of instrumentation is to modify the original executable image so that we can detect when blocks are "hit" during execution. To accomplish this, a few instructions (in the case of x86: a push, two movs, and a pop) are inserted before every block to toggle a byte in a buffer that says that the block was executed. You can easily see these instructions by disassembling an instrumented image. Additional information that is required to map a block back to the original source code is also stored in the instrumented image in the form of PE sections. The instrumentation tool relies on debug information to be present to perform the instrumentation. Because the instrumentation process modifies the image, a new debug information database will be written out and referenced from the instrumented executable. By default, the instrumentation tool does an "in-place" instrumentation and overwrites the image being instrumented. The original image is backed up just in case something goes wrong. The instrumented debug information database is actually written out as a different file, so the original debug information database does not need to be backed up.

The instrumentation tool used by Visual Studio is called vsinstr.exe. This is also the same instrumentation tool used by the profiler to perform trace profiling, however the instrumentation process is quite different than the one used for code coverage. You can find this tool (assuming you're using Orcas) in "%Program Files%\Microsoft Visual Studio 9.0\Team Tools\Performance Tools", which is not actually on the default path for a Visual Studio command prompt. In Orcas, vsinstr.exe supports the instrumentation of native x86, mixed mode x86, and managed images. Unfortunately, 64-bit support did not make it into Orcas, but it is something we would like to get in for a future release. To use vsinstr.exe, simply give it the /coverage switch and the path to the image to instrument:

vsinstr.exe /coverage foo.dll

This will generate foo.dll (instrumented), foo.instr.pdb (the instrumented pdb referenced by the instrumented foo.dll), and foo.dll.orig (the backed-up original foo.dll). I will cover exactly how to automatically accomplish the same thing inside of Visual Studio using a screenshot walkthrough in an upcoming blog entry.

Once an image is instrumented, it is ready for coverage collection, which will be the focus of my next post.

Welcome to my blog!

phuene — Wed, 02 May 2007 23:38:00 GMT

Hello, my name is Peter and I'm a recovering ex-softie. I returned a few months ago to Microsoft after a little more than two years away from the company. I graduated with a BS in Computer Science from Virginia Tech (I'm also a diehard hokie fan) in 2002. I was a three-time intern before joining Microsoft in as a developer on the Visual C++ project system team. During that time I contributed to Visual C++ 2002, 2003, and 2005.

I return as a developer on the diagnostics team that is part of Team Developer (Visual Studio Team Edition for Developers), or TeamDev as we like to call it. The diagnostics team owns the debugger, profiler, and code coverage tools within Visual Studio. I am responsible for the code coverage tools and that will be the primary focus of this blog. I will try to also talk about the great things the rest of TeamDev is doing as there are some fantastic things in the works for both Orcas and beyond.

I am by no means a prolific blogger, but I will try to get some good content sooner than later :).