Peter Huene's Blog

Code Coverage Collection

2007-05-14T23:52:37Z

By the end of my last post about code coverage instrumentation, we have instrumented our executable image and it's now ready for code coverage collection. Code coverage collection is the process of collecting the runtime data of what functions were covered during test execution. The end result of coverage collection will be a coverage data file which can be analyzed to show the coverage results to us in a meaningful way.

Visual Studio uses a separate tool to monitor, coordinate, and flush the coverage data to the coverage file. This tool is VSPerfMon.exe - called the "Visual Studio Performance Monitor" (named so because it was first used by the profiler). When the monitor is started, it listens for incoming connections from executables that have been instrumented for coverage. One of the details I didn't mention about instrumentation is that when an executable image is instrumented, the instrumentation process creates a dependency on VSCover90.dll (9.0 for Orcas, 8.0 for 2005) for the image being instrumented. This is accomplished either though the PE import table for native images or through an injected P/Invoke for managed images. VSCover90.dll is essentially the code coverage runtime and it contains the logic for communicating with the monitor. When VSCover90.dll is loaded and initialized, it will attempt a connection to the monitor. If the monitor is not running, the connection will silently fail and coverage collection will be disabled for the lifetime of the process. An instrumented image will execute normally without the monitor running. If, however, the monitor is running and the connection attempt succeeds, VSCover90.dll will register the hosting process and will begin coverage collection.

VSPerfMon.exe can be found with the other profiler and code coverage tools in "%ProgramFiles%\Microsoft Visual Studio 9.0\Team Tools\Performance Tools". To run the monitor as a foreground process, execute the following command from a command prompt:

vsperfmon /coverage /output:<path_to_coverage_file>

This will run the monitor in "stand-alone mode". In this mode you can see the processes that have connected to and disconnected from the monitor. The monitor traps Control-C, so to shutdown the monitor you will need to use another tool called VSPerfCmd ("Visual Studio Performance Command"). This tool communicates with the running monitor process to perform a variety of tasks. You would execute the following command line to shutdown the monitor:

vsperfcmd /shutdown

The command tool can also be used to start the monitor in the background. To do so, execute the following command line:

vsperfcmd /start:coverage /output:<path_to_coverage_file>

This command will cause the monitor to be spawned as a background process. You will use the same command as the one given above to shutdown the monitor.

There are a few important things to note about the monitor that can prevent successful coverage collection:

By default, the monitor only accepts connections from processes running under an identity that is a local administrator. The most common problem experienced with this is that no data is collected for ASP.NET web sites. This is due to ASP.NET web sites running under the Network Service identity by default, which is a restrictive identity meant for services that accept network connections. To solve this, there is a /USER switch on VSPerfMon/VSPerfCmd to allow the specified user access to the monitor. When you perform code coverage on a web site inside Visual Studio, a request is made by Visual Studio to discover what the identity of the worker process is. This identity is then passed to the monitor to allow the connection. This happens automatically before test execution when there is a web site checked in the code coverage section of the test run configuration.
By default, the monitor creates the objects and communication channels needed by the coverage runtime in a local session scope. Therefore, if you are running the monitor in a different session from the process that will connect to the monitor (e.g. if you are not logged on from the console session and you want a service to connect to the monitor), then you need to specify the /CrossSession switch to VsPerfMon/VsPerfCmd. This will instruct the monitor to create those objects in a global scope so that they are accessible from other sessions.

Let's recap what we have so far:

Instrumentation is performed using vsinstr.exe for native, mixed mode, and managed images. To instrument an image, execute the following: "vsinstr.exe /coverage <path_to_image>".
To start the monitor process that will collect the coverage data, execute the following: "vsperfcmd /start:coverage <path_to_coverage_file>".
Run the test(s) you want to collect coverage data for using the instrumented images.
To stop collection, execute the following: "vsperfcmd /shutdown".

We now have a coverage file which we can use to view coverage results. The coverage file can be imported into the "Code Coverage Results" tool window inside of Visual Studio. This will show you per-method statistics and double clicking on a method will result in opening the source file in the editor with coverage coloring turned on. I will present a walk-through of how to accomplish everything I've been talking about so far inside of Visual Studio in a future blog entry.

I will discuss how to use the coverage analysis API to read the coverage file programmatically in my next blog entry.

Code Coverage Instrumentation

2007-05-04T06:02:00Z

For my first on-topic blog post, I would like to give an overview of how code coverage instrumentation works in Visual Studio (it seems like a good a place to start as any). For basic information about what code coverage is, check out wikipedia's Code Coverage topic. Code coverage is a feature available in Visual Studio Team Edition for Developers, Visual Studio Team Edition for Testers, and Visual Studio Team Suite.

One of the most common questions I get by people that know about code coverage is what kind of code coverage does Visual Studio support. Visual Studio uses a block-based statement (also known as C1 coverage) and condition coverage methodology. A block is commonly defined as a sequence of instructions (in this case x86 or CIL instructions) that have a single entry point and a single exit point. We consider an exit point to be a branch instruction, a function call, a return instruction, or, in the case of managed code, a throw instruction.

I think it'll be useful to to relate this to source code, so take this (rather silly) example in C++:

   1:  int foo(bool condition)

   2:  {

   3:      int i = 0;         /* block 0 */

   4:      if (condition)     /* block 0 */

   5:      {

   6:          i = 5;         /* block 1 */

   7:      }

   8:      else

   9:      {

  10:          i = bar();     /* block 2 and 3 */

  11:      }

  12:      return i;          /* block 4 */

  13:  }

Here is the generated debug x86 of the code above (I've replaced the real addresses with easier to read "pseudo-addresses" that erroneously assumes x86 is a one byte fixed-length instruction set):

Address:	Instruction:	Arguments:	Block:
0000	push	ebp	0
0001	mov	ebp,esp	0
0002	push	ecx	0
0003	mov	dword ptr [i],0	0
0004	movzx	eax,byte ptr [condition]	0
0005	test	eax,eax	0
0006	je	0009h	0
0007	mov	dword ptr [i],5	1 (due to branch)
0008	jmp	000Bh	1
0009	call	bar	2 (due to branch)
000A	mov	dword ptr [i],eax	3 (due to call)
000B	mov	eax,dword ptr[i]	4 (multiple entry points)
000C	mov	esp,ebp	4
000D	pop	ebp	4
000E	ret		4

Notice that everything up to and including the first branch instruction (je) is considered to be part of the first block. Everything up to and including the next branch instruction (jmp) is part of the second block. The third block is comprised of only the call instruction for the bar function. The fourth block is comprised of storing the return value from the call to bar in the variable i. You'll notice that per the definition of a block as a single entry and single exit sequence of instructions, the instruction at 0x000B has two entry points: as the next instruction from 0x000A and also from the unconditional jump at 0x0008. Therefore, it is also considered the start of a new block and everything up to the final ret instruction is the fifth block. So as you can see, a single line of source code can actually be more than one block.

The job of instrumentation is to modify the original executable image so that we can detect when blocks are "hit" during execution. To accomplish this, a few instructions (in the case of x86: a push, two movs, and a pop) are inserted before every block to toggle a byte in a buffer that says that the block was executed. You can easily see these instructions by disassembling an instrumented image. Additional information that is required to map a block back to the original source code is also stored in the instrumented image in the form of PE sections. The instrumentation tool relies on debug information to be present to perform the instrumentation. Because the instrumentation process modifies the image, a new debug information database will be written out and referenced from the instrumented executable. By default, the instrumentation tool does an "in-place" instrumentation and overwrites the image being instrumented. The original image is backed up just in case something goes wrong. The instrumented debug information database is actually written out as a different file, so the original debug information database does not need to be backed up.

The instrumentation tool used by Visual Studio is called vsinstr.exe. This is also the same instrumentation tool used by the profiler to perform trace profiling, however the instrumentation process is quite different than the one used for code coverage. You can find this tool (assuming you're using Orcas) in "%Program Files%\Microsoft Visual Studio 9.0\Team Tools\Performance Tools", which is not actually on the default path for a Visual Studio command prompt. In Orcas, vsinstr.exe supports the instrumentation of native x86, mixed mode x86, and managed images. Unfortunately, 64-bit support did not make it into Orcas, but it is something we would like to get in for a future release. To use vsinstr.exe, simply give it the /coverage switch and the path to the image to instrument:

vsinstr.exe /coverage foo.dll

This will generate foo.dll (instrumented), foo.instr.pdb (the instrumented pdb referenced by the instrumented foo.dll), and foo.dll.orig (the backed-up original foo.dll). I will cover exactly how to automatically accomplish the same thing inside of Visual Studio using a screenshot walkthrough in an upcoming blog entry.

Once an image is instrumented, it is ready for coverage collection, which will be the focus of my next post.

Welcome to my blog!

2007-05-02T23:38:00Z

Hello, my name is Peter and I'm a recovering ex-softie. I returned a few months ago to Microsoft after a little more than two years away from the company. I graduated with a BS in Computer Science from Virginia Tech (I'm also a diehard hokie fan) in 2002. I was a three-time intern before joining Microsoft in as a developer on the Visual C++ project system team. During that time I contributed to Visual C++ 2002, 2003, and 2005.

I return as a developer on the diagnostics team that is part of Team Developer (Visual Studio Team Edition for Developers), or TeamDev as we like to call it. The diagnostics team owns the debugger, profiler, and code coverage tools within Visual Studio. I am responsible for the code coverage tools and that will be the primary focus of this blog. I will try to also talk about the great things the rest of TeamDev is doing as there are some fantastic things in the works for both Orcas and beyond.

I am by no means a prolific blogger, but I will try to get some good content sooner than later :).