Peter Huene's Blog

Test Impact Analysis with Microsoft Test Manager 2010

2010-03-12T18:55:53Z

So I’ve had a little bit of a dry spell with respect to blog entries of late. I was planning on (eventually) writing a detailed walkthrough of using test impact analysis with Microsoft Test Manager to view impact to manual tests. However, I stumbled upon Danny Crone from nFocus’ video that covers it in more detail than what I was going to do. So check out Danny’s video if you’re interested in seeing test impact analysis in action with MTM and Team Foundation Build.

Test Impact with Team Foundation Build 2010

2009-12-09T23:36:43Z

So in my last blog entry, I touched upon how you could use the Test Impact View in Visual Studio 2010. Today’s topic is how to use the test impact feature with Team Foundation Build 2010. Team Foundation Build 2010 is part of Team Foundation Server 2010 and has many new features, including the very cool "Gated Check-in” feature that allows you to configure a build to validate a shelveset before it ever gets checked into the product. Among those many new features is support for test impact analysis. Let’s walk through creating a simple class library, creating unit tests, creating a build definition, and modifying source code to view test impact in Team Foundation Build 2010.

To get started, let’s create a new C# class library project called “DemoLibrary” (a more ingenious name there never was):

Note that I’ve checked the “Add to source control” checkbox. This will prompt us to add the solution to Team Foundation Server after the project is created, like so:

Here, I am adding the solution to the “Demo Project” team project I already created on my Team Foundation Server. Next, I renamed the generic “Class1.cs” to “DemoClass.cs”, which refactored Class1 to DemoClass. Here’s what my brilliant demo library source looks like:

        1: using System;

       2:  

       3: namespace DemoLibrary

       4: {

       5:     public class DemoClass

       6:     {

       7:         public static void TestMe()

       8:         {

       9:             Console.WriteLine("TestMe() was called.");

      10:         }

      11:     }

      12: }

So now that we have a rather useless library to test, let’s create a unit test for the TestMe method. Right click inside of TestMe in the editor and select “Create Unit Tests…”. Doing so will bring up this dialog:

After clicking the OK button, it will prompt you for a name of the new test project. Let’s call our test project “DemoLibraryTests”. Having “Tests” in the name will come in handy later on. Let’s change the code for the unit test to simply:

       1: using DemoLibrary;

       2: using Microsoft.VisualStudio.TestTools.UnitTesting;

       3: using System;

       4:  

       5: namespace DemoLibraryTests

       6: {

       7:     [TestClass()]

       8:     public class DemoClassTest

       9:     {

      10:         [TestMethod()]

      11:         public void TestMeTest()

      12:         {

      13:             DemoClass.TestMe();

      14:         }

      15:     }

      16: }

As you can see, all the unit test does is call the TestMe method in the DemoClass class. As TestMe never throws and the test never asserts, this test should always pass. Obviously it’s not a useful test, but this walkthrough is more about how to use the test impact feature than how to properly test your source code. Now let’s turn on test impact locally by changing the active test settings to the “Trace and Test Impact” settings (the “trace” part of the name is for IntelliTrace; if the test fails, you’ll get an .iTrace file you can examine for diagnostic events and exceptions):

Now that we’ve switched the active test settings to collect test impact data, let’s run the test locally before checking in. Once we’ve verified the test passes, check the files in. Right click on the solution in Solution Explorer and select “Check In…” to bring up this dialog:

After checking in the files, it’s time to setup a Team Foundation Build to build our projects and run tests. To do so, bring up Team Explorer, right click on “Builds” and select “New Build Definition…”, to bring up the build definition editor:

The first thing you’ll need to do is give the build definition a name on the “General” tab. I gave my build definition the creative name of “Demo Build”. The next thing to do is to set the build drop location on the “Build Defaults” tab, shown above. The build drop location is a share you would like the builds published to. The share needs to grant read/write privileges to the identity of the build agent service so that it can publish the build outputs.

Next, we need to change properties in the “Process” tab in the build definition editor. This section controls what gets built and tested. There are three important properties for us in this scenario: Items to Build (i.e. what solutions/projects to build), Automated Tests (i.e. what tests to run), and Analyze Test Impact (i.e. should test impact analysis be performed? The default is yes.). Clicking on the “…” button in the “Items to Build” property brings up this dialog:

With the build of Visual Studio 2010 I’m using to walkthrough this scenario, Team Foundation Build was smart enough to pre-populate this with the solution we’re working on already. I’m not sure if this functionality is in Beta 2 or not, so I’m displaying this screenshot to show where you would select the solution to build. Next, let’s configure what tests to run by clicking on the “…” button in the “Automated Tests” property:

Again, the build of Team Foundation Build I’m using was smart enough to default to running all tests in files that match *test*.dll. Because we named our unit test project “DemoLibraryTests”, Team Foundation Build will automatically run our tests without us changing anything. By default, Team Foundation Build will run tests using the Local.testsettings file. This file will not have the test impact diagnostic data adapter enabled. To enable test impact data collection, you need to either enable the test impact diagnostic data adapter in Local.testsettings, or switch the test settings to one that already has it enabled. The test project already created a test settings file that supports test impact data collection, so let’s click the Browse button above to change it to TraceAndTestImpact.testsettings:

Once this is done, save the new build definition. A “Demo Build” node should appear under the “Builds” node in Team Explorer. Let’s queue up a new build by right clicking on the “Demo Build” node and selecting “Queue New Build…”. This brings up a dialog to override build definition defaults if so desired. Just click the “Queue” button to queue the build using the defaults provided in the build definition. Double click on the build in progress in Build Explorer. This will show you what the build is currently doing. The build should succeed and the build summary page should look like this:

We see here that our unit test ran successfully and no tests were impacted. As this is the very first build of this build definition, there won’t be any impacted tests because tests have never been run with test impact data collected. Now comes the fun part: changing the code. Open up DemoClass.cs and simply change the “TestMe() was called.” string to something different; this will automatically check out the file for edit. I chose to change the period to an exclamation point. An innocuous change, but even a seemingly innocuous change can cause tests to fail. Build the solution and view the test impact view by going to Test –> Windows –> Test Impact View. Test Impact View should have detected the code change and recommended that the unit test be run, like this:

As you can see, it does recommend TestMeTest be run to verify the change I made to TestMe. Let’s click the “Run Tests” link in the Test Impact view to run the impacted tests. The test should pass. Now we’re ready to check in the change. Right click on the solution in Solution Explorer and select “Check In…” again to bring up this dialog:

Check in the change. Queue up a new “Demo Build” build. During the build, a special activity will notice the change to TestMe and record that the unit test was impacted for that build. You will find this in the Test Impact section of the build summary page:

This shows that the unit test was impacted by 1 code change. What if the change had actually caused the test to fail? Wouldn’t it be nice if we could easily figure out what changes might have caused the test to fail? That’s where the “1 code change(s)” link shown above comes in. Clicking this will give you more detail in what code changes likely caused the test to be impacted. Let’s click the link to bring up the code changes dialog:

As we can see here, the method DemoLibrary.DemoClass.TestMe() was modified and this caused the test to be impacted. We then get a list of all changesets that went into the build that modified the file. We can then go through each changeset to identify which may have broken the test. Clicking the “Compare Changes” link will compare only the file containing an impacting code change with the previous version:

Here we see I changed the ‘.’ to a ‘!’ which changed TestMe and ultimately impacted the test in question.

Let’s end this long walkthrough here. Hopefully you find this information useful in setting up test impact analysis in Team Foundation Build 2010. My next related blog entry will be using the test impact analysis feature from the new Microsoft Test and Lab Management product for manual tests.

Test Impact Analysis in Visual Studio 2010

2009-12-08T01:31:59Z

One of the new features you’ll find in Visual Studio 2010 (Premium and Ultimate editions) is test impact analysis. Inside of Visual Studio, this feature is used to immediately inform the developer of what tests they should run to verify the code changes they are making. You can also find the feature as part of Team Foundation Build and Microsoft Test and Lab Management, but I will limit today’s blog post to what you’ll find in Visual Studio.

I should call out the fact that this feature only supports managed code (e.g. VB.NET,C#,C++/CLI) in 2010. Native code is not yet supported, but it is something we would like to add support for in a future release.

To get started, let’s create a C# unit test project called “DemoProject”. Here’s what it might look like with some additional unit test code:

As you can see, I’ve defined two tests that each call a different private method. This is obviously a very contrived example, but hopefully I’ll be able to demonstrate how this is applicable to your own source base by the end of this blog entry.

Next, let’s bring up the Test Impact View. This is a new tool window that exposes the test impact analysis feature in Visual Studio. You can show the Test Impact View by selecting the “Test –> Windows –> Test Impact View” menu item, as seen here:

It should look something like this (note, the images here may display slightly different text than what you’ll find in Beta 2 because these are from a newer build):

I’ve highlighted an important link in the view above. This link is visible when the active test settings does not have the “test impact diagnostic data adapter” (that’s a mouthful, isn’t it?) enabled. Diagnostic data adapters allow the capturing of data while tests are executing. You’ll also find a diagnostic data adapter for IntelliTrace, code coverage, and more, in the test settings. The way the test impact diagnostic data adapter works is by dynamically instrumenting your code at runtime to record what was executed by each test. Each test that executed will produce its own “testimpact.xml” file attachment. The contents of this file are not something I’m going to discuss at this time.

Clicking the link will enable test impact data collection when tests are run. Let’s click the link and then run all tests by selecting the “Test –> Run –> All Tests In Solution” menu item, to arrive at this:

One thing I’ll mention is that we only collect data for passing tests. If a test fails, we’ll continue to use the last set of data we successfully collected. Now comes the interesting part: making a change to the source code. Let’s insert the following line into CalledByFirstTest and then build the solution:

        1: System.Console.WriteLine("Hello world");

Test impact analysis will analyze, in the background, the resulting assemblies produced by the build to look for meaningful changes to method instructions or data such as strings. This means that changes to comments or changes that are optimized away by the compiler (i.e. foo + 10 => foo + 5 + 5) are ignored by test impact analysis. Since we know TestMethod1 calls the changed method CalledByFirstTest, the tool window immediately recommends that we run TestMethod1 to verify the change:

Now let’s make the same change to the other private method and build again:

We now have two tests impacted. The selected impacted test is TestMethod1 in and the lower pane in the view is showing the code changes test impact analysis thinks is currently impacting the test. If we selected the other impacted test, we would see CalledBySecondTest in the lower pane, as expected.

I’ve highlighted three buttons at the top of the Test Impact View in the image above. From left to right: the first button is for displaying impacted tests (this is the default view), the second is for displaying the code changes remaining to be verified, and the third tab is for showing tests known to call a method. Clicking on the second button from the left results in this:

Here we see the two methods we changed. Selecting a code change shows the tests that need to run to verify it. Once all impacted tests are run successfully, the code change will no longer show up in this list until the code is changed again. To display tests known to call a particular method, right click inside of the method in the editor and select “Show Calling Tests”, like this:

Clicking this menu item will switch the test impact view into the “show calling tests” mode:

The view displays a list of tests known (i.e. have been run with test impact data collection enabled) to call the method in question. If you frequently run unit tests inside of Visual Studio, this is a good way to quickly tell if the method you’re modifying will impact tests if it is changed. Now let’s run all impacted tests by selecting “Test –> Run –> All Impacted Tests” menu item or by using the keyboard shortcut (Ctrl+R,Y). The two tests will run successfully, shown here:

As both tests have run successfully, the number of impacted tests falls to zero and there are no outstanding code changes to verify. If you change the methods again, the test impact view will display what needs to be run to verify the changes.

As I previously stated, this example is a rather silly one: we only use a single test project and the code being modified is in the test class itself. In practice, you may have a large source base of code and a large number of tests that test your code. If tests are run with test impact data collection enabled, the tests exist in the current solution context, and the code being modified also exists in a project in the current solution, then the test impact view will recommend what tests to run when you make your code changes.

I encourage you to play around with the feature beyond what is covered in this blog entry, such as with Coded UI tests (another new VS 2010 feature). I think it’s rather neat to be able to record UI tests, run them with test impact data collection enabled, make changes to methods that are traditionally hard to test via a unit test (e.g. some UI event you can’t mock up easily) and build, hit Ctrl+R,Y (run all impacted tests), and then watch your application be UI tested. Enjoy.

Programmatic Code Coverage Data Merging in Visual Studio 2010

2009-12-04T23:40:45Z

In my last post, I demonstrated how you could use the coverage analysis assembly to analyze coverage files. Today I will quickly cover how to programmatically merge multiple coverage files together. This replicates the functionality that the code coverage results tool window has for merging coverage data from multiple test runs together. The following assumes you’ve read the instructions in the previous post to reference the analysis assembly and use the Microsoft.VisualStudio.Coverage.Analysis namespace.

This function joins coverage files together into a single CoverageInfo using the Visual Studio 2010 API:

        1: static CoverageInfo JoinCoverageFiles(IEnumerable<string> files)

       2: {

       3:     if (files == null)

       4:         throw new ArgumentNullException("files");

       5:  

       6:     // This will represent the joined coverage files

       7:     CoverageInfo result = null;

       8:  

       9:     try

    {

        foreach (string file in files)

        {

            // Create from the current file

            CoverageInfo current = CoverageInfo.CreateFromFile(file);

            if (result == null)

            {

                // First time through, assign to result

                result = current;

                continue;

            }

  

            // Not the first time through, join the result with the current

            CoverageInfo joined = null;

            try

            {

                joined = CoverageInfo.Join(result, current);

            }

            finally

            {

                // Dispose current and result

                current.Dispose();

                current = null;

                result.Dispose();

                result = null;

            }

  

            result = joined;

        }

    }

    catch (Exception)

    {

        if (result != null)

        {

            result.Dispose();

        }

        throw;

    }

  

    return result;

}

This function could be called like this:

       1: try

       2: {

       3:     using (CoverageInfo info = JoinCoverageFiles(files))

       4:     {

       5:         // Dump out the info like you would if it weren't joined

       6:     }

       7: }

       8: catch (CoverageAnalysisException ex)

       9: {

      10:     Console.WriteLine(ex);

      11: }

One thing I left out in my previous post’s examples is CoverageAnalysisException. This is the base type for exceptions thrown by the analysis engine, so it will be one you will want to catch when using the analysis API. Analysis exceptions are usually the result of a missing instrumented module (ImageNotFoundException) or .pdb file (SymbolsNotFoundException).

How does joining two coverage files work under the covers? It’s rather simple, actually. When joining two coverage files together, we first union the modules together in the two coverage files. Remember the ICoverageModule.Signature and ICoverageModule.SignatureAge properties? These two properties are the unique identity of the module. If the module is found in both of the coverage files we simply bitwise OR their coverage buffers together to arrive at the merged coverage data. If a module is only found in one coverage file, we use the coverage buffer that was collected for it as is.

As you might notice from this scheme, we will only merge module data together for modules that are identical. When you instrument a module, we record the original file’s signature data in the instrumented module. This means you can instrument an original module multiple times and merge the data between different instrumented copies together, provided that each instrumentation was performed the same way. However, once the original module is rebuilt, you can no longer merge coverage data collected for the old version with data collected from the newer version. When you attempt to do so, you’ll get two modules of the same name in the resulting coverage data distinguished by the time they were built. In the future we hope to do a better job when merging coverage data between different builds of the same module.

Please note that what I’ve described here is specific to Visual Studio 2010. Earlier versions used a weaker heuristic to determine if module data can be merged together, sometimes resulting in undesirable behavior. With 2010, you get a consistent merge every time, provided the coverage data came from the same original module.

Well, this is it for my code coverage blog posts for a while. My next series of posts will be a walk through of the test impact features in Visual Studio 2010, including a walk through of how to use it from Team Foundation Build and the new Microsoft Test and Lab Management product.

Programmatic Coverage Analysis in Visual Studio 2010

2009-12-01T19:31:00Z

As hinted upon in my last post, today’s entry will be on how to programmatically perform analysis on a Visual Studio coverage file in Visual Studio 2010.

The first step is to reference the coverage analysis assembly, Microsoft.VisualStudio.Coverage.Analysis.dll, in your project. This managed assembly can be found in “%ProgramFiles%\Microsoft Visual Studio 10.0\Common7\IDE\PrivateAssemblies” (use %ProgramFiles(x86)% on a 64-bit OS). The project system should copy the assembly locally for your application to use. This assembly has native x86 dependencies, so it can only be used from an x86 process. It is therefore marked as 32-bit required and if you plan on loading it on a 64-bit OS, ensure your entry point assembly is also marked as 32-bit required (i.e. built for the “x86” platform).

To run your application, you will also need to copy Microsoft.VisualStudio.Coverage.Symbols.dll and dbghelp.dll to your output directory. These are both native modules, so you won’t be able to add them as a reference. The easiest way I’ve found to do this is to “Add Existing Item” to the project, browse to the files (both are also in PrivateAssemblies), click the little down arrow next to the “Add” button, select “Add As Link”, select the files in Solution Explorer, bring up the Properties Window, and change “Copy to Output Directory” to “Copy if newer”. Note that Microsoft.VisualStudio.Coverage.Symbols.dll has a dependency on the Microsoft Debug Interface Access library (msdia100.dll), which is a native COM component that is registered when you install Visual Studio. You will need to register msdia100.dll using regsvr32.exe on a machine that doesn’t have Visual Studio installed if you plan on using the coverage analysis API.

Next you’ll want to use the namespace containing the core implementation:

        1: using Microsoft.VisualStudio.Coverage.Analysis;

Now that we’ve setup a project to use the analysis API, let’s recall John Cunningham’s old post about using the Visual Studio 2005/2008 coverage analysis API. There has been a minor breaking change with the old API. The CoverageInfoManager class no longer exists. Instead, CoverageInfo objects are created via static methods on CoverageInfo itself. Here’s a quick comparison:

The old way to analyze a file:

       1: CoverageInfoManager.ExePath = "<executable_search_paths>";

       2: CoverageInfoManager.SymPath = "<symbol_search_paths>";

       3: CoverageInfo info = CoverageInfoManager.CreateInfoFromFile("<path_to_coverage_file>");

       4:  

       5: CoverageDS dataSet = info.BuildDataSet(null);

       6:  

       7: CoverageInfoManager.Shutdown();

And the new way:

       1: using (CoverageInfo info = CoverageInfo.CreateFromFile(

       2:         "<path_to_coverage_file>", 

       3:         new string[] { "<exe_path1>", "<exe_path2>" }, 

       4:         new string[] { "<sym_path1>", "<sym_path2>" }))

       5: {

       6:     CoverageDS dataSet = info.BuildDataSet();

       7: }

The executable search paths are locations where the instrumented modules can be found and the symbol search paths are where the instrumented symbols can be found. They are optional and there is an overload of CoverageInfo.CreateFromFile that takes only the path to the coverage file. The analysis engine will always check the same location as the coverage file to locate the instrumented modules and symbols in addition to the paths supplied.

The examples above use the CoverageDS type that is unchanged from Visual Studio 2008. This type is a typed data set containing the following tables: Module, Namespace, Class, Method, Lines, and SourceFileNames. It also has methods for exporting/importing the dataset’s data called ExportXml and ImportXml. The exported XML is the same format that Visual Studio uses when it exports a coverage file to XML in the coverage results tool window.

The downside to using CoverageDS is that it will load all of the coverage data into memory at the same time. This is simply not scalable when dealing with large numbers (i.e. many millions) of basic blocks, which usually results in 500 MB or more of data. Therefore, the Visual Studio 2010 coverage analysis API has an alternative method for enumerating the coverage data on demand. This also allows for easier filtering before method statistics are rolled up in the statistics of their classes, namespaces, and modules.

How to dump out the coverage statistics for each method:

       1: using (CoverageInfo info = CoverageInfo.CreateFromFile("foo.coverage"))

       2: {

       3:     List<BlockLineRange> lines = new List<BlockLineRange>();

       4:  

       5:     foreach (ICoverageModule module in info.Modules)

       6:     {

       7:         byte[] coverageBuffer = module.GetCoverageBuffer(null);

       8:  

       9:         using (ISymbolReader reader = module.Symbols.CreateReader())

        {

            uint methodId;

            string methodName;

            string undecoratedMethodName;

            string className;

            string namespaceName;

  

            lines.Clear();

            while (reader.GetNextMethod(

                out methodId,

                out methodName,

                out undecoratedMethodName,

                out className,

                out namespaceName,

                lines))

            {

                CoverageStatistics stats = CoverageInfo.GetMethodStatistics(coverageBuffer, lines);

  

                Console.WriteLine("Method {0}{1}{2}{3}{4} has:",

                    namespaceName == null ? "" : namespaceName,

                    string.IsNullOrEmpty(namespaceName) ? "" : ".",

                    className == null ? "" : className,

                    string.IsNullOrEmpty(className) ? "" : ".",

                    methodName

                    );

                Console.WriteLine("    {0} blocks covered", stats.BlocksCovered);

                Console.WriteLine("    {0} blocks not covered", stats.BlocksNotCovered);

                Console.WriteLine("    {0} lines covered", stats.LinesCovered);

                Console.WriteLine("    {0} lines partially covered", stats.LinesPartiallyCovered);

                Console.WriteLine("    {0} lines not covered", stats.LinesNotCovered);

                lines.Clear();

            }

        }

    }

}

This example creates a CoverageInfo in the same manner as above, but instead of calling BuildDataSet, it enumerates through each module in the coverage file and then enumerates each method in the module to dump out its statistics. You’ll notice that we call GetCoverageBuffer on the module which returns a byte[]. There is a byte in this array for each basic block in the module. If the byte is zero, it means the basic block was not covered. If it is non-zero, it means the basic block was covered. So a simple method for counting the “raw” basic blocks covered/not covered would be to count the zero vs. non-zero bytes in this array. However, the total number of basic blocks reported in coverage statistics is usually less than the total number of basic blocks in the module because certain basic blocks are discarded by GetNextMethod (usually these basic blocks are for compiler-generated code). So keep that in mind if you want to analyze the coverage buffer directly.

Note that each method identifier is unique for a particular build of a module. Each module has two properties that uniquely identify it: Signature (a Guid) and SignatureAge (a uint). These properties actually correspond go the debug information’s signature information in the module and change whenever a module is re-linked. For incremental links (VC++), the signature may remain the same while the signature’s age counter will increment, so these two values need to be taken together to version a particular build of a module.

Also, with modules built by VC++ that were linked with COMDAT folding enabled (an optimization that usually comes into play when using templates), you may see multiple functions returned by GetNextMethod that map to the same basic blocks because the functions contained duplicate code and were folded into a single copy. To get around this, we only roll up a method’s statistics to the module’s statistics as long as we haven’t seen the starting basic block before (lines[0].BasicBlockIndex). That way, the module’s totals are always accurate, although this means you may not always sum up the namespace numbers in a CoverageDS to arrive at the module’s numbers. I generally recommend disabling identical COMDAT folding when collecting native code coverage data, as it gives you a better idea of what was actually executed by your tests.

I’ll leave filtering and doing more complicated analysis (i.e. rollup statistics) as an exercise for the reader. Hopefully this will serve as a good starting point for those interested in programmatically analyzing code coverage data using Visual Studio 2010.

Code Coverage in Visual Studio 2010

2009-12-01T01:00:00Z

It’s been far, far too long since I’ve updated my blog, but I wanted to quickly cover what’s new in Visual Studio 2010 for code coverage (Premium and higher SKUs). Almost all of the changes are under the covers, but some are noteworthy:

Support for x86-64 instrumentation (native, mixed, and platform-specific managed code) and collection on a 64-bit Windows OS. To utilize this support in VS, set the host platform to “x64” in the test settings. From the command line, use the tools in “%ProgramFiles(x86)%\Microsoft Visual Studio 10.0\Team Tools\Performance Tools\x64”. NOTE: the vsperfcmd/vsperfmon you use must match the architecture of the process you’re collecting data from. Hopefully this will be fixed in the future.
Support for ASP.NET collection in IIS under Vista+ from within Visual Studio. Due to Session 0 Isolation (a security feature of Vista+), code coverage collection was prevented from working within VS. Collection had to be performed manually using the command line tools. In an elevated Visual Studio 2010 instance, Session 0 collection now works correctly.
Support for concurrent code coverage sessions in Visual Studio 2010 or Team Foundation Build 2010. Previously, code coverage collection was restricted to a single test run in a Windows session, so if multiple test runs were taking place at the same time with code coverage collection enabled, one could possibly fail saying the code coverage monitor was already running. This was occasionally observed in previous versions of Team Foundation Build when a test run had code coverage enabled. Now, each test run’s collection is isolated from one another, so this limitation has been removed.
Eliminated the common cause of “out of memory” errors from the TFS Coverage Analysis Service (CoverAn) that was responsible for merging code coverage data published against a Team Foundation Build. The service has been replaced with a background job that is capable of efficiently merging very large sets of coverage data. This change also eliminates the requirement that the service identity has to be granted access to the build drop location to perform its work.
Added the System.Diagnostics.CodeAnalysis.ExcludeFromCodeCoverageAttribute class to System.dll. This attribute can now be applied to classes, methods, and properties to exclude the collection of code coverage data. Instrumentation still respects the DebuggerNonUserCode and DebuggerHidden attributes to prevent instrumentation. However, as these attributes are also used by the debugger when stepping, consider switching to the ExcludeFromCodeCoverage attribute instead.
Revamped the analysis API to be more efficient and consistent. I will try to touch on this bullet point in a future blog post describing how to analyze Visual Studio code coverage data programmatically.

Additionally, a bunch of analysis fixes went in to improve the code coverage results and the merging of code coverage files. Unfortunately, the overall code coverage results UI (something I did not own when I owned code coverage) has remained unchanged from the previous version.

That’s it for now. I hope to blog a little bit more about code coverage before going in-depth into the test impact analysis features of Visual Studio 2010. Following that, I hope to discuss IntelliTrace since that is my current feature area.

Code Coverage Collection

2007-05-14T23:52:37Z

By the end of my last post about code coverage instrumentation, we have instrumented our executable image and it's now ready for code coverage collection. Code coverage collection is the process of collecting the runtime data of what functions were covered during test execution. The end result of coverage collection will be a coverage data file which can be analyzed to show the coverage results to us in a meaningful way.

Visual Studio uses a separate tool to monitor, coordinate, and flush the coverage data to the coverage file. This tool is VSPerfMon.exe - called the "Visual Studio Performance Monitor" (named so because it was first used by the profiler). When the monitor is started, it listens for incoming connections from executables that have been instrumented for coverage. One of the details I didn't mention about instrumentation is that when an executable image is instrumented, the instrumentation process creates a dependency on VSCover90.dll (9.0 for Orcas, 8.0 for 2005) for the image being instrumented. This is accomplished either though the PE import table for native images or through an injected P/Invoke for managed images. VSCover90.dll is essentially the code coverage runtime and it contains the logic for communicating with the monitor. When VSCover90.dll is loaded and initialized, it will attempt a connection to the monitor. If the monitor is not running, the connection will silently fail and coverage collection will be disabled for the lifetime of the process. An instrumented image will execute normally without the monitor running. If, however, the monitor is running and the connection attempt succeeds, VSCover90.dll will register the hosting process and will begin coverage collection.

VSPerfMon.exe can be found with the other profiler and code coverage tools in "%ProgramFiles%\Microsoft Visual Studio 9.0\Team Tools\Performance Tools". To run the monitor as a foreground process, execute the following command from a command prompt:

vsperfmon /coverage /output:<path_to_coverage_file>

This will run the monitor in "stand-alone mode". In this mode you can see the processes that have connected to and disconnected from the monitor. The monitor traps Control-C, so to shutdown the monitor you will need to use another tool called VSPerfCmd ("Visual Studio Performance Command"). This tool communicates with the running monitor process to perform a variety of tasks. You would execute the following command line to shutdown the monitor:

vsperfcmd /shutdown

The command tool can also be used to start the monitor in the background. To do so, execute the following command line:

vsperfcmd /start:coverage /output:<path_to_coverage_file>

This command will cause the monitor to be spawned as a background process. You will use the same command as the one given above to shutdown the monitor.

There are a few important things to note about the monitor that can prevent successful coverage collection:

By default, the monitor only accepts connections from processes running under an identity that is a local administrator. The most common problem experienced with this is that no data is collected for ASP.NET web sites. This is due to ASP.NET web sites running under the Network Service identity by default, which is a restrictive identity meant for services that accept network connections. To solve this, there is a /USER switch on VSPerfMon/VSPerfCmd to allow the specified user access to the monitor. When you perform code coverage on a web site inside Visual Studio, a request is made by Visual Studio to discover what the identity of the worker process is. This identity is then passed to the monitor to allow the connection. This happens automatically before test execution when there is a web site checked in the code coverage section of the test run configuration.
By default, the monitor creates the objects and communication channels needed by the coverage runtime in a local session scope. Therefore, if you are running the monitor in a different session from the process that will connect to the monitor (e.g. if you are not logged on from the console session and you want a service to connect to the monitor), then you need to specify the /CrossSession switch to VsPerfMon/VsPerfCmd. This will instruct the monitor to create those objects in a global scope so that they are accessible from other sessions.

Let's recap what we have so far:

Instrumentation is performed using vsinstr.exe for native, mixed mode, and managed images. To instrument an image, execute the following: "vsinstr.exe /coverage <path_to_image>".
To start the monitor process that will collect the coverage data, execute the following: "vsperfcmd /start:coverage <path_to_coverage_file>".
Run the test(s) you want to collect coverage data for using the instrumented images.
To stop collection, execute the following: "vsperfcmd /shutdown".

We now have a coverage file which we can use to view coverage results. The coverage file can be imported into the "Code Coverage Results" tool window inside of Visual Studio. This will show you per-method statistics and double clicking on a method will result in opening the source file in the editor with coverage coloring turned on. I will present a walk-through of how to accomplish everything I've been talking about so far inside of Visual Studio in a future blog entry.

I will discuss how to use the coverage analysis API to read the coverage file programmatically in my next blog entry.

Code Coverage Instrumentation

2007-05-04T03:02:00Z

For my first on-topic blog post, I would like to give an overview of how code coverage instrumentation works in Visual Studio (it seems like a good a place to start as any). For basic information about what code coverage is, check out wikipedia's Code Coverage topic. Code coverage is a feature available in Visual Studio Team Edition for Developers, Visual Studio Team Edition for Testers, and Visual Studio Team Suite.

One of the most common questions I get by people that know about code coverage is what kind of code coverage does Visual Studio support. Visual Studio uses a block-based statement (also known as C1 coverage) and condition coverage methodology. A block is commonly defined as a sequence of instructions (in this case x86 or CIL instructions) that have a single entry point and a single exit point. We consider an exit point to be a branch instruction, a function call, a return instruction, or, in the case of managed code, a throw instruction.

I think it'll be useful to to relate this to source code, so take this (rather silly) example in C++:

   1:  int foo(bool condition)

   2:  {

   3:      int i = 0;         /* block 0 */

   4:      if (condition)     /* block 0 */

   5:      {

   6:          i = 5;         /* block 1 */

   7:      }

   8:      else

   9:      {

  10:          i = bar();     /* block 2 and 3 */

  11:      }

  12:      return i;          /* block 4 */

  13:  }

Here is the generated debug x86 of the code above (I've replaced the real addresses with easier to read "pseudo-addresses" that erroneously assumes x86 is a one byte fixed-length instruction set):

Address:	Instruction:	Operands:	Block:
0000	push	ebp	0
0001	mov	ebp,esp	0
0002	push	ecx	0
0003	mov	dword ptr [i],0	0
0004	movzx	eax,byte ptr [condition]	0
0005	test	eax,eax	0
0006	je	0009h	0
0007	mov	dword ptr [i],5	1 (due to branch)
0008	jmp	000Bh	1
0009	call	bar	2 (due to branch)
000A	mov	dword ptr [i],eax	3 (due to call)
000B	mov	eax,dword ptr[i]	4 (multiple entry points)
000C	mov	esp,ebp	4
000D	pop	ebp	4
000E	ret		4

Notice that everything up to and including the first branch instruction (je) is considered to be part of the first block. Everything up to and including the next branch instruction (jmp) is part of the second block. The third block is comprised of only the call instruction for the bar function. The fourth block is comprised of storing the return value from the call to bar in the variable i. You'll notice that per the definition of a block as a single entry and single exit sequence of instructions, the instruction at 0x000B has two entry points: as the next instruction from 0x000A and also from the unconditional jump at 0x0008. Therefore, it is also considered the start of a new block and everything up to the final ret instruction is the fifth block. So as you can see, a single line of source code can actually be more than one block.

The job of instrumentation is to modify the original executable image so that we can detect when blocks are "hit" during execution. To accomplish this, a few instructions (in the case of x86: a push, two movs, and a pop) are inserted before every block to toggle a byte in a buffer that says that the block was executed. You can easily see these instructions by disassembling an instrumented image. Additional information that is required to map a block back to the original source code is also stored in the instrumented image in the form of PE sections. The instrumentation tool relies on debug information to be present to perform the instrumentation. Because the instrumentation process modifies the image, a new debug information database will be written out and referenced from the instrumented executable. By default, the instrumentation tool does an "in-place" instrumentation and overwrites the image being instrumented. The original image is backed up just in case something goes wrong. The instrumented debug information database is actually written out as a different file, so the original debug information database does not need to be backed up.

The instrumentation tool used by Visual Studio is called vsinstr.exe. This is also the same instrumentation tool used by the profiler to perform trace profiling, however the instrumentation process is quite different than the one used for code coverage. You can find this tool (assuming you're using Orcas) in "%Program Files%\Microsoft Visual Studio 9.0\Team Tools\Performance Tools", which is not actually on the default path for a Visual Studio command prompt. In Orcas, vsinstr.exe supports the instrumentation of native x86, mixed mode x86, and managed images. Unfortunately, 64-bit support did not make it into Orcas, but it is something we would like to get in for a future release. To use vsinstr.exe, simply give it the /coverage switch and the path to the image to instrument:

vsinstr.exe /coverage foo.dll

This will generate foo.dll (instrumented), foo.instr.pdb (the instrumented pdb referenced by the instrumented foo.dll), and foo.dll.orig (the backed-up original foo.dll). I will cover exactly how to automatically accomplish the same thing inside of Visual Studio using a screenshot walkthrough in an upcoming blog entry.

Once an image is instrumented, it is ready for coverage collection, which will be the focus of my next post.

Welcome to my blog!

2007-05-02T23:38:00Z

Hello, my name is Peter and I'm a recovering ex-softie. I returned a few months ago to Microsoft after a little more than two years away from the company. I graduated with a BS in Computer Science from Virginia Tech (I'm also a diehard hokie fan) in 2002. I was a three-time intern before joining Microsoft in as a developer on the Visual C++ project system team. During that time I contributed to Visual C++ 2002, 2003, and 2005.

I return as a developer on the diagnostics team that is part of Team Developer (Visual Studio Team Edition for Developers), or TeamDev as we like to call it. The diagnostics team owns the debugger, profiler, and code coverage tools within Visual Studio. I am responsible for the code coverage tools and that will be the primary focus of this blog. I will try to also talk about the great things the rest of TeamDev is doing as there are some fantastic things in the works for both Orcas and beyond.

I am by no means a prolific blogger, but I will try to get some good content sooner than later :).