VS Team System Test : Instrumentation

Using Instrumented Builds to Analyze Testing and Find Dead Code

VSTS Quality Tools — Mon, 07 Jul 2008 21:55:00 GMT

As part of my weekly duties since joining the Quality Tools team, I have been providing code coverage analysis to the team, ensuring that our automated tests cover a required percentage of our shipping code. This has been a fun exercise for me, allowing me to both gain a better understanding of what code coverage means from a quality standpoint, as well as flexing my knowledge of PivotTables and charts in Excel to provide user-friendly results. Performing basic code coverage of automated tests (such as unit tests) can be done by following the instructions at the Quality Tools blog at http://blogs.msdn.com/vstsqualitytools/archive/2005/06/08/426979.aspx. A further dive into the details of Code Coverage can be found at https://blogs.msdn.com/ms_joc/articles/406608.aspx.

For this particular exercise, rather than analyzing the coverage of automation tests, we opted for a different approach: instrument a build and collect coverage information as the tester performs ad-hoc/manual testing. The process (in a nutshell) looks like:

· Build the Application Under Test (AUT)’s binaries.

· Instrument the binaries that you wish to collect coverage on. (using vsinstr –coverage)

· Create a Setup Package containing the instrumented assemblies.

· Each Tester installs the Setup Package, and turns on data collection (using vsperfcmd)

· Each Tester performs a pass through the AUT.

· When vsperfcmd is shut down, a .coverage file is generated.

· Analyze the results of all of the .coverage files.

· Merge the separate .coverage files into a single result.

Now, once you’ve accomplished the work above, you’ve got yourself two important artifacts:

· One coverage file for each tester’s work.

· One coverage file that merges all tester’s work.

The two of these can used to be answer some interesting questions about both the testing process, and the application under test.

· What classes/methods were never covered?

o How can you tell? Look for classes/methods with 0% coverage in the merged result.

o What does this mean? This generally indicates either scenarios that aren’t being tested, or “dead code” that, while covered by the automated tests that were created when it was checked in, is no longer used by the application.

o What else can I tell from this? What kind of classes are getting 0% coverage? If there are lots of exception classes that are uncovered, this may indicate that testers aren’t evaluating error conditions (ex: opening a file from the application with invalid data). You may also find that there was an application state (such as docked, minimized, or placed in the system tray) that is required to test certain code paths.

· Is each tester testing the same things?

o How can you tell? Look for classes where the coverage for each tester is reasonably similar to each other and the overall merged result.

o What does this mean? If the merged result is higher than the top individual tester coverage, this indicates that each tester exercised different code. On the other hand, a class where all testers cover ~70%, and the merged result is also ~70%, this indicates that all testers were covering the same scenarios and code paths.

Performing this testing on our own group has already helped us to identify over 700 blocks of dead code, several scenarios that weren’t fully being tested, and some insights on the similarities and differences between each tester’s approach. While these insights were helpful, making this a regular part of our process going forward will help provide continuous improvement in our testing process.

Running Tests with Code Coverage

VSTS Quality Tools — Wed, 08 Jun 2005 14:55:00 GMT

As I talk to customers I find that there are many different work flow models. I use the term “work flow” to describe all the things you do to get complete your objectives. That work flow can be documented and shared with others at the office. Having a positive and efficient work flow is important, otherwise we get cranky.

Meeting the needs of the different work flows can be difficult. In this post, I’ve outlined three common work flow situations with steps to successfully collect code coverage data. These scenarios were built around different models of access to source code because that seems to be the underlying issue customers experience.

Every scenario minimally requires product binaries, product PDBs, test binaries, and Visual Studio Team System. For better, richer, and more flexible experiences you’ll need more, like access to test and/or product source code. In the end, you just want code coverage data so it doesn’t matter how you go about it. Pick a method that works for your setup and enjoy what VS Team System has to offer. J

For more information about how you can go “off-roading” with code coverage, I’d recommend looking at John Cunningham’s blog.

Scenario 1: Local Product and Test Sources

This is the most ideal scenario. One has the product and test projects loaded in VS. One has full ability to build both which provides the most flexible experience.

This is probably a developer or a tester who work directly on the product. They are trusted to and it is logistically possible to have full access.

Gathering Steps

Load the product and test sources in a single solution in Visual Studio Team System 2005.
Perform a full build.
Open the Run Configuration you intend to use for this run (‘localtestrun.tesrunconfig’ is the default)
Select the Code coverage page in the resulting Dialog
Check the Binaries you wish to be instrumented
Execute the tests you desire to run, either through Test Manager or Test View.

Review Code Coverage data

When tests are finished, open the Code Coverage window and review the results.

Scenario 2: Product Binaries and Test Sources

This is probably the most likely scenario. One has product binaries (and PDB files!) from a daily build. One has access to the test sources and can build test binaries. This nearly provides the same flexibility as Scenario 1.

In this model, the user may be the same developer or tester as in Scenario 1, but they may be encouraged to only test against “official” builds from a common source. This could also be a person testing offsite from core development who receives regular drops of builds from the development team. They may be interested in gauging the comprehensiveness of their testing as measured by code coverage, or this activity may have been requested by the development team. In the latter case, the code coverage results file will be sent to the development team for analysis.

Gathering Steps

Load the test sources in Visual Studio Team System 2005.
Copy the desired product binaries and PDB files locally.
Build the test sources.
Open the Run Configuration you intend to use for this run (‘localtestrun.tesrunconfig’ is the default)
Select the Code coverage page in the resulting Dialog
Check the Binaries you wish to be instrumented
Execute the tests you desire to run, either through Test Manager or Test View.

Review Code Coverage data

When tests are finished, open the Code Coverage window and review the results. If you don’t have access the source code, you will only be able to see the blocks covered etc., rather than full detail and coloring of the source code.

Scenario 3: Product and Test Binaries Only

This scenario requires the most work and provides the least amount of benefit, but unfortunately it is all too common. All the great work VS does for you when you have your own test project is gone because you only have test binaries now. Additionally, this could include running manual tests so there might be no automation to run at all.

That said the scenario still works. You can get code coverage data. Also, if that data is shared with someone who has access to source code, then they will get full benefit from the results as they can see code coverage detail and line coloring.

Gathering Steps

Copy the desired product binaries, product PDBs, and test binaries files locally.
Instrument the product binaries:
1. Open up a Visual Studio 2005 Command Prompt (Start | All Programs | Microsoft Visual Studio 2005 | Visual Studio tools | Visual Studio Command Prompt).
2. Change to the directory where your binaries are located
3. Execute “vsinstr –coverage <assemblyname>” for all your binaries. e.g. If your binary name is ‘coolproduct.dll’, then type “vsinstr –coverage coolproduct.dll”.
Start the Code Coverage monitoring process
1. Return to the previous command prompt
2. Type “vsperfcmd –start:coverage –output:outputFilename.coverage”.
Execute your tests
1. For automated tests, navigate to the directory where your test binaries are located and execute “mstest /testcontainer:TestDll.dll”.
2. Run manual tests, if you have them
End the Code Coverage monitoring process
1. In the command prompt opened earlier, type “vsperfcmd –shutdown”. This shuts down the coverage monitor and saves the coverage data to disk.

Note: There are alternatives to manually executing these commands, but they are slightly more complex.

Start VS and create a test project
Add your production DLLs as references to the project
Open the Run Configuration you intend to use for this run (‘localtestrun.tesrunconfig’ is the default)
Select the Code coverage page in the resulting Dialog
Check the Binaries you wish to be instrumented
Save the runconfig file
Exit the IDE
Copy the Product DLLs and Test DLLs into to the correct place along with the runconfig
Execute “mstest /testcontainer:testdll.dll /runconfig:runconfig.testrunconfig”

This will instrument and execute the tests, while correctly starting & shutting down the coverage collector. If you still wish to instrument the binaries yourself, then you still need a dummy-dll to be instrumented. The tools do not start the coverage monitor unless the tool itself has instrumented a DLL.

Review Code Coverage data

Open the .coverage file in Visual Studio.