Customizing Code Coverage in Visual Studio 11
The Visual Studio 11 Code Coverage tool by default analyzes all solution binaries (exe / dll) that get loaded during unit tests. We suggest you retain this default; this works most of the times.
However, there may be cases where you want to customize this behavior. One common case is to include / exclude specific binaries from code coverage. When this is done, there is a possibility you might lose track of new assemblies introduced into the project, and not measure code coverage for them. Still, if it needs to be done, it can be achieved by customizing code coverage settings. The other common case is to specify symbol search path for binaries that don’t have symbols installed locally.
This blog talks about customizing code coverage for scenarios like these.
1. The .runsettings file
Advanced VS 11 code coverage settings are specified in a .runsettings file. This is the configuration file used by VS 11 unit testing tools.
Some of you, who have used VS 2010, may be familiar with .testsettings file. You may be wondering why the same file was not used for VS 11. The reason for this was that the .testsettings file applies only to the MSTest based unit tests of VS 2010. The VS 11 unit testing tools not only apply for MSTest, but also 3rd party unit testing adapters such as NUnit and xUnit.net. Using a .testsettings file will not work with these. Hence a different settings file, the .runsettings, was used.
To customize code coverage, you will need to add .runsettings file to your solution. There isn’t a built-in template for this right now. You would need to add it as an xml file and edit its contents:
1. Right click Solution -> Add New Item -> XML File -> Save file as CodeCoverage.runsettings (the extension of .runsettings is important).
2. Copy the sample runsettings content given later in this blog into your file.
3. Edit it to customize code coverage, as explained in next section.
4. Set it as the default setting file to be used via Test -> Test Settings -> Select Test Settings File
After this, the specified settings should be used whenever you analyze code coverage.
2. Common customization scenarios
2.1. Customizing the set of binaries analyzed for code coverage
Here, we want include / exclude specific binaries from code coverage. Using the runsettings file, there are two approaches to do this:
Customize by exclusion
Here we include everything, and exclude what we don’t need – an opt-out approach. This may be useful if the code of interest changes often and new binaries get introduced frequently: you would like these new binaries to be automatically included for code coverage when they are loaded. There are some known binaries that you don’t want coverage, like some test automation code. These are excluded.
Here, leave the includes as empty to include all assemblies. Specify the assemblies to be excluded thru by their name / path.
<ModulePaths>
<Include>
<!-- Do not specify any includes. This will attempt to include all binaries -->
</Include>
<Exclude>
<!-- Exclude modules that aren't to be processed, by their name / path -->
<ModulePath>.*MyCompany.MyTests1.dll</ModulePath>
<ModulePath>.*CPPUnitTestFramework.*</ModulePath>
</Exclude>
</ModulePaths>
Customize by inclusion
Here we include exactly what we need, and exclude everything else – an opt-in approach. This may be useful if the code of interest is present in well-known binaries; you just want these binaries to be covered.
Here explicitly include the binaries of interest. Leave the excludes as empty to exclude everything.
<ModulePaths>
<Include>
<!-- Include modules of interest, by their name / path -->
<ModulePath>.*MyCompany.MyComponent.dll</ModulePath>
</Include>
<Exclude>
<!—- Do not specify any excludes. Anything not included will get excluded -->
</Exclude>
</ModulePaths>
Note that while both inclusions and exclusions can be used together, it can make things very confusing. So it is preferable to use one mechanism to customize.
2.2. Specifying symbol search paths
Code coverage requires symbols (PDB) for binaries to be instrumented. For binaries built by your solution, symbols are generally present alongside the binary and code coverage automatically works. In some cases, you may be referencing binaries external to your solution, but still need code coverage for these external binaries. In such cases, you can specify the symbol search path to look for symbols as part of the runsettings file.
<SymbolSearchPaths>
<Path>>\\mybuildshare\builds\ProjectX</Path>
</SymbolSearchPaths>
A set of local and remote paths can be specified here. Note however that symbol resolution can take time, especially when using a remote build share with a lot of binaries. Best would be to avoid this by copying symbols alongside the binary, and not specify any search paths.
3. Sample .runsettings file
Below is a sample runsettings file. When creating a new runsettings, start with this, and customize it as needed.
<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
<DataCollectionRunSettings>
<DataCollectors>
<DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
<Configuration>
<CodeCoverage>
<ModulePaths>
<!--
About include/exclude lists:
Empty "Include" clauses imply all; empty "Exclude" clauses imply none.
Each element in the list is a regular expression (ECMAScript syntax).
An item must first match at least one entry in the include list to be included.
Included items must then not match any entries in the exclude list to remain included.
It is considered an error to exclude all items from instrumentation as no data would be collected.
-->
<Include>
<ModulePath>.*\\UnitTestProject1\.dll</ModulePath>
</Include>
<Exclude>
<ModulePath>.*CPPUnitTestFramework.*</ModulePath>
</Exclude>
</ModulePaths>
<UseVerifiableInstrumentation>True</UseVerifiableInstrumentation>
<AllowLowIntegrityProcesses>True</AllowLowIntegrityProcesses>
<CollectFromChildProcesses>True</CollectFromChildProcesses>
<CollectAspDotNet>False</CollectAspDotNet>
<!--
Additional paths to search for symbol files. Symbols must be found for modules to be instrumented.
If symbols are alongside the binaries, they are automatically picked up. Otherwise specify the here.
Note that searching for symbols increases code coverage runtime. So keep this small and local.
<SymbolSearchPaths>
<Path>C:\Users\User\Documents\Visual Studio 11\Projects\ProjectX\bin\Debug</Path>
<Path>\\mybuildshare\builds\ProjectX</Path>
</SymbolSearchPaths>
-->
<Functions>
<Exclude>
<Function>^std::.*</Function>
<Function>^ATL::.*</Function>
<Function>.*::__GetTestMethodInfo.*</Function>
<Function>^Microsoft::VisualStudio::CppCodeCoverageFramework::.*</Function>
<Function>^Microsoft::VisualStudio::CppUnitTestFramework::.*</Function>
<Function>.*::YOU_CAN_ONLY_DESIGNATE_ONE_.*</Function>
</Exclude>
</Functions>
<Attributes>
<Exclude>
<Attribute>^System.Diagnostics.DebuggerHiddenAttribute$</Attribute>
<Attribute>^System.Diagnostics.DebuggerNonUserCodeAttribute$</Attribute>
<Attribute>^System.Runtime.CompilerServices.CompilerGeneratedAttribute$</Attribute>
<Attribute>^System.CodeDom.Compiler.GeneratedCodeAttribute$</Attribute>
<Attribute>^System.Diagnostics.CodeAnalysis.ExcludeFromCodeCoverageAttribute$</Attribute>
</Exclude>
</Attributes>
<Sources>
<Exclude>
<Source>.*\\atlmfc\\.*</Source>
<Source>.*\\vctools\\.*</Source>
<Source>.*\\public\\sdk\\.*</Source>
<Source>.*\\microsoft sdks\\.*</Source>
<Source>.*\\vc\\include\\.*</Source>
</Exclude>
</Sources>
<CompanyNames>
<Exclude>
<CompanyName>.*microsoft.*</CompanyName>
</Exclude>
</CompanyNames>
<PublicKeyTokens>
<Exclude>
<PublicKeyToken>^B77A5C561934E089$</PublicKeyToken>
<PublicKeyToken>^B03F5F7F11D50A3A$</PublicKeyToken>
<PublicKeyToken>^31BF3856AD364E35$</PublicKeyToken>
<PublicKeyToken>^89845DCD8080CC91$</PublicKeyToken>
<PublicKeyToken>^71E9BCE111E9429C$</PublicKeyToken>
<PublicKeyToken>^8F50407C4E9E73B6$</PublicKeyToken>
<PublicKeyToken>^E361AF139669C375$</PublicKeyToken>
</Exclude>
</PublicKeyTokens>
</CodeCoverage>
</Configuration>
</DataCollector>
</DataCollectors>
</DataCollectionRunSettings>
</RunSettings>
4. Specifying runsettings file while running unit tests
Unit tests can be run in different contexts. The below specifies how runsettings are specified in each case.
4.1. In Visual Studio IDE
Select setting file to be used via Test -> Test Settings -> Select Test Settings File. Then invoke ‘Analyze Code Coverage’.
4.2. In command line
The tool vstest.console.exe is the way to run unit tests from command line. It is the successor of MSTest – and can run not only the MSTest based tests, but also tests of third party unit adapters (like NUnit, xUnit.net, etc).
To run tests with the runsettings file here:
1. Launch the Visual Studio Native Tools command prompt.
2. Run:
> vstest.console.exe TestAssembly.dll /EnableCodeCoverage /settings:CodeCoverage.runsettings
4.3. In Build
Here, there is a TFS Build Definition, as part of which unit tests are run and overall code coverage is measured. If there is a .runsettings file customizing code coverage, the same runsettings file can be used in the Build as well.
Below are the steps to do it.
1. Edit the build definition. Go to Process tab, and edit test run details.
2. Make sure the test runner is ‘Visual Studio Test Runner’. Set Options to ‘Custom’. Hit Browse to choose the settings file.
3. Pick the settings file and save the definition.
