vikramagrawal

Writing loggers for command line test runner vstest.console.exe

Vikram Agrawal [MSFT] — Thu, 26 Jul 2012 08:23:00 GMT

Command line test runner for VS 2012 comes with a parameter /logger where user can specify logger to be used. Two loggers shipped with VS are Console and Trx. Console logger is default logger that prints output of test execution to console window. Trx logger is can be used to generate trx file for test run (/logger:trx).

Here I am describing how to write a custom logger that can be used with vstest.console.exe

There are two requirements for custom test logger

1. Implement interface Microsoft.VisualStudio.TestPlatform.ObjectModel.Client.ITestLogger

Interface can be found in “C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\Microsoft.VisualStudio.TestPlatform.ObjectModel.dll”

2. Assembly containing logger implementation to be present in place where extensions are searched like "C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\Extensions"

Assembly can be installed with VSIX like test adapters. In that case use /UseVSIXExtensions parameter also.

ITestLogger has only one method

      /// <summary>
    
    /// Initializes the Test Logger.
    
    /// </summary>
    
    /// <param name="events">Events that can be registered for.</param>
    
    /// <param name="testRunDirectory">Test Run Directory</param>
    
    void Initialize(TestLoggerEvents events, string testRunDirectory);

This function is used to hookup to following events in TestLoggerEvents

    /// <summary>
    
    /// Raised when a test message is received.
    
    /// </summary>
    
    public abstract event EventHandler<TestRunMessageEventArgs> TestRunMessage;
    
    /// <summary>
    
    /// Raised when a test result is received.
    
    /// </summary>
    
    public abstract event EventHandler<TestResultEventArgs> TestResult;
    
    /// <summary>
    
    /// Raised when a test run is complete.
    
    /// </summary>
    
    public abstract event EventHandler<TestRunCompleteEventArgs> TestRunComplete;

For Ex:

    /// <summary>
    
    /// Initializes the Test Logger.
    
    /// </summary>
    
    /// <param name="events">Events that can be registered for.</param>
    
    /// <param name="testRunDirectory">Test Run Directory</param>
    
    public void Initialize(TestLoggerEvents events, string testRunDirectory)
    
    {
    
        // Register for the events.
    
        events.TestRunMessage += TestMessageHandler;
    
        events.TestResult += TestResultHandler;
    
        events.TestRunComplete += TestRunCompleteHandler;
    
    }

Each Event handler now can be used to log event however you want (in file, db, console, etc). Following example prints in on console:

    /// <summary>
    
    /// Logger for sending output to the console.
    
    /// </summary>
    
    [ExtensionUri("logger://SimpleConsoleLogger/v1")] /// Uri used to uniquely identify the console logger. 
    
    [FriendlyName("SimpleLogger")] /// Alternate user friendly string to uniquely identify the logger.
    
    internal class SimpleConsoleLogger : ITestLogger
    
    {
    
        /// <summary>
    
        /// Initializes the Test Logger.
    
        /// </summary>
    
        /// <param name="events">Events that can be registered for.</param>
    
        /// <param name="testRunDirectory">Test Run Directory</param>
    
        public void Initialize(TestLoggerEvents events, string testRunDirectory)
    
        {
    
            // Register for the events.
    
            events.TestRunMessage += TestMessageHandler;
    
            events.TestResult += TestResultHandler;
    
            events.TestRunComplete += TestRunCompleteHandler;
    
        }
    
        /// <summary>
    
        /// Called when a test message is received.
    
        /// </summary>
    
        private void TestMessageHandler(object sender, TestRunMessageEventArgs e)
    
        {
    
            switch (e.Level)
    
            {
    
                case TestMessageLevel.Informational:
    
                    Console.WriteLine("Information: " + e.Message);
    
                    break;
    
                case TestMessageLevel.Warning:
    
                    Console.WriteLine("Warning: " + e.Message);
    
                    break;
    
                case TestMessageLevel.Error:
    
                    Console.WriteLine("Error: " + e.Message);
    
                    break;
    
                default:
    
                    break;
    
            }
    
        }
    
        /// <summary>
    
        /// Called when a test result is received.
    
        /// </summary>
    
        private void TestResultHandler(object sender, TestResultEventArgs e)
    
        {
    
            string name = !string.IsNullOrEmpty(e.Result.DisplayName) ? e.Result.DisplayName : e.Result.TestCase.FullyQualifiedName;
    
            if (e.Result.Outcome == TestOutcome.Skipped)
    
            {
    
                Console.WriteLine(name + " Skipped");
    
            }
    
            else if (e.Result.Outcome == TestOutcome.Failed)
    
            {    
    
                Console.WriteLine(name + " Failed");
    
                if(!String.IsNullOrEmpty(e.Result.ErrorStackTrace))
    
                {
    
                    Console.WriteLine(e.Result.ErrorStackTrace);                    
    
                }
    
            }
    
            else if (e.Result.Outcome == TestOutcome.Passed)
    
            {
    
                Console.WriteLine(name + " Passed");
    
            }
    
        }
    
        /// <summary>
    
        /// Called when a test run is completed.
    
        /// </summary>
    
        private void TestRunCompleteHandler(object sender, TestRunCompleteEventArgs e)
    
        {
    
            Console.WriteLine("Total Executed: {0}", e.TestRunStatistics.ExecutedTests);
    
            Console.WriteLine("Total Passed: {0}", e.TestRunStatistics[TestOutcome.Passed]);
    
            Console.WriteLine("Total Failed: {0}", e.TestRunStatistics[TestOutcome.Failed]);
    
            Console.WriteLine("Total Skipped: {0}", e.TestRunStatistics[TestOutcome.Skipped]);
    
        }
    
    }

Once binary is placed at proper location, above logger can be used as

vstest.console.exe <testdll> /logger:SimpleLogger

If you've any feedback or issue then please do send us on Connect or in the Forums.

Running selective unit tests in VS 2012 RC using TestCaseFilter

Vikram Agrawal [MSFT] — Mon, 23 Jul 2012 13:17:00 GMT

New unit testing platform in Visual Studio 2012 provides a new way to selectively execute test based on filtering condition through TestCaseFilter. TestCaseFilter can as specified as a string while executing tests through command line (vstest.console.exe), Team Build (when running test using “Visual Studio Test Runner”) or through Test platform client API.

Support for adapters to leverage this feature is provided in test platform. MSTest adapter shipped with Visual Studio 2012 for executing managed test provide basic filtering as described below (more enhanced filtering expression support to be added in future versions).

Specifying TestCaseFilter:

Command Line
New command line runner for executing tests in VS 2012 (vstest.console.exe) has an optional command line parameter /TestCaseFilter for specifying filtering expression. /TestCaseFilter cannot be specified with /Tests argument.
E.g. vstest.console.exe test.dll /TestCaseFilter:”TestCategory=Nightly”

Team Build
When adding a test activity in build definition, if user selects to execute test using “Visual Studio Test Runner” then user get to specify test case filter (optional can be left blank to run all tests).

Client API
When using Test Platform Client API for executing test then filtering expression can be specified in TestRunCriteria.TestCaseFilter property.
E.g.
TestRunCriteria runCriteria = CreateTestRunCriteria(); runCriteria.TestCaseFilter = “TestCategory=Nightly”;

Syntax for filtering expression:

Adapters can choose their own syntax for filtering or take advantage of support provided by Test Platform by MSTest adapter syntax. Following is syntax for MSTest adapter for managed test execution. Invalid filtering expressions will be ignored and all tests will be executed.

Operators supported in RC are:

= (equals)
!= (not equals)
~ (contains or substring only for string values)
& (and)
| (or)
( ) (paranthesis for grouping)

Expresssion can be created using these operators as any valid logical condition. & (and) has higher precedence over | (or) while evaluating expression.

E.g.

"TestCategory=NAR|Priority=1"
"Owner=vikram&TestCategory!=UI"
"FullyQualifiedName~NameSpace.Class"
"(TestCategory!=UI&(Priority=1|Priority=2))|(TestCategory=UI&Priority=1)"

"Priority~1" // Invalid as priority is int not string

Properties supported by MSTest adapter for filtering are

Name=<TestMethodDisplayNameName>

FullyQualifiedName=<FullyQualifiedTestMethodName>

Priority=<PriorityAttributeValue>

TestCategory=<TestCategoryAttributeValue>

ClassName=<ClassName> (Valid only for unit tests for Windows store apps, currently not available for classic MSTest)

Using TestCaseFilter in Test Adapters

Adapters can leverage default format of TestCaseFilter that MS Test uses. Parsing and matching is done by ITestCaseFilterExpression provided by test platform.

Following method from IRunContext is for getting an implementation of ITestCaseFilterExpression.
ITestCaseFilterExpression GetTestCaseFilter(IEnumerable<String> supportedProperties, Func<string, TestProperty> propertyProvider);
ITestCaseFilterExpression provides following method to match test case (only if MSTest format is used for TestCaseFilter)
bool MatchTestCase(TestCase testCase, Func<string, object> propertyValueProvider);

E.g.

       1: /// <summary> 

       2: /// Supported properties for filtering 

       3: /// </summary> 

       4: private static Dictionary<string, TestProperty> supportedPropertiesCache;

       5:  

       6: supportedPropertiesCache = new Dictionary<string, TestProperty>(StringComparer.OrdinalIgnoreCase); 

       7: supportedPropertiesCache[“Priority”] = PriorityProperty; 

       8: supportedPropertiesCache[“TestCategory”] = TestCategoryProperty; 

       9: supportedPropertiesCache[“FullyQualifiedName”] = TestCaseProperties.FullyQualifiedName; 

supportedPropertiesCache[“Name”] = TestCaseProperties.DisplayName; 

supportedPropertiesCache[“MyProperty”] = MyProperty;

  

ITestCaseFilterExpression filterExpression = runContext.GetTestCaseFilter(supportedPropertiesCache.Keys, (p) => PropertyProvider(p));

  

foreach (TestCase currentTest in tests) 

{ 

    // Skip test if not fitting filter criteria. 

    if (null != filterExpression && filterExpression.MatchTestCase(currentTest, (p) => PropertyValueProvider(currentTest, p)) == false) 

    { 

        continue; 

    }

  

    // Execute test

  

}

  

/// <summary> 

/// Provides value of TestProperty corresponding to property name 'propertyName' as used in filter. 

/// Return value should be a string for single valued property or array of strings for multi valued property (e.g. TestCategory) 

/// </summary> 

private static object PropertyValueProvider(TestCase currentTest, string propertyName) 

{ 

    TestProperty testProperty; 

    if (supportedPropertiesCache.TryGetValue(propertyName, out testProperty)) 

    { 

        // Test case might not have defined this property. In that case GetPropertyValue() 

        // would return default value. For filtering, if property is not defined return null. 

        if (currentTest.Properties.Contains(testProperty)) 

        { 

            return currentTest.GetPropertyValue(testProperty); 

        } 

    } 

    return null; 

}

  

/// <summary> 

/// Provides TestProperty for property name 'propertyName' as used in filter. 

/// </summary> 

private static TestProperty PropertyProvider(string propertyName) 

{ 

    TestProperty testProperty = null; 

    supportedPropertiesCache.TryGetValue(propertyName, out testProperty); 

    return testProperty; 

}

If you've any feedback or issue then please do send us on Connect or in the Forums.

Running Unit Tests for Windows Metro style apps from Command Line

Vikram Agrawal [MSFT] — Sun, 06 May 2012 16:36:00 GMT

Visual Studio 11 Beta was announced sometime back along with Win 8 consumer preview in which Unit testing has been redefined and support for testing Windows Metro style apps is added.

Tests for Metro style apps can be run from Visual Studio, Command Line and Team Build. Steps to create and run unit test from Visual Studio and Team Build are described in Walkthrough: Creating and Running Unit Tests for Metro style Applications and Running Windows 8 Metro style tests in TeamBuild respectively.

This post explains how to run Unit Test for Metro style apps from Command Line tool (vstest.console.exe)

Installing Unit Testing Command Line tool

Unit testing command line tool get installed with Visual Studio 11 Beta. Command Line test runner, vstest.console.exe, is installed at $(VSInstallDir)/Common7/IDE/CommonExtensions/Microsoft/TestWindow. It's also get installed with Remote Debugger SKU.

Creating App Package for Unit Test Library

Create a Unit Test Library Project for Managed (C#/VB) or Native (C++) in Visual Studio. Steps for creating project are same as in Walkthrough: Creating and Running Unit Tests for Metro style Applications.

After tests are authored and ready to run from command line create an app package (.appx file) as follows:

Right click on the project in Solution Explorer.
Go to Store -> Create App Package...
Select "Create a package to use locally only" and click Next
Set desired Package location and Build configuration
Click Create
Click OK

Inside a folder in "Package location" specified in above steps following files can be found:

<filename>.appx -> App package file for test project. It contains test binary and all Unit Testing platform binaries.
<filename>.cer -> Certificate file corresponding to .pfx file used to sign package. It needs to be added to trusted certificates to install package.
<filename>.appxsym -> Package containing public symbols of app package. (Not required for our scenario)
<filename>.bat -> Batch file created with package to install package. (Not required for our scenario)

If you want to execute test on some other system (like ARM machine with Remote Debugger SKU) then copy .appx and .cer files on to that machine.

Executing Unit Tests for Metro style apps from command line

Install Dev License:

Install developer license on the box where test needs to be executed. This is needed to run unit tests for Metro style apps. This is onetime task per machine (may require renewing license, if expired).

Installing the certificate:

On machine where want to run test from command line, install certificate. This is a one time operation per certificate and can be done as follows:

a. From an elevated command prompt execute command “Certutil -addstore root <filename>.cer”

Executing tests:

Test for Metro style app can be executed from command line as follows:

"<TestToolPath>\vstest.console.exe" "<AppxPath>\<filename>.appx" /InIsolation

<TestToolPath> is location where vstest.console.exe is installed (VS installs it at $(VSInstallDir)/Common7/IDE/CommonExtensions/Microsoft/TestWindow).

<AppxPath> is where App Package for test is present.

/InIsolation is required as tests for Metro style apps can't be run in inproc mode. If option is not given then it will generate a warning.

Command line test runner (vstest.console.exe) will install app package, execute tests and uninstall package.

More options:

/Logger:trx option can be used to generate trx file which can be opened in VS to analyze failures.

/TestCaseFilter can be used to selectively run tests. More info can be found at TestCaseFilter in VS11 Unit Testing

/Tests & /ListTests options are not valid for Metro style app

Common Errors and Resolutions:

a. Don't give dll file directly for test execution. Directly giving dll file will not execute test in appcontainer mode (in which Metro style app executes). Create app package as described above and use it for test execution in appcontainer mode.

b. Test project has its own manifest file and it is not derived from references. Set proper capabilities required to run tests in manifest file of test project.

c. To execute test as 64 bits on a 64 bits machine, use /platform:x64 option. By default, tests are executed as 32 bits on 64s bit machine.

d. If you are taking dependency on some SDK (like C++ test project takes dependency on VSLib SDK) then required dependency packages are copied in "Dependencies" folder with app package by package builder. If copying app package to another location or machine then copy "Dependencies" folder along with package.

If you've any feedback or issue then please do send us on Connect or in the Forums.

TestCaseFilter in VS11 Unit Testing (Beta)

Vikram Agrawal [MSFT] — Fri, 27 Apr 2012 05:27:00 GMT

(For VS2012 RC new filter operators are described at Running selective unit tests in VS 2012 RC using TestCaseFilter)

New unit testing platform in Visual Studio 11.0 Beta (http://blogs.msdn.com/b/visualstudioalm/archive/2012/03/08/what-s-new-in-visual-studio-11-beta-unit-testing.aspx) provides a new way to selectively execute test based on filtering condition through TestCaseFilter. TestCaseFilter can as specified as a string while executing tests through command line (vstest.console.exe), Team Build (when running test using “Visual Studio Test Runner”) or through Test platform client API.

Support for adapters to leverage this feature is provided in test platform. MSTest adapter shipped with Visual Studio 11.0 Beta for executing managed test provide basic filtering in Beta version described below (more enhanced filtering expression support to be added in future versions).

Specifying TestCaseFilter:

Command Line

New command line runner for executing tests in VS 11.0 (vstest.console.exe) has an optional command line parameter /TestCaseFilter for specifying filtering expression. /TestCaseFilter cannot be specified with /Tests argument.
E.g. vstest.console.exe test.dll /TestCaseFilter:”TestCategory=Nightly”
Team Build

When adding a test activity in build definition, if user selects to execute test using “Visual Studio Test Runner” then user get to specify test case filter (optional can be left blank to run all tests).
Client API

When using Test Platform Client API for executing test then filtering expression can be specified in TestRunCriteria.TestCaseFilter property.
E.g.
TestRunCriteria runCriteria = CreateTestRunCriteria(); runCriteria.TestCaseFilter = “TestCategory=Nightly”;

Syntax for filtering expression:

Adapters can choose their own syntax for filtering or take advantage of support provided by Test Platform by MSTest adapter syntax. Following is syntax for MSTest adapter for managed test execution. Invalid filtering expressions will be ignored and all tests will be executed.

Two operators supported in Beta are

= (equals): For specifying <property>=<value>. A test is selected for execution if specified property for test has given value.

E.g. TestCaseFilter=”Priority=1”
[TestMethod, Priority(1)] // Executed public void TestMethod1(){} [TestMethod, Priority(2)] // Not Executed as value not matching public void TestMethod2(){}
1. Expression having invalid property will be ignored.
  E.g. for MSTest filter expression “abcd=xyz” will be ignored as “abcd” is not a valid property.
2. Absence of value for property is treated as false.
  E.g. TestCaseFilter=”Priority=1”
  [TestMethod] // Not executed as no priority public void TestMethod3(){}
3. For multi valued property like TestCategory, it will check in value in present in all values of property.
  E.g. TestCaseFilter=”TestCategory=Nightly”
  [TestMethod, TestCategory(“TeamBuild”), TestCategory(“Nightly”)] // Execute as one of multiple values matches public void TestMethod4(){}
4. Leading & trailing Spaces for properties and values in filter expressions are ignored.
  E.g. TestCaseFilter=” Prioity = 1 ”
  [TestMethod, Priority(1)] // Executed as leading and trailing spaces ignored. public void TestMethod5(){}
5. Properties and values are case insensitive.
  E.g. TestCaseFilter=”prioity=1”
  [TestMethod, Priority(1)] // Executed as its case insensitive public void TestMethod5(){}
|| (OR): Logical OR for specifying multiple property value pair. If one of the expression joined by || evaluates to true then test is executed else filtered out.
E.g. TestCaseFilter=”TestCategory=Nightly||Priority=1”>
[TestMethod, Priority(1)] // Executed public void TestMethod6(){} [TestMethod, TestCategory(“Nightly”)] // Executed public void TestMethod7(){} [TestMethod, Priority(1), TestCategory(“TeamBuild”)] // Executed public void TestMethod8(){} [TestMethod, TestCategory(“TeamBuild”)] // Not Executed public void TestMethod9(){} [TestMethod, Priority(2), TestCategory(“TeamBuild”)] // Not Executed public void TestMethod10(){}

Properties supported by MSTest adapter for filtering are

Name=<FullyQualifiedTestMethodName>
Priority=<PriorityAttributeValue>
TestCategory=<TestCategoryAttributeValue>

Note: MSTest adapter in VS11 Beta also works in legacy mode (equivalent of running tests with mstest.exe) for compatibility. In legacy mode, it can not take advantage of new VS11 features TestCaseFilter. Adapter can switch to legacy mode when .testsettings file is specified, forcelegacymode is set to true in .runsettings file or using attributes like HostType.

If you've any feedback or issue then please do send us on Connect or in the Forums.

How to write tests in WinRT Component DLL project [for VS11 Beta/Consumer Preview]

Vikram Agrawal [MSFT] — Thu, 26 Apr 2012 10:11:00 GMT

Note: In this post I just described a way to enable discovery and execution of tests in a WinRT Component DLL. This is not a recommended or supported way to write unit test for WinRT Component. Recommended way is to create a separate 'Unit Test Library' project. Hence, side effects of adding manifest file etc. in project are not analyzed. Please use this info at your own risk only if you don't have any other option.

Typically developers should test public APIs of components. Recommended way of testing "WinRT Component DLL" is to add "Unit Test Library" project from "Visual C++" -> "Windows Metro style", take reference of WinRT Componenet DLL and test public APIs.

Following steps can enable discovery and execution of tests in WinRT Component DLL and is only for cases where you don't have any option but to access non public APIs from WinRT Component DLL for testing.

Add new WinRT Component DLL project

File -> New -> Project...
Visual C++ -> Windows Metro style -> WinRT Component DLL

Add Test SDK references

Right click on project from Solution Explorer -> References ... -> Common Properties -> Framework and References
Add New Reference ... -> Windows -> Extensions
Select "Microsoft Visual Studio Test Core" and "MSTest for Native Projects"

Add Test Include Directory

Right click on project from Solution Explorer -> Properties ... -> Configuration Properties -> C/C++
Change "Additional Include Directories" to $(ProjectDir);$(IntermediateOutputPath);$(VCInstallDir)UnitTest\include;%(AdditionalIncludeDirectories)

Add Test Library Directory

Right click on project from Solution Explorer -> Properties ... -> Configuration Properties -> Linker
Change "Additional Library Directories" to $(VCInstallDir)UnitTest\lib;%(AdditionalLibraryDirectories)

Add required Property in vcxproj

Right click on project from Solution Explorer -> Unload Project
Right click on project from Solution Explorer -> Edit '...vcxproj'
Add following property to vcxproj file: <PropertyGroup><AppxPackage>true</AppxPackage></PropertyGroup>
Close vcxproj file
Right click on project from Solution Explorer -> Reload Project

Add package.appxmanifest

Add package.appxmanifest from a unit test library project to current project.

Add images

Add images folder from a unit test library project and add them to project with relative path "Images\<name>.png"

In cpp file where adding tests add
#include CppUnitTest.h"
using namespace Microsoft::VisualStudio::CppUnitTestFramework;
Add tests in cpp file\
TEST_CLASS(TestClass1)
{
    TEST_METHOD(TestMethod1)
    {
        Assert::Fail(L"Failing Test");
    }

    TEST_METHOD(TestMethod2)
    {
        Assert::AreEqual(1, 1);
    }
};
Run test

Compile
Tests should get discover
Execute tests

Effectively your WinRTComponent DLL will be a unit test library also.

Update: Marcelo has found a way to test WinRT internal class which avoids adding package.appxmanifest file & editing vcxproj. Steps are described in his blog http://marcecaro.blogspot.com/2012/04/unit-testing-private-classes-using-ccx.html

If you've any feedback or issue then please do send us on Connect or in the Forums.