As we know, the unit testing tools has been completely revamped in Visual Studio 2011. One such component is the new command line runner – vstest.console.exe

What is it ?

new command line test runner “vstest,console.exe” which would run tests written against any test framework – MSTest framework, Nunit, Xunit and so on.

Where can I find it ?

vstest.console.exe could be invoked from directly from “Visual Studio Command Prompt”. The executable would be found under “Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow” folder. Vstest.console.exe /? would give more details on the various switches provided by the test runner.

How is it different from Mstest.exe ?

With Visual Studio 2011, the tests written using visual studio unit test framework could be run using either vstest,console.exe or using mstest.exe. The behavior would be same with subtle differences (which is briefed in the below table).

vstest.console.exe has some additional options /platform /framework which would let the user to run their tests in a specific platform/framework to increase test coverage.

Mstest and Vstest:

Below is a brief summary on the various switches provided by Mstest.exe and the equivalent switches in vstest.console.exe

 

Mstest

Vstest.console

Example

/testcontainer

 

There is no switch needed in front of the input test assembly in vstest.console.exe unlike mstest.exe.

Example:

Mstest /testcontainer :abc.dll /testcontainer:xyz.dll

Vstest.console.exe abc.dll xyz.dll

/minpriority,

/maxpriority,

/category

/testcasefilter

Inv stest.console.exe /testcasefilter switch handles filtering based on all categories like priority , test category and so on.

Example:

Mstest /testcontainer:abc.dll /minpriority:1

Vstestabc.dll /testcasefilter:"Priority=1 | priority=2"

/testmetadata

/testlist

N/A

VSMDI file is deprecated in Visual Studio 2011 and there is no equivalent switch for this.

/testsettings

/runconfig

/settings

Example:

Vstest.console.exe accepts the run settings in two formats:

  1. .runsettings file which is the new settings format. .runsettings file can be used as configuration file for all test frameworks like Nunit, Vs unit, xunit and so on.
  1. .testsettings file which was present in Visual Studio 2010 and is specific to Visual studio unit test framework.

Example:

Mstest.exe /testcontainer:abc.dll /testsettings:nightly.testsettings

Vstest.console.exe abc.dll /settings:nightly.testsettings

/resultsfile

/settings

If the default location of results need to be overriden user need to pass this value using a runsettings file.

Example:

Mstest.exe /testcontainer:abc.dll /results:C:\Results.trx

Vstest.console.exe abc.dll /settings:output.runsettings where the context of the .runsettings file would be something like below :

<?xml version="1.0" encoding="UTF-8"?>

<RunSettings>

<RunConfiguration>

<ResultsDirectory>c:\</ResultsDirectory>

</RunConfiguration>

</RunSettings>

/test

/tests

Mstest.exe /testcontainer:abc.dll /test:API

Vstest.console.exe abc.dll /tests:API

/noisolation

 

In Visual Studio 2011, the command line runner runs tests in no isolation mode by default. Hence no switch is needed explictly.

/inisolation switch could be specified explicitly to force isolation if required.

Note : Though the default behavior is to run in process, if the input test assembly requires isolation (like needed a different .net platform or different architecture of the test runner, tests would be run in isolation)

Example:

Mstest.exe /testcontainer:abc.dll /noisolation

Vstest.console.exe abc.dll

/noresults

 

Vstest.console.exe doesn’t create the TRX file by default.

Results would be sent only to console. If TRX is needed, user has to enable it explicitly by specifying /logger:trx

Example:

Mstest.exe /testcontainer:abc.dll /noresults

Vstest.console.exe abc.dll

Mstest.exe /testcontainer:abc.dll

Vstest.console.exe abc.dll /logger:trx

/detail

 

Vstest.console.exe prints the stack trace and error message of each test to the console by default.

Hence there is no equivalent of /detail switch

Example:

Mstest.console.exe /testcontainer:abc.dll /detail:errormessage /detail:errorstacktrace

Vstest.console.exe abc.dll

/pulish and the related options

N/A

Vstest.console.exe doesn’t expose any functionality to publish results to TFS

/unique

N/A

There is no equivalent switch to unique. But the functionality could be achieved specifying the fully qualified name of the test.