Visual Studio 11 Beta was announced sometime back along with Win 8 consumer preview in which Unit testing has been redefined and support for testing Windows Metro style apps is added.

Tests for Metro style apps can be run from Visual Studio, Command Line and Team Build. Steps to create and run unit test from Visual Studio and Team Build are described in Walkthrough: Creating and Running Unit Tests for Metro style Applications and Running Windows 8 Metro style tests in TeamBuild respectively.

This post explains how to run Unit Test for Metro style apps from Command Line tool (vstest.console.exe)


Installing Unit Testing Command Line tool

Unit testing command line tool get installed with Visual Studio 11 Beta. Command Line test runner, vstest.console.exe, is installed at $(VSInstallDir)/Common7/IDE/CommonExtensions/Microsoft/TestWindow. It's also get installed with Remote Debugger SKU.


Creating App Package for Unit Test Library

Create a Unit Test Library Project for Managed (C#/VB) or Native (C++) in Visual Studio. Steps for creating project are same as in Walkthrough: Creating and Running Unit Tests for Metro style Applications.

After tests are authored and ready to run from command line create an app package (.appx file) as follows:

  1. Right click on the project in Solution Explorer.
  2. Go to Store -> Create App Package...
  3. Select "Create a package to use locally only" and click Next
  4. Set desired Package location and Build configuration
  5. Click Create
  6. Click OK

Inside a folder in "Package location" specified in above steps following files can be found:

  1. <filename>.appx -> App package file for test project. It contains test binary and all Unit Testing platform binaries.
  2. <filename>.cer -> Certificate file corresponding to .pfx file used to sign package. It needs to be added to trusted certificates to install package.
  3. <filename>.appxsym -> Package containing public symbols of app package. (Not required for our scenario)
  4. <filename>.bat -> Batch file created with package to install package. (Not required for our scenario)

If you want to execute test on some other system (like ARM machine with Remote Debugger SKU) then copy .appx and .cer files on to that machine.


Executing Unit Tests for Metro style apps from command line

Install Dev License:

Install developer license on the box where test needs to be executed. This is needed to run unit tests for Metro style apps. This is onetime task per machine (may require renewing license, if expired).


Installing the certificate:

On machine where want to run test from command line, install certificate. This is a one time operation per certificate and can be done as follows:

a. From an elevated command prompt execute command “Certutil -addstore root <filename>.cer”


Executing tests:

Test for Metro style app can be executed from command line as follows:

"<TestToolPath>\vstest.console.exe" "<AppxPath>\<filename>.appx" /InIsolation 

<TestToolPath> is location where vstest.console.exe is installed (VS installs it at $(VSInstallDir)/Common7/IDE/CommonExtensions/Microsoft/TestWindow).

<AppxPath> is where App Package for test is present.

/InIsolation is required as tests for Metro style apps can't be run in inproc mode. If option is not given then it will generate a warning.


Command line test runner (vstest.console.exe) will install app package, execute tests and uninstall package.


More options:

/Logger:trx option can be used to generate trx file which can be opened in VS to analyze failures.

/TestCaseFilter can be used to selectively run tests. More info can be found at TestCaseFilter in VS11 Unit Testing

/Tests & /ListTests options are not valid for Metro style app


Common Errors and Resolutions:

a. Don't give dll file directly for test execution. Directly giving dll file will not execute test in appcontainer mode (in which Metro style app executes). Create app package as described above and use it for test execution in appcontainer mode.

b. Test project has its own manifest file and it is not derived from references. Set proper capabilities required to run tests in manifest file of test project.

c. To execute test as 64 bits on a 64 bits machine, use /platform:x64 option. By default, tests are executed as 32 bits on 64s bit machine.

d. If you are taking dependency on some SDK (like C++ test project takes dependency on VSLib SDK) then required dependency packages are copied in "Dependencies" folder with app package by package builder. If copying app package to another location or machine then copy "Dependencies" folder along with package.


If you've any feedback or issue then please do send us on Connect or in the Forums.