Publishing test results through command line test runner

Publishing test results through command line test runner

  • Comments 5

Generally, a subset of tests are executed as part of a build as Build Verification Tests. You may want to run the full set of automated tests on a local machine and publish them to a TFS build. You can then get cumulative report of all tests run for that particular build. Alternately, you may have other standalone systems which execute automated tests as part of testing process. Now you can report results from these systems to a build.

TfsPublisher is a logger extension (see writing custom loggers for command line test runner) for Visual Studio 2012 Update 1 command line test runner, vstest.console.exe. It enables publishing test results from the command line to either on-premise or hosted TFS build. This capability allows publishing results from various sessions or machines to a single TFS build so that cumulative reports can be obtained for that build.

This blog briefs on the usage and options of the new logger - TfsPublisher. It’s installed at

“C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\Extensions\Microsoft.VisualStudio.TestPlatform.Extensions.TfsLogger.dll”

Usage:

/logger:TfsPublisher;

Collection=<team project collection URL>;

BuildName=<build name>;

TeamProject=<team project name>

[;Platform=<Defaults to "Any CPU">]

[;Flavor=<Defaults to "Debug">]

[;RunTitle=<title>]

Note: Optional parameters are wrapped with [ ]

Command line options:

· Collection : Team project collection URL.

· BuildName : Target build to publish test results against.

· TeamProject : Target team project.

· Platform : Optional build target platform (defaults to "Any CPU").

· Flavor : Optional build target flavor (defaults to "Debug").

· RunTitle : Name for the published results.

If Platform or Flavor is not specified, publishing would use default values. On successful publish, MTM URL of the published results is printed onto the console.

Published results report can be viewed in Microsoft Test Manager or through build reports from Visual Studio and Web access.

Publishing fails if either the default or user supplied values for Platform or Flavor doesn't match with the target build and an appropriate error message is reported.

Test results can be published through TfsPublisher only for the current test run and publishing results from TRX files is not supported. If /EnableCodeCoverage switch is used, code coverage data is published along with the test results.

Following examples illustrates the usage of TfsPublisher logger from vstest.console.exe.

Examples:

1. Publishing using defaults

# vstest.console.exe MyTest.dll /logger:TfsPublisher;Collection=http://localhost:8080/tfs/DefaultCollection;TeamProject=MyProject;BuildName=DailyBuild_20121130.1

image

Published results can be viewed through the reported MTM URL: mtm://localhost:8080/tfs/DefaultCollection/p:MyProject/Testing/testrun/open?id=3

image

2. Publishing by supplying all parameters

# vstest.console.exe MyTest.dll  /logger:TfsPublisher;Collection=http://localhost:8080/tfs/DefaultCollection;TeamProject=MyProject;BuildName=DailyBuild_20121129.2;Platform=x86;Flavor=Release;RunTitle="Project tests"

image

Here is how the published results appear from TFS web access build report.

clip_image004

3. Publishing with code coverage data (platform and flavor are left to defaults)

# vstest.console.exe MyTest.dll /logger:TfsPublisher;Collection=http://localhost:8080/tfs/DefaultCollection;TeamProject=MyProject;BuildName=DailyBuild_20121129.1;RunTitle="Project tests with code coverage data" /EnableCodeCoverage

image

Here is how the published results appear from Visual Studio 2012 build report.

image

Note: TfsPublisher requires Visual Studio 2012 Update 1. It’s compatible with Team Foundation Server 2012.

 

This blog is from Shyam Prakash Velupula. Here is Shyam in his own words.

I am passionate about programming and reading tech stuff, currently a member of Visual Studio Product development team.

Leave a Comment
  • Please add 8 and 1 and type the answer here:
  • Post
  • Hi,

    Thanks for your post. We are using TFS 2010 but since our project is using Microsoft Fakes, I had to change our build templates to use VSTest.console.exe to run our test. I am passing the /EnableCodeCoverage switch as well but doesn't see any code coverage information in our build.

    Do you know if this would work with TFS 2010?

    Cheers,

    Hamid

  • @Hamid. This scenario will not work because the VS2012 .coverage file format would not be recognizable by TFS2010.

  • Hi,

    I am planning to integrate TFS Test manager with Eggplant, mobile app testing tool. My aim is to get the automated tests created using eggplant into TFS, invoke tests through TFS and have the results and report summary in TFS.

    Ie that possible? through CLI?

  • @Nilesh

    It is possible and there are multiple options available to you.

    You can create a generic test and invoke the eggplant command line from it.

    You can create your own custom wrapper on unit test and then invoke the command line from it.

    In either case, you will then attach the generic test or unit test to a test case in TFS and you can then invoke it from Microsoft Test Manager and store results in TFS.

  • The TfsPublisher (v120) does check in test results, but it does not update the BuildDefinition.TestStatus property of the targeted Build in the TFS, that is passed in with the /BuildName argument. Is this by design? Will you add this in upcoming versions?

Page 1 of 1 (5 items)