Publishing test results through command line test runner

Publishing test results through command line test runner

  • Comments 8

Generally, a subset of tests are executed as part of a build as Build Verification Tests. You may want to run the full set of automated tests on a local machine and publish them to a TFS build. You can then get cumulative report of all tests run for that particular build. Alternately, you may have other standalone systems which execute automated tests as part of testing process. Now you can report results from these systems to a build.

TfsPublisher is a logger extension (see writing custom loggers for command line test runner) for Visual Studio 2012 Update 1 command line test runner, vstest.console.exe. It enables publishing test results from the command line to either on-premise or hosted TFS build. This capability allows publishing results from various sessions or machines to a single TFS build so that cumulative reports can be obtained for that build.

This blog briefs on the usage and options of the new logger - TfsPublisher. It’s installed at

“C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\Extensions\Microsoft.VisualStudio.TestPlatform.Extensions.TfsLogger.dll”

Usage:

/logger:TfsPublisher;

Collection=<team project collection URL>;

BuildName=<build name>;

TeamProject=<team project name>

[;Platform=<Defaults to "Any CPU">]

[;Flavor=<Defaults to "Debug">]

[;RunTitle=<title>]

Note: Optional parameters are wrapped with [ ]

Command line options:

· Collection : Team project collection URL.

· BuildName : Target build to publish test results against.

· TeamProject : Target team project.

· Platform : Optional build target platform (defaults to "Any CPU").

· Flavor : Optional build target flavor (defaults to "Debug").

· RunTitle : Name for the published results.

If Platform or Flavor is not specified, publishing would use default values. On successful publish, MTM URL of the published results is printed onto the console.

Published results report can be viewed in Microsoft Test Manager or through build reports from Visual Studio and Web access.

Publishing fails if either the default or user supplied values for Platform or Flavor doesn't match with the target build and an appropriate error message is reported.

Test results can be published through TfsPublisher only for the current test run and publishing results from TRX files is not supported. If /EnableCodeCoverage switch is used, code coverage data is published along with the test results.

Following examples illustrates the usage of TfsPublisher logger from vstest.console.exe.

Examples:

1. Publishing using defaults

# vstest.console.exe MyTest.dll /logger:TfsPublisher;Collection=http://localhost:8080/tfs/DefaultCollection;TeamProject=MyProject;BuildName=DailyBuild_20121130.1

image

Published results can be viewed through the reported MTM URL: mtm://localhost:8080/tfs/DefaultCollection/p:MyProject/Testing/testrun/open?id=3

image

2. Publishing by supplying all parameters

# vstest.console.exe MyTest.dll  /logger:TfsPublisher;Collection=http://localhost:8080/tfs/DefaultCollection;TeamProject=MyProject;BuildName=DailyBuild_20121129.2;Platform=x86;Flavor=Release;RunTitle="Project tests"

image

Here is how the published results appear from TFS web access build report.

clip_image004

3. Publishing with code coverage data (platform and flavor are left to defaults)

# vstest.console.exe MyTest.dll /logger:TfsPublisher;Collection=http://localhost:8080/tfs/DefaultCollection;TeamProject=MyProject;BuildName=DailyBuild_20121129.1;RunTitle="Project tests with code coverage data" /EnableCodeCoverage

image

Here is how the published results appear from Visual Studio 2012 build report.

image

Note: TfsPublisher requires Visual Studio 2012 Update 1. It’s compatible with Team Foundation Server 2012.

 

This blog is from Shyam Prakash Velupula. Here is Shyam in his own words.

I am passionate about programming and reading tech stuff, currently a member of Visual Studio Product development team.

Leave a Comment
  • Please add 6 and 3 and type the answer here:
  • Post
  • Hi,

    Thanks for your post. We are using TFS 2010 but since our project is using Microsoft Fakes, I had to change our build templates to use VSTest.console.exe to run our test. I am passing the /EnableCodeCoverage switch as well but doesn't see any code coverage information in our build.

    Do you know if this would work with TFS 2010?

    Cheers,

    Hamid

  • @Hamid. This scenario will not work because the VS2012 .coverage file format would not be recognizable by TFS2010.

  • Hi,

    I am planning to integrate TFS Test manager with Eggplant, mobile app testing tool. My aim is to get the automated tests created using eggplant into TFS, invoke tests through TFS and have the results and report summary in TFS.

    Ie that possible? through CLI?

  • @Nilesh

    It is possible and there are multiple options available to you.

    You can create a generic test and invoke the eggplant command line from it.

    You can create your own custom wrapper on unit test and then invoke the command line from it.

    In either case, you will then attach the generic test or unit test to a test case in TFS and you can then invoke it from Microsoft Test Manager and store results in TFS.

  • The TfsPublisher (v120) does check in test results, but it does not update the BuildDefinition.TestStatus property of the targeted Build in the TFS, that is passed in with the /BuildName argument. Is this by design? Will you add this in upcoming versions?

  • Hi,

    I've followed these steps exactly. The output from the tool says, "Publish completed successfully". I also get an mtm link (which I can't do anything with since I don't have Microsoft Test Manager).

    Since it says "Publish completed successfully", I expect to be able to see the results of the test run against the build. However, nothing appears.  I don't know what I can be doing wrong, since everything seems to be working fine apart from the actual publishing of the results.

  • Agree with Paul Simkins, I do have MTM installed, and the link works and shows results, but if I try to view test runs from MTM by click on test, analyze test runs, there is nothing there. Is there a way we can associate this with a specific test plan so we can see the results?

  • We use TFS 2013. I  wanted to execute XUnit.net test cases as part of build process. I was not sure how to do this with  Default Build Template 's Visual Studio Test Runner Settings. But I could XUnit Test Cases like following

    vstest.console.exe MyTests.dll /logger:TfsPublisher;Collection=%TF_BUILD_COLLECTIONURI%;TeamProject=MyProject.MVC;BuildName="%TF_BUILD_BUILDNUMBER%";RunTitle="Project tests with code coverage data" /EnableCodeCoverage /TestAdapterPath:"%TF_BUILD_BUILDDIRECTORY%\src\packages\xunit.runner.visualstudio.2.0.0\build\_common\"

    With above command my XUnit Test commands are running successfully. I have put the above command in a PostBuildScript.cmd file and execute this Post Build Scripts.

    From Build Diagnostic i can see that all tests are executed and have the following message also

    Publish completed successfully. Test Results: mtm://<Server>:8080/tfs/automation/p:MyBuild.MVC/Testing/testrun/open?id=9698.

    But Build execution summary(or Build Report)  in Visual studio 2013 does not show anything about test execution.

    Can you please let me know if i have missed some thing?

Page 1 of 1 (8 items)