How to run tests in a build without test metadata files and test lists (.vsmdi files)
Since the beginning, running tests in Team Build (or MSBuild in general) has meant having to use .vsmdi files to specify the tests to run. Tons of people have complained about it, as it's a burden to create and edit the files, as either VSTS for Testers or the full suite is required in the 2005 release, and merging changes to the file is painful when multiple developers are updating the file. While mstest.exe has a /testcontainer option for simply specifying a DLL or load/web test file, the test tools task used with MSBuild does not expose an equivalent property.
Attached to this post is a zip archive containing the files needed to run tests without metadata. There are three files.
- TestToolsTask.doc contains the instructions for installing the new task DLL and modifying the TfsBuild.proj files to use it.
- Microsoft.TeamFoundation.Build.targets is a replacement for the v1 file by the same name that resides on the build machine.
- Microsoft.TeamFoundation.PowerToy.QualityTools.dll contains the new TestToolsTask class that extends the TestToolsTask class that shipped in v1 and exposes a TestContainers property that is is the equivalent of mstest.exe's /testcontainer option.
After you read the instructions (please, read the instructions), you'll be able to run all of your unit tests by simply specifying the DLLs or even specifying a file name pattern in TfsBuild.proj. Here are a couple of examples. The TestToolsTask.doc file explains how they work, including what %2a means.
<TestContainerInOutput Include="HelloWorldTest.dll" />
<TestContainerInOutput Include="%2a%2a\%2aTest.dll" />
<TestContainer Include="$(SolutionRoot)\TestProject\WebTest1.webtest" />
This new task will be included in future releases of the Team Foundation Power Toys (soon to be called Power Tools). The TestToolsTask in the shipping product will support test containers and Team Build will support using it in the next release of the product.
I started with code and documentation that someone else wrote and finished it off. Thanks to Clark Sell for digging up an old email thread about the original work and Tom Marsh, development lead in VSTS Test, for pointing me to the code and doc. Thanks to Aaron Hallberg and Patrick Carnahan, Team Build developers, for their help. Thanks to Brian Harry for letting me post it.
Please post your feedback in the comments to this post. We'd like to know if this hits the mark, or if there's something else we need to support.
[UPDATE 4/26/2007] New features: Test categories and test names
Pierre Greborio, a developer over in MSTV, has contributed a great new feature: test categories. Those of you who have used NUnit are probably familiar with the Category attribute. Test categories allow you to execute specific groups of unit tests. Unlike the test container feature, the test category feature will not be in Orcas.
While the details are discussed in the TestToolsTask.doc included in the zip file attached to this blog post, here's the quick version.
- Add the Category attribute to your unit test method.
- Specify the category to run in your tfsbuild.proj file.
The other feature that's new in this release is the support for test names. This is equivalent to the mstest.exe /test command line switch. The name that's specified is implicitly a wildcard match. Specifying "Blue" as the test name, for example, will execute every test method that has Blue in the name, including DarkBlue and BlueLight.
Pierre did his testing on Vista. Because the dll is not signed, he ran into some trust issues. If you hit a similar problem, he recommends this forum post for how to get it to work.
[UPDATE 11/9/2006] Bug fix
I've updated the attachment with a new zip archive file. Unfortunately, the task I posted originally didn't result in builds being marked as Failed when the tests had errors. I missed that in my testing. The reason for the problem is that the v1 Team Build logger looks for the task name, TestToolsTask, and the power toy task was originally called TestToolsTaskEx. With this new release, the task has the same name as the original v1 task, so that builds will be marked as failed when the tests fail.
If you downloaded the original release, you simply need to copy the Microsoft.TeamFoundation.Build.targets and Microsoft.TeamFoundation.PowerToys.Tasks.QualityTools.dll files from the zip to get the fix (see the Word doc for the paths).
Thanks to Thomas for pointing out this bug!
tags: vsts, team system, tfs, team foundation, team build, tests, msbuild
