Learn to use Visual Studio, Visual Studio Online, Application Insights and Team
Foundation Server to decrease rework, increase transparency into your application and increase the rate at which
you can ship high quality software throughout the application lifecycle
As a result of writing the post: Edit Test Case properties directly from the test runner of Microsoft Test Manager I asked one the ALM MVPs, Terje Sandstrom, to also look at one of the most requested Visual Studio Update 1 features – but an area I didn’t have a post on yet: Grouping in the new Unit Test Explorer. He takes a look and by morning sends me this post and indicates he wants to do three more in this series!
Part 2: Using traits in TFS Build Part 3: Testing and building using different test frameworks Part 4: Managing code coverage for unit tests
Part 2: Using traits in TFS Build
Part 3: Testing and building using different test frameworks
Part 4: Managing code coverage for unit tests
Part 1: Using Traits in the Unit Test Explorer by Terje Sandstrom
This is the first post in a series on Unit Testing in Visual Studio 2012, focusing on the improvements in Update 1.
Visual Studio 2012 has a great new Test Explorer. The Test Explorer is where you see the results of your unit tests. In Update 1 it has been extended, and you can now organize and filter the test runs based on several conditions, among them your Projects, and Traits. Traits is a new concept, a common denominator for several underlying terms, Test Category, Test Property, Priority, and Owner.
Traits are not only useful as a grouping mechanism in the Test Explorer, it also part of what can control which tests you run. This has been a big request – because it means the developer can focus on the tests relevant to the work, and not be bothered with running other, perhaps long-running tests, which would otherwise slow down the whole development experience. No longer with Update 1!
Traits are also used in TFS Build as one of the criteria to determine which tests to run for the different builds there. TFS Build is not “trait-aware” in itself, but TFS Build can act on the underlying types, Categories and Properties. The details are in Part 3 in this series of posts.
Using Traits and Projects for grouping
Let us start with a MSTest based example. In a sample project we add 3 tests, and put them into separate categories. This is done by adding the TestCategory attribute to the method, and the name of the category as a string.
Open up the Test Explorer, and choose the Group button. Compared to RTM we have now got 2 more options, Project and Trait.
Under the menu the three test methods are listed based on Outcome. Changing that to Trait gives us the test grouped by their MSTest Category.
The other option is to group by Project, and adding a new test project it looks like this:
Using more trait concepts
As mentioned above, Traits can also be based on some other properties. Owner and Priority are explicit and straight forward enough, whereas TestProperty gives you a property-value system. That means you can extend the Traits with your own concepts using TestProperties.
In the code below, we have added these properties, and also a new property called “Time”, to indicate if it is a fast or long running test.
In the Text Explorer, still grouping by Trait, this now looks like:
Notice that the same test method appears multiple times, but this is just visual, they are only run once, so the total time here is not 57mS but still 19mS.
Did I say “just visual” ? It is in fact more than that .
Using Traits to control test running
The Search field is named and looks like a search field. But a better term for it is that it is a Filter controlling what to run. If you click the small arrow to the right, a drop down menu appears.
You see a set of possible Search Filters, and above that a recent list which contains a previous set filter.
If you click on Trait, it appears inside the search field , and you have to fill out the specific trait-name.
Filtering on trait “CI” gives us this list:
The filter term will match any trait you have, be it in a category, a priority, a owner or in a property. In fact it matches also these words, so you can set a filter to
and it will match all tests that have any priority set.
The filter set will stay there until you change or clear it, or restart Visual Studio.
With this filter set, the Run All button now runs ONLY the tests within this filter.
This also applies to the automatic test runner – I assume you are aware of that one ?
The automatic test runner
The icon to the upper left is enabled here, it means that the background automatic test runner will run every time a build is performed.
Make it a habit of pressing Shift-Ctrl-B instead of just saving your work. This keyboard shortcut will do a save all and then build your application.
Enabling the automatic test runner can be done either from this button in the Test Explorer or from the menu:
Control test runs based on other filter terms
The filter menu allows you to set up filters for not only Traits and Projects, but also for other aspects. The table below summarizes these with the syntax to use:
A filter expression can be negated by placing a ‘-‘ in front.
Filter expressions can be combined, that is, you can add more together, just separate them by spaces. The combined filter expression is the logical AND of the two expressions.
If we look at first only tests for CI, we find two tests:. In
Adding another search term, looking for only tests with Owner = Terje, reduces the list.
The most typical use for the logical AND is to handle property value settings. If you want to explicitly filter on Priority = 2, and be sure not to filter on anything else which uses the number 2, that could be expressed as:
By combining the different traits you should be able to limit the number of tests you run locally. Pay attention to creating a set of useful and unique trait names, categories and property/value based ones. This will make it easier to later set up filters. Also take into account that you use the same traits when filtering unit test cases for running in different builds. We will cover this in Part 3.
Also note that these traits are not limited to MSTest, but works just as fine with the other test frameworks, we cover that in Part 2.
Some useful links:
------ Chief Software Geek at Inmeta Consulting in Scandinavia ----- and a Visual Studio ALM MVP
This post is very good!
When will Update 1 be available? We need trait filtering!
Until I can group based on test class name, the new test explorer is practically useless. Resharper, CodeRush and VS2010 all understood this. Who made this decision, because I'm willing to bet they've never written any meaningful tests before
Please clarify if this helps native unit testing - I suspect not.
One annoying thing about many VS2012 articles is they fail to make it clear when the C++ world is excluded and some of us then have to deal with management expectations they set.
This can be applied to C++ Unit Tests. While I did call this out I didn't give an example: the syntax looks like:
When will this be availble for download? I can see that there is a CTP version but when will it be available from visual studio?
"Visual Studio 2012 has a great new Test Explorer"
What!?!?! You cannot be serious. Sorry to be blunt but the new test explorer is the single worst piece of willful software degrading I have ever witnessed.
Disclaimer: I haven't read the whole article yet but right now I am so furious that I just had to just comment on this before reading on to see whether your plans for improvement has any hope of alleviating the horrible pain I am suffering as a user of this piece of s*** software component right now.
I am sorry for using harsh words here and I know I am breaking netiquette, but guys! please!?! The quoted statement is a solid insult to my intelligence and I know a lot of people agree with me.
I totally agree with Martin Moe. Calling the new Test Explorer "great"? So I guess Microsoft don't want to admit to their mistakes or just trying to sell this miserable feature to everyone.
We have always used Resharper to execute unit tests, mostly because we use NUnit. When I heard about Test Adapters the idea of the integrated Test Explorer sounded interesting. The Test View in VS2012 was not at all perfect, but when a new product is release I would assume that things would only get better. Guess not. Event after the Update 1 I will still keep to Resharper. The idea of creating multiple sessions and actually execute them in parallel is just great.
It seems like the Test Explorer is written for small projects only. We have over 5000 tests of different kind that we should be able to handle. Unit tests, Data access tests, communication tests, Integration tests/Function tests (SpecFlow). So organisation for execution is important. Here Resharper just handles that for us.
Also, the "Test after build" feature is not very useful unless you can define what kind of tests to run.
There it is. It's off my shoulder. Sorry about that.
Looking forward to see how the new Test Explorer will do.
Organization by project if pretty useless for us since we have all our unit tests in a single project. It would be much more useful for us to be able to pivot (er filter) by test class name.
Great post, very useful to finally exclude slow native integration tests without having to ignore/unignore them.
Martin/Michael: What is keeping you from using ReSharper or TestDriven.NET or any other UI? For .NET there are many to choose from and Test Explorer is certainly a more simple choice. For native code however there is not much to choose from and Test Explorer is absolutely great. But I still use only NCrunch and ReSharper over in the .NET world :)
I have to agree with Martin and Michael. The Test Explorer and Test Results is a horrible UI design and implementation. The update just puts lipstick on a pig. To pretend that it resolves the major concerns about this ui is ridiculous.
Is there a way to assign a category/trait at the Test Class level? Is it possible to group all tests defined within a single class?
@Babak: The next part blogs.msdn.com/.../part-2-using-traits-with-different-test-frameworks-in-the-unit-test-explorer.aspx shows how to decorate classes with traits for NUnit and XUnit. MSTest does not have this possibility. But since it is now pretty easy to use one of the others, and as they coexist nicely - as shown in part 2 -, you can mix in NUnit/XUnit with your MSTest tests, and use class traits with them.
Is it possible to do a "not equal" filter for a Category. I have several tests with the category "Integration" and I would like to run all tests that are not in the category "Integration".
For the size of project I am currently using Visual Studio for, Test Explorer has worked OK so far, but I am now facing Visual Studio crash whenever I open Test Explorer. So from my point of view, Test Explorer is a disaster.