How to manage unit tests in Visual Studio 2012 Update 1 : Part 1–Using Traits in the Unit Test Explorer

How to manage unit tests in Visual Studio 2012 Update 1 : Part 1–Using Traits in the Unit Test Explorer

Rate This
  • Comments 21

As a result of writing the post: Edit Test Case properties directly from the test runner of Microsoft Test Manager I asked one the ALM MVPs, Terje Sandstrom, to also look at one of the most requested Visual Studio Update 1 features – but an area I didn’t have a post on yet: Grouping in the new Unit Test Explorer.   He takes a look and by morning sends me this post and indicates he wants to do three more in this series!

Part 2: Using traits in TFS Build

Part 3: Testing and building using different test frameworks

Part 4: Managing code coverage for unit tests

***************************************************************

Part 1:  Using Traits in the Unit Test Explorer by Terje Sandstrom

This is the first post in a series on Unit Testing in Visual Studio 2012, focusing on the improvements in Update 1.

Visual Studio 2012 has a great new Test Explorer.  The Test Explorer is where you see the results of your unit tests.  In Update 1 it has been extended, and you can now organize and filter the test runs based on several conditions, among them your Projects, and Traits.   Traits is a new concept,  a common denominator for several underlying terms, Test Category, Test Property, Priority, and Owner. 

Traits are not only useful as a grouping mechanism in the Test Explorer, it also part of what can control which tests you run.  This has been a big request – because it means the developer can focus on the tests relevant to the work, and not be bothered with running other, perhaps long-running tests, which would otherwise slow down the whole development  experience.  No longer with Update 1!  

Traits are also used in TFS Build as one of the criteria to determine which tests to run for the different builds there. TFS Build is not “trait-aware” in itself, but TFS Build can act on the underlying types, Categories and Properties. The details are in Part 3 in this series of posts.

 

Using Traits and Projects for grouping

Let us start with a MSTest based example.  In a sample project we add 3 tests, and put them into separate categories.  This is done by adding the TestCategory attribute to the method, and the name of the category as a string.

image_thumb1

Open up the Test Explorer, and choose the Group button.  Compared to RTM we have now got 2 more options, Project and Trait.

image_thumb4

Under the menu the three test methods are listed based on Outcome. Changing that to Trait gives us the test grouped by their MSTest Category.

image_thumb5

 

The other option is to group by Project, and adding a new test project it looks like this:

image_thumb12

 

Using more trait concepts

As mentioned above, Traits can also be based on some other properties.   Owner and Priority are explicit and straight forward enough,  whereas TestProperty gives you a property-value system.  That means you can extend the Traits with your own concepts using TestProperties.

In the code below, we have added these properties, and also a new property called “Time”, to indicate if it is a fast or long running test.

image_thumb2

 

In the Text Explorer, still grouping by Trait, this now looks like:

image_thumb3

 

Notice that the same test method appears multiple times, but this is just visual, they are only run once, so the total time here is not 57mS but still 19mS.

Did I say “just visual” ?  It is in fact more than that Smile

 

Using Traits to control test running

The Search field is named and looks like a search field.  But a better term for it is that it is a Filter controlling what to run.  If you click the small arrow to the right, a drop down menu appears.

SNAGHTML23dfc1c4_thumb1

You see a set of possible Search Filters, and above that a recent list which contains a previous set filter. 

If you click on Trait, it appears inside the search field image_thumb41, and you have to fill out the specific trait-name.

Filtering on trait “CI” gives us this list:

image_thumb6

 

The filter term will match any trait you have, be it in a category, a priority, a owner or in a property.  In fact it matches also these words, so you can set a filter to

Trait:”Priority”

and it will match all tests that have any priority set.

 

 

The filter set will stay there until you change or clear it, or restart Visual Studio. 

With this filter set, the Run All button now runs ONLY the tests within this filter. 

image_thumb8

This also applies to the automatic test runner – I assume you are aware of that one ?

The automatic test runner

SNAGHTML23ec2f46_thumb1

The icon to the upper left is enabled here, it means that the background automatic test runner will run every time a build is performed.

Make it a habit of pressing Shift-Ctrl-B instead of just saving your work.  This keyboard shortcut will do a save all and then build your application. 

Enabling the automatic test runner can be done either from this button in the Test Explorer or from the menu:

image_thumb10

 

Control test runs based on other filter terms

The filter menu allows you to set up filters for not only Traits and Projects, but also for other aspects.  The table below summarizes these with the syntax to use:

Filter term Syntax: Example: Comment:
Error Message Message: <any part of  text after message output> Message:”Assert.Fail”  
Output Output: <any part of the output text> Output:”Result” Assume a trace command sending out “Result is”+resultvalue
Fully Qualified Name Fullname: <namespace.classname.methodname> FullName:”UnitTestProject1” Takes all tests in that namespace
FilePath FilePath: <any part of the file path name> FilePath:”Integration.Test” All tests with this in the name. Includes folders above the file too, that is the file’s fully qualified name.
Outcome Outcome: <Failed|Passed> Outcome:”Failed”  
       

A filter expression can be negated by placing a ‘-‘ in front.

image_thumb14

 

Filter expressions can be combined, that is, you can add more together, just separate them by spaces.  The combined filter expression is the logical AND of the two expressions.

If we look at first only tests for CI, we find two tests:. In

image_thumb22

Adding another search term, looking for only tests with Owner = Terje, reduces the list.

image_thumb18

The most typical use for the logical AND is to handle property value settings.  If you want to explicitly filter on Priority = 2, and be sure not to filter on anything else which uses the number 2, that could be expressed as: image_thumb

 

By combining the different traits you should be able to limit the number of tests you run locally.  Pay attention to creating a set of useful and unique trait names, categories and property/value based ones.  This will make it easier to later set up filters.  Also take into account that you use the same traits when filtering unit test cases  for running in different builds.  We will cover this in Part 3.

Also note that these traits are not limited to MSTest, but works just as fine with the other test frameworks, we cover that in Part 2. 

 

 

Some useful links:

  1. Peter Provost has written a great blog post on the features of the new Unit testing in Visual Studio 2012. 
  2. Running unit tests with the Test Explorer is described in this MSDN article
  3. The different unit test frameworks that can be plugged in, is listed in this post
  4. Installing the unit test frameworks is described here.
  5. Controlling code coverage is described here,
  6. Modifying the runsettings is here
  7. Details on Test Categories

     

    By

    Terje Sandstrom

    ------ Chief Software Geek at Inmeta Consulting in Scandinavia ----- and a Visual Studio ALM MVP

    Leave a Comment
    • Please add 8 and 7 and type the answer here:
    • Post
    • This post is very good!

    • When will Update 1 be available?  We need trait filtering!

    • Until I can group based on test class name, the new test explorer is practically useless. Resharper, CodeRush and VS2010 all understood this. Who made this decision, because I'm willing to bet they've never written any meaningful tests before

    • Please clarify if this helps native unit testing - I suspect not.

      One annoying thing about many VS2012 articles is they fail to make it clear when the C++ world is excluded and some of us then have to deal with management expectations they set.

    • Hello Andy,

      This can be applied to C++ Unit Tests.   While I did call this out I didn't give an example: the syntax looks like:

      BEGIN_TEST_METHOD_ATTRIBUTE(TestMethod1)

                                                     TEST_OWNER(L"ForAndy")

                                     END_TEST_METHOD_ATTRIBUTE()

                                     TEST_METHOD(TestMethod1)

                                     {

                                     }

                                     BEGIN_TEST_METHOD_ATTRIBUTE(TestMethod2)

                                                     TEST_PRIORITY(1)

                                     END_TEST_METHOD_ATTRIBUTE()

                                     TEST_METHOD(TestMethod2)

                                     {

                                     }

                                     BEGIN_TEST_METHOD_ATTRIBUTE(TestMethod3)

                                                     TEST_METHOD_ATTRIBUTE(L"Name", L"Value")

                                     END_TEST_METHOD_ATTRIBUTE()

                                     TEST_METHOD(TestMethod3)

                                     {

                                     }

    • When will this be availble for download? I can see that there is a CTP version but when will it be available from visual studio?

    • "Visual Studio 2012 has a great new Test Explorer"

      What!?!?! You cannot be serious. Sorry to be blunt but the new test explorer is the single worst piece of willful software degrading I have ever witnessed.

      Disclaimer: I haven't read the whole article yet but right now I am so furious that I just had to just comment on this before reading on to see whether your plans for improvement has any hope of alleviating the horrible pain I am suffering as a user of this piece of s*** software component right now.

      I am sorry for using harsh words here and I know I am breaking netiquette, but guys! please!?! The quoted statement is a solid insult to my intelligence and I know a lot of people agree with me.

      Martin

    • I totally agree with Martin Moe. Calling the new Test Explorer "great"? So I guess Microsoft don't want to admit to their mistakes or just trying to sell this miserable feature to everyone.

      We have always used Resharper to execute unit tests, mostly because we use NUnit. When I heard about Test Adapters the idea of the integrated Test Explorer sounded interesting. The Test View in VS2012 was not at all perfect, but when a new product is release I would assume that things would only get better. Guess not. Event after the Update 1 I will still keep to Resharper. The idea of creating multiple sessions and actually execute them in parallel is just great.

      It seems like the Test Explorer is written for small projects only. We have over 5000 tests of different kind that we should be able to handle. Unit tests, Data access tests, communication tests, Integration tests/Function tests (SpecFlow). So organisation for execution is important. Here Resharper just handles that for us.

      Also, the "Test after build" feature is not very useful unless you can define what kind of tests to run.

      There it is. It's off my shoulder. Sorry about that.

      Looking forward to see how the new Test Explorer will do.

      /Michael

    • Organization by project if pretty useless for us since we have all our unit tests in a single project.  It would be much more useful for us to be able to pivot (er filter) by test class name.

    • Great post, very useful to finally exclude slow native integration tests without having to ignore/unignore them.

      Martin/Michael: What is keeping you from using ReSharper or TestDriven.NET or any other UI? For .NET there are many to choose from and Test Explorer is certainly a more simple choice. For native code however there is not much to choose from and Test Explorer is absolutely great. But I still use only NCrunch and ReSharper over in the .NET world :)

    • I have to agree with Martin and Michael. The Test Explorer and Test Results is a horrible UI design and implementation. The update just puts lipstick on a pig. To pretend that it resolves the major concerns about this ui is ridiculous.

    • Is there a way to assign a category/trait at the Test Class level? Is it possible to group all tests defined within a single class?

    • @Babak:  The next part blogs.msdn.com/.../part-2-using-traits-with-different-test-frameworks-in-the-unit-test-explorer.aspx shows how to decorate classes with traits for NUnit and XUnit.  MSTest does not have this possibility.  But since it is now pretty easy to use one of the others, and as they coexist nicely - as shown in part 2 -,  you can mix in NUnit/XUnit with your MSTest tests, and use class traits with them.

    • Is it possible to do a "not equal" filter for a Category. I have several tests with the category "Integration" and I would like to run all tests that are not in the category "Integration".

    • For the size of project I am currently using Visual Studio for, Test Explorer has worked OK so far, but I am now facing Visual Studio crash whenever I open Test Explorer. So from my point of view, Test Explorer is a disaster.

    Page 1 of 2 (21 items) 12