How to manage unit tests in Visual Studio 2012 Update 1 : Part 1–Using Traits in the Unit Test Explorer

How to manage unit tests in Visual Studio 2012 Update 1 : Part 1–Using Traits in the Unit Test Explorer

Rate This
  • Comments 18

As a result of writing the post: Edit Test Case properties directly from the test runner of Microsoft Test Manager I asked one the ALM MVPs, Terje Sandstrom, to also look at one of the most requested Visual Studio Update 1 features – but an area I didn’t have a post on yet: Grouping in the new Unit Test Explorer.   He takes a look and by morning sends me this post and indicates he wants to do three more in this series!

Part 2: Using traits in TFS Build

Part 3: Testing and building using different test frameworks

Part 4: Managing code coverage for unit tests

***************************************************************

Part 1:  Using Traits in the Unit Test Explorer by Terje Sandstrom

This is the first post in a series on Unit Testing in Visual Studio 2012, focusing on the improvements in Update 1.

Visual Studio 2012 has a great new Test Explorer.  The Test Explorer is where you see the results of your unit tests.  In Update 1 it has been extended, and you can now organize and filter the test runs based on several conditions, among them your Projects, and Traits.   Traits is a new concept,  a common denominator for several underlying terms, Test Category, Test Property, Priority, and Owner. 

Traits are not only useful as a grouping mechanism in the Test Explorer, it also part of what can control which tests you run.  This has been a big request – because it means the developer can focus on the tests relevant to the work, and not be bothered with running other, perhaps long-running tests, which would otherwise slow down the whole development  experience.  No longer with Update 1!  

Traits are also used in TFS Build as one of the criteria to determine which tests to run for the different builds there. TFS Build is not “trait-aware” in itself, but TFS Build can act on the underlying types, Categories and Properties. The details are in Part 3 in this series of posts.

 

Using Traits and Projects for grouping

Let us start with a MSTest based example.  In a sample project we add 3 tests, and put them into separate categories.  This is done by adding the TestCategory attribute to the method, and the name of the category as a string.

image_thumb1

Open up the Test Explorer, and choose the Group button.  Compared to RTM we have now got 2 more options, Project and Trait.

image_thumb4

Under the menu the three test methods are listed based on Outcome. Changing that to Trait gives us the test grouped by their MSTest Category.

image_thumb5

 

The other option is to group by Project, and adding a new test project it looks like this:

image_thumb12

 

Using more trait concepts

As mentioned above, Traits can also be based on some other properties.   Owner and Priority are explicit and straight forward enough,  whereas TestProperty gives you a property-value system.  That means you can extend the Traits with your own concepts using TestProperties.

In the code below, we have added these properties, and also a new property called “Time”, to indicate if it is a fast or long running test.

image_thumb2

 

In the Text Explorer, still grouping by Trait, this now looks like:

image_thumb3

 

Notice that the same test method appears multiple times, but this is just visual, they are only run once, so the total time here is not 57mS but still 19mS.

Did I say “just visual” ?  It is in fact more than that Smile

 

Using Traits to control test running

The Search field is named and looks like a search field.  But a better term for it is that it is a Filter controlling what to run.  If you click the small arrow to the right, a drop down menu appears.

SNAGHTML23dfc1c4_thumb1

You see a set of possible Search Filters, and above that a recent list which contains a previous set filter. 

If you click on Trait, it appears inside the search field image_thumb41, and you have to fill out the specific trait-name.

Filtering on trait “CI” gives us this list:

image_thumb6

 

The filter term will match any trait you have, be it in a category, a priority, a owner or in a property.  In fact it matches also these words, so you can set a filter to

Trait:”Priority”

and it will match all tests that have any priority set.

 

 

The filter set will stay there until you change or clear it, or restart Visual Studio. 

With this filter set, the Run All button now runs ONLY the tests within this filter. 

image_thumb8

This also applies to the automatic test runner – I assume you are aware of that one ?

The automatic test runner

SNAGHTML23ec2f46_thumb1

The icon to the upper left is enabled here, it means that the background automatic test runner will run every time a build is performed.

Make it a habit of pressing Shift-Ctrl-B instead of just saving your work.  This keyboard shortcut will do a save all and then build your application. 

Enabling the automatic test runner can be done either from this button in the Test Explorer or from the menu:

image_thumb10

 

Control test runs based on other filter terms

The filter menu allows you to set up filters for not only Traits and Projects, but also for other aspects.  The table below summarizes these with the syntax to use:

Filter term Syntax: Example: Comment:
Error Message Message: <any part of  text after message output> Message:”Assert.Fail”  
Output Output: <any part of the output text> Output:”Result” Assume a trace command sending out “Result is”+resultvalue
Fully Qualified Name Fullname: <namespace.classname.methodname> FullName:”UnitTestProject1” Takes all tests in that namespace
FilePath FilePath: <any part of the file path name> FilePath:”Integration.Test” All tests with this in the name. Includes folders above the file too, that is the file’s fully qualified name.
Outcome Outcome: <Failed|Passed> Outcome:”Failed”  
       

A filter expression can be negated by placing a ‘-‘ in front.

image_thumb14

 

Filter expressions can be combined, that is, you can add more together, just separate them by spaces.  The combined filter expression is the logical AND of the two expressions.

If we look at first only tests for CI, we find two tests:. In

image_thumb22

Adding another search term, looking for only tests with Owner = Terje, reduces the list.

image_thumb18

The most typical use for the logical AND is to handle property value settings.  If you want to explicitly filter on Priority = 2, and be sure not to filter on anything else which uses the number 2, that could be expressed as: image_thumb

 

By combining the different traits you should be able to limit the number of tests you run locally.  Pay attention to creating a set of useful and unique trait names, categories and property/value based ones.  This will make it easier to later set up filters.  Also take into account that you use the same traits when filtering unit test cases  for running in different builds.  We will cover this in Part 3.

Also note that these traits are not limited to MSTest, but works just as fine with the other test frameworks, we cover that in Part 2. 

 

 

Some useful links:

  1. Peter Provost has written a great blog post on the features of the new Unit testing in Visual Studio 2012. 
  2. Running unit tests with the Test Explorer is described in this MSDN article
  3. The different unit test frameworks that can be plugged in, is listed in this post
  4. Installing the unit test frameworks is described here.
  5. Controlling code coverage is described here,
  6. Modifying the runsettings is here
  7. Details on Test Categories

     

    By

    Terje Sandstrom

    ------ Chief Software Geek at Inmeta Consulting in Scandinavia ----- and a Visual Studio ALM MVP

    Leave a Comment
    • Please add 3 and 8 and type the answer here:
    • Post
    • Hi Terje!

      Thanks for good article. But I'm very concerned why Microsoft doesn't use grouping by namespaces, such as Reshareper and other runners?

      Why we need to mess or tests with custom attributes when mostly simple case is to use grouping by namespace?

    • With Test List you where able to right click on a Unit Test it manage the properties of this.  Is this option no longer available.  The allowed you to easily catagorise tests and make them data driven.

      Now the functionality is no longer available?  Why would you remove functionality like that and replace it with the annoying and stupid "TRAITS"...

    • Hi,

      I have VS2012 Ultimate Upgrade 3.

      I recorded WebTest and generated a code (using VS2012 tool).

      I want to "Expose" the TestMethod that was created to the MS Test Runner so I will be able to run it automatically.

      My code looks like :

         [TestClass]

         public class WebClientLoginLogoutCoded : WebTest {

             public WebClientLoginLogoutCoded() {

                 this.PreAuthenticate = true;

             }

             [TestMethod]

             public override IEnumerator<WebTestRequest> GetRequestEnumerator() {...}

      and When I try to complie it, I get the error :

      "

      UTA007: Method GetRequestEnumerator defined in class ReadyMind_Load.WebClientLoginLogoutCoded does not have correct signature. Test method marked with the [TestMethod] attribute must be non-static, public, does not return a value and should not take any parameter. for example: public void Test.Class1.Test().  Additionally, return-type must be Task if you are running async unit tests. Example: public async Task Test.Class1.Test2().

      "

      how can I solve my problem (I have few directions) :

      1. should I change the GetRequestEnumerator() signature, if yes to what?

      2. should I change one (or both) the attribute, if yes to what?

      3. is there another way that I can "Expose"/"Declare" the test method to ms test runner?

      Thanks

    Page 2 of 2 (18 items) 12