Part 2–Using Traits with different test frameworks in the Unit Test Explorer

Part 2–Using Traits with different test frameworks in the Unit Test Explorer

Rate This
  • Comments 5

As promised one of the ALM MVPs, Terje Sandstrom, has delivered Part 2 of his Unit Test Series: “Using Traits with different test frameworks in the Unit Test Explorer”.

From the traffic and comments on his first post these are clearly very appreciated articles!

Thanks Terje!

 

*********************************************************************************************************************************************’

In Part 1  traits in general was described, and we used examples from using MSTest as a testing framework.  In this part we will built upon what we did in Part 1, but extend it with many of the other frameworks.  The Unit Test Explorer is test framework agnostic!  It works with any framework, as long as there exist an adapter, and such adapters are coming for NUnit, XUnit, MBUnit, Javascript-Chutzpah, MS Native C++ and even more.  A list of these and links to information and downloads for this can be found in this blog post. Note that some of these are being updated along with the release of Update 1. See summary at the end of the blog post.

The summary at the end also shows the syntax for the different traits in the different test frameworks.

We will continue with the code examples from Part 1, and do something that perhaps is not so usual, mixing in some of the other frameworks and let them play together !  That way it is also easier to see the differences between these.  It will be done step by step, so the examples expand as we add more.  The sequence is completely random and doesn’t express any preferences.

What is pretty cool, is that we show that in principle all types of tests can be run at the same time, many even in the same project, although Managed C++, Native C++ and Javascript tests stay in their own projects, but all in the same solution, and the results will be merged together in the Test Explorer !

The different frameworks and their adapters are at somewhat different levels regarding how compliant they are to the new Trait system.  Information is given for each framework on what is missing, and also expected time of arrival for the improvements where we have managed to obtain that information.

 

Adding in NUnit

Now let us mix in a NUnit Test first. In order to do this you must install the NUnit Test Adapter from Visual Studio Gallery, and you must add a NUnit package from NuGet to your project. Adding the NuGet is done in a few steps as shown in the image series below.

Step A: From Visual Studio Tools menu, choose Library Package Manager/Manage NuGet Packages for Solution.

image_thumb4

Step B: If this is the first time you add this to any solution, choose Online, and add “NUnit” in the search box, select the NUnit. (Note, you don’t need the NUnit.Runners, as VIsual Studio has its own runner that will run these for you). If you already have done this once, go instead to the Installed Packages, NUnit will be listed there and press the button “Manage”. Any of these will take you to step ‘C’.

image_thumb5

Step C: Choose the test projects where you want to include NUnit.

image_thumb6

Your project will then look like this, where (1) is included Dll and the NuGet package configuration file. (If you ever want to remove it, these two files are the ones to remove. If you got more packages, you must open and edit the packages.config.)

SNAGHTML1b4cca11_thumb1_thumb

Now you can add a new NUnit test file to your project, 2) in the screenshot above.

NUnit has a different way of setting up categories, in fact it can be done in two different ways. You can add the category as a text string, the same way as you do it in MSTest above, or you can derive a new class from the Unit CategoryAttribute class. We have done both things below:

image_thumb7_thumb

When opening up the Test Explorer now we see the following:

image_thumb9_thumb

Both methods are in the same Trait, named Category [CI], even if they were defined differently. That is pretty cool !

Notice that the MSTest test that is in the CI category is placed in its own group.  The categories from NUnit and MSTest are different even if they are spelled equally.  That, however, only applies for categories.

If we add in another test method and decorate that with the same kind of attributes that we did in the MSTest example in Part 1:

image_thumb8

In NUnit there are no explicit Owner and Priority property, but you can use the general Property attribute instead.  The NUnit Property is more advanced than the MSTest Property, it has overloads for string, int and double, so implementing Priority with this makes it equal to the MSTest specific Priority.

The output from this then becomes:

image_thumb11

Note that except for the categories, all the other properties are collected together for both MSTest and NUnit.

Status

  1. Current version of the adapter at time of writing is 0.93.  There is a bug concerning interoperability with Native C++ which is fixed in the upcoming 0.94 version, ETA Nov 30th 2012.
  2. Version 0.94 will also add class based Categories, see discussion below under xUnit. 

 

Adding in xUnit.net

The xUnit.Net adapter can also be found at Visual Studio Gallery (traits implemented in version 0.9.3), and the framework is found on NuGet the same way, search for “xUnit”, and install both the same way as done for NUnit, adding the libraries to the test project. 

image_thumb91

In the xUnit.net framework, the different traits are implemented by decorating the test method with the Trait attribute.  This is used for all kind of properties.  Everything is a Trait in xUnit.  It doesn’t have any int overloads, so the priority must use a string representation, but the Test Explorer will treat that string and the MSTest int as the same.

The same code then becomes:

image_thumb3

You can also derive new classes from the TraitAttribute class, so if you like you can implement for example the Category, specific categories you want, and the Priority attributes that way:

image_thumb12

The results with both NUnit, MSTest and XUnit running at the same time is then:

image_thumb18

Both the NUnit and XUnit categores fall sweetly into the Categories group, MSTest are in its own special category types (CI, Developer and Production).  They all treat the other traits equally. Notice that the special new attributes made above for ‘CI’ and ‘Priority’ are treated the same as the original ones.

Class based attributes

When I write unit tests I often group them by class.  I rarely put two different types of unit tests in the same class.  So, what I would like is to be able to decorate the class with Traits too.  Those traits should then be treated as applying to all the methods in that class.  It turns out that XUnit is the only framework I have tested so far that have this capability.  MSTest refuses you to add the Traits to the class, they are bound to methods only.  NUnit allows you to do it, but these traits have no effect yet, but the next version of the NUnit adapter should have this capability too for categories.

We add some new test classes to all the projects, looking like this:

image_thumb20

(Equally for NUnit and MSTest, except different suffices, N11/N12 and Ms11/Ms12.

And the results show that the NUnit and MSTest ends up the No Traits group, whereas the XUnit comes in correctly

image_thumb23

 

Adding in MbUnit

To add in MbUnit one need to add first the Gallio framework, then the MbUnit adapter, and finally using NuGet to get the Gallio & MbUnit dll’s into your project:

image_thumb26

 

MbUnit has specific Traits, which matches some of the others, but not all.  It has Category, Author and Importance.

image_thumb41

MBUnit also allows for derivation of new attribute classes like NUnit and xUnit.net, but the adapter doesn’t support this for traits.

One important thing to notice is that the adapter don’t accept the Importance attribute shown above. 

The current version of Gallio/MbUnit, version 3.4 does not work correctly with its own version 3.4 of VS Test Adapter, giving an exception when you try to run it. The adapter seems to have been built for an earlier version.

I did download their adapter code and made a few corrections and added the Trait code, just to see how the underlying MbUnit/Gallio treats this.  The adapter is only a bridge between the test runner and the test framework.  I will try to get these code changes back up to the MbUnit project, so it should be ready to go in the next release.

image_thumb39

Notice that we now are running 3 adapters at the same time - cool !

Status

Current version of the adapter (3.3) does not support Traits, but next version of the adapter will do so.

 

Adding in C++

Managed C++

Test for Managed C++ is built-in in Visual Studio.  It uses the same Traits as for C#. No surprises here. 

image_thumb30

 

Native C++

Test for Native C++ is also built-in to Visual Studio 2012.  The set of traits available are different from managed, and the way they are set is different from managed C++. You can extend the traits by using the same property/value mechanism, the attribute to use then is the TEST_METHOD_ATTRIBUTE which takes a property and a value as parameters.

image_thumb1

The attributes that matches the traits are the three shown in the code example above.

The result when running this is then :

image_thumb7[1]

 

C++ also has full support for class based and even module based traits.  The example code below show a class based trait attribute

image_thumb10

and the result is then that this applies to all methods in this class.

image_thumb11[1]

So the C++ Unit Test framework supports all the traits and on all levels.  That’s very cool !

 

 

Adding in JavaScript testing with Chutzpah

Chutzpah is the test adapter for the QUnit and Jasmine test frameworks for JavaScript.

QUnit and Jasmine have the keywords ’Module’ and ‘Group’ respectively that can be used to decorate the test methods.  Those are the only ones available who can be used to implement Traits.  They are semantically not meant like Traits, but can be used that way.

Also note that Chutzpah currently runs tests per test file, not per test.  So if you select a single test and runs that, all the other tests in the same file are also being executed. This imply you should consider how your Modules and tests are organized by file.  One file should not contain more than one Module, if you intend to run them separately. 

Again, I added a few lines trait coding to the Chutzpah source code, and this is how it looks, using Module as the keyword for a Trait.

image_thumb43

And the results, together with the other tests.  In the screenshot below, I have chosen to execute the first test in group a, and as can be seen the other tests are also run.

clip_image001_thumb1

 

Status

  1. The current version at time of writing (Nov. 12th 2012) of Chutzpah is 2.2.1. This version does not support Traits.  The next version will support traits.

 

Summary

 

Release schedules for trait aware 3rd party framework adapters

Framework ETA Comment
NUnit Nov. 30th 2012, version 0.94  
xUnit.Net Current version 0.9.3  
MbUnit Next version, date TBD Code ready
Chutzpah Next version, date TBD Solution ready, needs coding and some time

 

Traits summarized

The first column merely sets a generic name for the traits, to be able to compare them between the adapters/frameworks. The ‘----‘ means “Not implemented”.

Trait ‘generic’ name MSTest (C# and Man C++) Native C++ NUnit xUnit.Net MbUnit Chutzpah
Category Category TEST_METHOD_ATTRIBUTE(“Category”,<value>) Category Trait(“Category”,<value>) Category Module
Priority Priority TEST_PRIORITY Property(“Priority”,<value>) Trait(“Priority”,<value>) ---- (Importance) ----
Owner Owner TEST_OWNER Property(“Owner”,<value>) Trait(“Category”,<value>) Author(kind of) ----
Property Property TEST_METHOD_ATTRIBUTE Property Trait ---- ----
             
Support inheritance No No Yes Yes No NA
Class based traits No Yes Yes Yes No NA

Property is a generic property/value set, and support for that is very nice to have. Implementing that means the traits are very extensible, and you can twist them to anything you like, and have full support of that in the Test Explorer.

Have fun !

 

******************************

 

By

Terje Sandstrom

------ Chief Software Geek at Inmeta Consulting in Scandinavia ----- and a Visual Studio ALM MVP

Leave a Comment
  • Please add 5 and 5 and type the answer here:
  • Post
  • I just tried new Update and was suprised that the nunit Test Category attribute is only obsurved on the Test itself and not for the whole class. I think it is usual that the class is attributed by the ["Category"] and only in some exceptional cases on the test itself.

    Is there a way to activate the "switch" that the Test Explorer can also organize Traits on the TestClass ?

  • Hello Terje,

    Thank you for this very useful series!

    Be able to run NUnit tests from Visual Studio is great, but it seems be impossible to use these tests to automate TestCases, doesn't it?

    That means I cannot use them for automated tests...

    Do you know if it will be possible in the near future?

    -- Elena

  • Hi Elena!

    To automate functional manual tests, the approach is to use Coded UI tests.  You generate these from a test run of a test case, and it will produce a Coded UI test which is based on the MSTest unit test framework.  

    See this article from Visual Studio magazine visualstudiomagazine.com/.../coded-ui-tests-1.aspx , or go here blogs.msdn.com/.../coded+ui+testing to see a list of Coded UI blog articles from MSDN.

    Due to the multiadapter approach in VS2012, there should no longer be any issues running ordinary tests with NUnit and Coded UI tests with MSTest either.

    Hope this helps !

    Terje

  • I see that Chutzpah 2.3 has been released but I do not see anything about traits in the changelog. Do you know if it is included?

  • Great post (in a great series of posts).

    Just to add something to this (I know the post is a bit old now but why not), SpecFlow test tags (e.g. @mytag) can be hooked up using nUnit to categories. Not entirely on topic but not far off :)

Page 1 of 1 (5 items)