What's New in Visual Studio 11 Beta Unit Testing

What's New in Visual Studio 11 Beta Unit Testing

Rate This
  • Comments 23

For those of you who haven’t been following the changes to unit testing that we first previewed back at the //BUILD/ conference, and for those you who did but want to know what has changed, this post is going to take you through the whole thing. We have made a lot of changes to testing in Visual Studio. These changes are pretty drastic in some cases, but were driven by years of customer feedback and a clarification of our focus and vision.

Customer Feedback

Ever since we first shipped unit testing back in Visual Studio Team System we’ve been given praise and criticism for what it was. It was too slow. Some called it a bad copy of NUnit. Many said we should have just shipped NUnit. There was no support for C++ or Javascript or anything else. As time went on, even though we did a lot of work to bring the performance to parity with other frameworks, the label “too slow” still stuck and more and more people went back to using other test frameworks like NUnit or xUnit.net. But then they had no integrated experience and had to use third party VS plugins to run their tests in the IDE. Additionally, our test framework (colloquially known as MS-Test) was not as fast to evolve as other frameworks, due to the nature of our release cycle and of our back-compat requirements.

Another criticism we got from developers, and agile developers in particular, was that our user experience felt like it was designed for testers and didn’t support a rapid test-code-test-code rhythm like that followed by agile developers. Again, we made small steps to improve this over time (double click to code in VS2010 was one step) but still people wanted more. It just didn’t feel like a tool for developers and agile folks.

So for Visual Studio 11, we bit the bullet and committed to changing this, so let’s see what we did…

The Unit Test Explorer

imageIn VS11, we have replaced the old Test View and Test Results windows with the Unit Test Explorer. This new user interface has a number of important elements that let developers quickly interact with their tests.

Red Green Bar
This seemingly simple feature has been one of the top asks from agile developers since the beginning. Give me progress as the tests are run, changing the bar from green to red as soon as any tests fail. Simple, and now there.

Search
Like most of the rest of VS11, we have integrated search. Search also supports using tokens, like in Windows Explorer, to search for certain kinds of things. So you might choose to add Result:”Failed” to your search string and we will limit the results to only failed tests. In the Beta you can use the Result, FilePath and FullName tokens to control your search results.

Most Important Tests First
Again it seems obvious when you look at it, but which tests do you care about most after a run? The ones that failed. So we float those to the top of the list, making it easy to see what you need to fix.

Run Details
All of the details of the run are in a pane at the bottom of the window. Quickly see how many failed, how many passed, how long the run took, etc. Click on a single test and see even more information about that test including error message, source file, stack trace and links to more information.

We are continuing to do a lot of work on this user experience, so please let us know what you think. Some of the things we’re looking at right now include grouping options, more searching options and better access to specific test result information.

To read more about the Unit Test Explorer in VS11, please check out the MSDN Documentation.

Support For 3rd Party Frameworks

But even before tackling the UI issues, we knew we had to support 3rd party test frameworks. To enable this, we created a new test meta-runner. This is a layer that simply coordinates and controls the flow of data between the user interface and the underlying test frameworks. The architecture of it looks something like this.

image

Using simple plugin adapters, third party test frameworks can plug into the test platform layer and get the full experience of running inside of Visual Studio. One of our design goals for this has been to make sure the test frameworks didn’t have to do an insane amount of work to get this integration and if you look at the source for some of the adapters that have already been released, you will see we succeeded.

These adapters simply have to translate commands like “discover tests” and “run tests” into their own corresponding actions, and then translate their own test case and result data back into our data structures. Occasionally, they have to implement another interface or two to tell us about a new kind of test source (e.g. JavaScript files) but for languages that produce DLLs and EXEs, that step can be skipped.

The best part of all is that since these frameworks are all running the same way, they all get the same experiences in the IDE including things like Debugging support, Code Coverage and the new Fakes framework (more on that in a minute).

To learn about finding and installing plugins for third-party test frameworks, see the MSDN Documentation.

MS-Test Improvements

But we also haven’t ignored the built-in Visual Studio Unit Testing framework (aka MS-Test). This framework is not dead and we are going to continue to invest in making it better.

There were a couple of low hanging fruit that we tackled:

  • Performance and scale improvements – more tests faster is a good thing
  • Proper support for .NET 64-bit and multi-targeting

Also, you can rest assured that your VS2010 Test Projects will continue to work in VS11, and in fact, can go back and forth without issue.

But we also added some new things!

Async Unit Test Support

If you’ve been exploring some of the new .NET 4.5 APIs and the new Windows 8 APIs, you probably have noticed a bunch of methods are using the new Task Async Pattern (TAP) first introduced in the Task Parallel Library.

One of the common things you will find yourself doing when coding with long running methods that return Task or Task<T> is wanting to “await” on the result. This is particularly true when unit testing, because you probably need the result value to test against!

In VS11 Beta, MS-Test now supports creating test methods that are marked async and that can therefore use the await keyword inside the method body. Here is an example:

[TestMethod]
public async Task MyAsyncTest()
{
var result = await SomeLongRunningOperation();
Assert.IsTrue( result );
}

(BTW, xUnit.net has also added support for async test method. Expect more to follow suit soon.)

Native Unit Testing Support

One thing that has C/C++ people very excited about this release is that we now have a true native unit testing framework in the box. No longer will you be required to either use the dreaded “/clr” flag or fall back to 3rd party frameworks for your C++ unit testing.

Here’s a quick example:

#include "stdafx.h"
#include
#include "..\MyProjectUnderTest\MyCodeUnderTest.h"

using namespace Microsoft::VisualStudio::CppUnitTestFramework;

TEST_CLASS(TestClassName)
{
public:
TEST_METHOD(TestMethodName)
{
// Run a function under test here.
ASSERT::AreEqual(expectedValue, actualValue, L"message", LINE_INFO());
}
}

More information on that VS11 native unit testing can be found in the MSDN Documentation.

The Visual Studio Fakes Framework

In VS11 we have included a new test isolation and stubbing framework for managed .NET code that we call Visual Studio Fakes.

One of the best things you can do to ensure your unit tests run fast, are resilient and stable, and actually test what you want to test, is to isolate the code you’re testing from everything else. I sometimes describe this as “the scientific method for code”. Like the scientific method, we want to control the variables at play and be sure the thing we’re evaluating does what we think it should do under controlled conditions.

Visual Studio fakes lets you easily create tests that have this kind of isolation. It provides two ways to do this:

  • Stubs are concrete implementations of interfaces and abstract classes that can be passed in to your system under test. It is important to note that our Stubs are not mocks in the purist sense, because they do not provide a behavioral verification check.
  • Shims are run-time method interceptors. With Shims you can provide your own implementation for almost any method available to your code in .NET, and that includes types and methods from the .NET base class libraries.

When you create Stubs and Shims you provide simple delegates or lambdas for the methods implementations you care about, and we do the rest. Here’s an example Stub from the MSDN docs:

[TestMethod]
public void GetValue()
{
var stub = new StubIGenericMethod();
stub.GetValueOf1 = () => 5;

IGenericMethod target = stub;
Assert.AreEqual(5, target.GetValue());
}

Creating Fakes is as easy as right-clicking on one of your project references and choosing Add Fakes Assembly.

You can read more about Fakes in the MSDN Documentation, and I will be making a more detailed post about it very soon.

Code Coverage Improvements

Have you tried to quickly get code coverage information inside Visual Studio 2010? I know I did, and I didn’t like all of the configuration steps it required. Apparently many of our customers felt the same way, so we made it a lot easier.

In the Unit Test Explorer, to get the Code Coverage for your solution, you now simply click the Run… dropdown button and choose Analyze Code Coverage. That’s it. When the test run is complete, the Code Coverage window will open and you can see the same results you have in VS 2010, including being able to turn on in-editor coloring.

But sometimes, especially when you are getting existing code bases under test, you want to see exactly what one or two tests cover. That is now easy too. Just select those tests in the Unit Test Explorer, right click on them and choose Analyze Code Coverage. We will perform a code coverage test run with only those tests and give you the results. Nice and simple.

Continuously Running Your Tests

One of the things I find myself doing when I am in the code-test-code-test loop, is running my tests after every successful build. Since this is such a common pattern, especially for people doing TDD or writing lots of unit tests, we decided to just make it a feature.

To enable this, open the Unit Test main menu, go to Unit Test Settings, and select Run Tests After Build. We have recently added a toggle the button in the Unit Test Explorer tool window, so in the next release it is easier to toggle on and off.

When this option is enabled, we will run tests after each successful build. If you had any failing tests in your last run, we will take the shortcut of running those first. If they pass, we will then run all tests. Our goal here was to get you feedback quickly and to keep you coding “in the green”.

What is not there?

This is a big set of changes and consequently some things are not there in the Beta. Some may come back, some will not, but these changes are setting us up to do a lot of great things in the future, and we are confident that it was the right decision.

Removed or deprecated in VS11

  • Test Lists – Deprecated in VS11. If you have Test Lists from your VS2010 test projects, they will still work, but VS11 does not have a way for you to create new ones. We have some very cool ideas of how to address this cut, and we think you will like what we have in mind.
  • Private Accessors – This feature was deprecated in the VS2010 product and has been removed in VS11. If you have test projects from VS2010 that include private accessors, they will still work in VS11, but the wizard for creating new ones no longer exists.
  • Test Impact in the IDE – Removed in VS11 (but still present in MTM and Team Build). The Test Impact feature was originally created for the Microsoft Test Manager (MTM) and Team Build experiences, and was added to VS2010 late in the cycle. And it shows. We had very consistent feedback that it was too slow and would often return incorrect results, so we removed it. We do still believe that the scenario of telling the developer which tests are important to run is a very important scenario, and we will continue to work to find good solutions here, but this feature simply was not the answer.
  • Generate Unit Test Wizard – In VS2010 you could right click on a method in your code and we would generate a unit test into your test project. This wizard was very tightly coupled to MS-Test and depended on features like Private Accessors to do its work, so it was cut. We are exploring alternatives here, but don’t have any good solutions yet.

Not there yet, but we are working on it

  • Unit Test Startup Performance – RIght now we have a pretty ugly delay after you start a test run and we are aggressively working to make that go away.
  • Grouping and sorting in the Unit Test Explorer – We know that there are a lot of experience gaps in the user experience right now and this is a painful one. We are working to address this right now.
  • Editor right-click to run – Right now we have the “Run Tests” menu in the editor context menu, but it runs all of the tests in that file. We know people want to run a single test and not the whole file and will be working to put this back soon.
  • Hard to see test details – One thing we’ve heard is that sometimes people want a full-document view where they can see everything about a test: output messages, stack traces… everything. We are looking into a good solution here.
  • Copy test results to clipboard – In the Beta you can right click on any test and choose Copy, which will put that test’s result information on the clipboard. We want to make a better experience than that that lets people select and copy from the details themselves.
Leave a Comment
  • Please add 8 and 6 and type the answer here:
  • Post
  • What SKUs is Fakes available with?  I noticed in the MSDN documentation for Shims it mentions the IntelliTrace profiler, suggesting it's only available with Ultimate.

    Martin

  • @Martin Costello - The Fakes framework is available in the Premium and Ultimate VS11 SKUs.

  • Saw your VS 11 ALM / Unit Testing session at the MVP Summit. Great work going on in this space.  I've been playing with the UTE on a legacy project this week (1431 unit tests) and the performance is really good.  However, usabilty wise, you have defintely identifed three of the key areas that still need work: grouping/sorting, hard to see test details and details select/copy to clipboard.  The biggest one for us is the grouping/sorting.  With our 1431 tests, many of which have the same test method names, it is really hard to find an interesting subset of tests to run/debug.  At the very least, give us the ability to list tests by ClassName.TestMethodName.  That would help a lot.  Anyway, it really is awesome that you guys are investing so much in unit testing for VS 11.

  • @Keith Hill - Yeah, we're working hard on these usability things so expect lots of improvements there over time. Thanks for the feedback, please keep it coming.

  • Very useful, thanks

  • You say that you are working hard on "these usability things" like grouping/sorting [which is great news], but I notice that Keith Hill's Connect issue on this [Bug 729524] has already been closed as Deferred.  I think that is more than just a minor usability inconvenience for folks with non-trivial numbers of tests to manage, especially if launching single tests from the editor is also missing.  Does ""working hard" on a "closed as deferred" issue mean "will be in the next release," I hope?

  • Hi Peter,

    When I run code coverage on the assemblies it's instrumenting my test dll too. I've specified a testsetttings file which has only the "real" code assembly selected in the diagnostics section, set this from the Unit Test menu and specified this in my build process but it seems to be ignored by both TFS 11 and VS 11. Is there a reason for this?

    My thanks for getting MSTest playing nicely with F#, that was a very pleasant surprise

    Cheers

  • @Dylan - Except for Dev10 back-compat scenarios, the .testsettings file is really for configuring your TeamBuild. Our Code Coverage runs over everything. Is that an issue? Can you help me understand why you wouldn't want to see code coverage on the test projects too (I know I do). Thanks!

  • @Byron Jennings - If you look in my "We are working on it section" you will see that I did say we are actually working on this right now. We have to use "Deferred" as the Connect way of saying "We put this on our backlog and are working on it." If you look at Ian's comment in Keith's bug, you will see he says we are going to get this done very soon. Trust me, we want this as bad as you guys do.

  • Can you confirm that Post Build Test Run is in Ultimate edition only as mentioned here

    www.dotnetcurry.com/ShowArticle.aspx

  • Hi Peter,

    I guess I just never have run code coverage on unit tests. It seems a bit odd to me that you'd want to do that. Take a test that sets a flag to true if a particular method gets invoked (ignore just calling verify on Moq for brevity here). I want the test to assert that the flag is false. The code in the callback should never get hit in a passing test, which lowers the code coverage metric. And for people that are all jacked up about the numbers having a lowered code coverage could start a minor panic. I'll have a think about it and see, but I take your point that you probably do want to have code coverage on your tests, and I realize that you can see how much coverage a particular assembly has in the report.

    Just as an aside, is there a way to view the code coverage from builds in a report?

  • Sorry if this is a dumb question, but for the 'Generate Unit Test Wizard', could the old version (or something like it) be made available as a NuGet package for those using MSTest?  (Perhaps then with a name of 'Generate MSTest Unit Tests'?)

    If it could be made available as an open-source project, then even if the existing source is MSTest-specific, maybe others could make it work for other testing frameworks, or worst-case fork it and make modified versions for the other frameworks?

  • One thing I really like about the nUnit test runner app is how it groups tests.  It would be nice to have a similar view: presenting a grouping/hierarchy view where assembly is the top-level, followed by a level for each namespace element, then by class name, and finally by the test name itself.  The tool also provides the ability to select any level of the hierarchy and run tests for all its children (or itself if it is a leaf node).

  • Is Code Coverage Coloring suppose to work for native code in v11 Beta?

  • In my software engineering role, I often create libraries that contain many internal mathematical routines which are behind the public face of the library.  It is very important to have robust automated unit tests for these routines to ensure the integrity of the library.  For reasons that are beyond the scope of this post, it is impractical to unit test solely at the public member layer and many of the internal routines have little or no meaning outside the context of the library in which they reside.  Sometimes a private method is created to increase the maintainability and readability of an otherwise inscrutable computation.

    Consequently I found the ‘private accessor’ feature set quite effective in reducing the amount of custom scaffolding created to support testing and I am disappointed at its termination in the latest version of Visual Studio.  I know that some have argued that unit tests should only be constructed for public members but I would suggest while that may be a good practice is most cases, it is a bad practice in some cases. Well, this post is already too long and I have some test scaffolding to write.

Page 1 of 2 (23 items) 12