I was just getting started on what I intended to be my next post on this blog when I realized that the topic in that post would be much easier to discuss and understand if I first wrote about Test Points. A Test Point is an object in our TCM Object Model that isn’t really explained or directly surfaced in the UI anywhere, the reason is that the user doesn’t necessarily need to know they’re working with a “Test Point” when they are.
A Test Point represents a Test Case to be run against a Test Configuration in a Test Suite.
In MTLM, if you go to the Testing Center –> Plan tab –> Contents, you see the list of Test Case’s in each suite. One of the columns in the grid is “Configurations” which shows a count of how many Test Configurations that Test Case is tested against in that suite. When you go to Testing Center –> Test tab –> Run Tests and click on the same suite you will see an entry in the grid there for each Test Case / Test Configuration combination. These are Test Points.
Knowing this terminology shouldn’t fundamentally change the way you use the Test Management product, but it should help make future posts easier for me :)
One of the key areas of focus for Visual Studio Team System 2010 is to enhance the experience for testers. There’s already lots of overview information on many of the new features out there so I’m going to dive right in to one of my favorite new features.
In short, the Test Impact Analysis feature gives you a list of recommended tests based on the changes that were made in a build. As you run your tests through Microsoft Test and Lab Manager (code named Camano) you will have the option to run the Test Impact Data Collector which is one of the many built-in Data Collectors available while running tests. This particular data collector will associate the code paths you execute in your application with the test you are running.
When Team Build generates a new build it will analyze what code has been changed since the last build was done and then provide you with a list of the impacted tests. These are the tests that touch the same pieces of code that were changed since the last build.
For a more thorough explanation of the Test Impact Data Collector read this blog post by Amit Chatterjee.
The first thing you need is an Application Under Test, for this walk-through I’ll use a simple calculator application. I’ll have a C# Class Library with some simple Add, Subtract, Multiply, and Divide functions and a C# Windows Forms project that will provide some input fields for the numbers and some buttons that call the library function. I’ll skip the creation of this sample application, you an use the attached sample application, mock up something similar or follow along with your own test application.
- Add your test application to Source Control
- Create a new Build Definition for your application
- Make sure “Analyze Test Impacts” in the Process tab is set to “True”:
- Queue a new build
- Start “Microsoft Test and Lab Manager” (aka Camano)
- If the dialog below comes up click “Set context”:
- On the “Set Plan Context” page click “New”
- Enter a name for your Test Plan and click Save & Close
- Go to Testing Center –> Plan –> Contents
- Click “New” in the toolbar to add a new Test case
- Add a Title for the Test case, mine is “Add – Basic”, I’ll repeat these steps and add “Subtract – Basic”, “Multiply – Basic”, and “Divide – Basic” as well. Here’s what my Test Plan Contents will look like when I’ve done this.
- Now it’s time to switch to the Lab Center so we can set up our Test Settings
- Switch to the Test Settings tab and click “New”
- Enter a Name and click “Next”
- To keep things simple uncheck all collectors except the “Test Impact Collector” and click “Save and Close”
- Go back to the Testing Center.
- Switch to the Track tab and go to the “Assign Build” activity
- The build you created earlier will be in the “Available builds” list, select it and click “Assign to plan”.
- Now it’s time to run our tests. Switch to Testing Center –> Test –> Run Tests
- Select all of your tests and click “Run with Options”
- In the Run Options dialog select your Test Settings and verify that your newest build is selected. Click Run.
- Test Runner will now appear
- Click Start test and record, then launch your Application Under Test. This ordering is important, you must start the test before launching the test application or the impact data will not be collected.
- Perform each test now. For my Add test I’ll set 2 numbers in my text boxes and click Add, for Subtract test I’ll set 2 numbers and click Subtract, etc. etc. Run your test then click “End Test”, then start the next test. There’s no need to re-launch your application between every test. When you’re done click “Save and Close” to return to Manual Test and Lab Manager (MTLM). It should look something like this:
- To verify the Test Impact Collector worked correctly you can open up one of your test results and look at it’s attachments, there should be an impact xml file listed there. If there isn’t, re-check your steps above.
- Now, switch to Visual Studio and make a change to your application under test. I’m going to make a change to my “Add” method and check it in.
- Queue a new build and wait for it to complete.
- Switch back to MTLM
- Go to Testing Center –> Track –> Assign Build and assign your new build to your plan. (See Step #12 if you need help)
- Now switch to Testing Center –> Track –> Recommended Tests
- Notice the “Add – Basic” test is now listed.
That’s it for this post, I look forward to any questions or comments you may have. If you have any questions on some of the steps above try looking them up on the MSDN site. Topics such as creating builds, queuing builds, etc. should be well covered there.
Overview:
This post will go over some simple but effective ways to setup common initialization and cleanup for Unit Tests on a larger scale than [ClassInitialize] and [TestInitialize] methods can provide for.
First, to establish a common starting point I’ll go over some of the basics, if you’re already familiar with using the ClassInitialize/Cleanup and TestInitialize/Cleanup attributes in unit tests you may wish to skip this next part.
Background:
When you have some setup and cleanup code that needs to run for several unit tests typically you would put them in the same Test Class and use a combination of Class Initialize/Cleanup and Test Initialize/Cleanup methods.
A simple start would be something like this:
[TestClass]
public class TestClass1
{
[TestInitialize]
public void TestInit()
{
Console.WriteLine("TestClass1.TestInit()");
}
[TestMethod]
public void TestMethod1()
{
Console.WriteLine("TestClass1.TestMethod1()");
}
[TestMethod]
public void TestMethod2()
{
Console.WriteLine("TestClass1.TestMethod2()");
}
[TestCleanup]
public void TestCleanup()
{
Console.WriteLine("TestClass1.TestCleanup()");
}
}
If you run both tests here the output will be:
TestClass1.TestInit()
TestClass1.TestMethod1()
TestClass1.TestCleanup()
and
TestClass1.TestInit()
TestClass1.TestMethod2()
TestClass1.TestCleanup()
The methods marked with the attributes [TestInitialize] and [TestCleanup] run before and after each test in that class. If instead you’d like to only run the initialization code once before all tests (not each individual test) you could use [ClassInitialize] and [ClassCleanup] instead, or you can also use them in combination.
Going Beyond Local ClassInitialize and TestInitialize
What can I do if I have a large project, with dozens or even hundreds of unit test methods spread across several classes and you want to share some setup or cleanup code between those tests?
One approach would be to create some initialize and cleanup helper methods in a separate class and call those methods from each of your individual test classes initialize and cleanup methods.
Another approach, the one I personally prefer, is to create a base class for your test classes. For example:
[TestClass]
public class TestBase
{
[TestInitialize]
public void BaseTestInit()
{
Log.AppendLine("TestBase.BaseTestInit()");
}
[TestCleanup]
public void BaseTestCleanup()
{
Console.WriteLine(Log.ToString());
}
public static StringBuilder Log
{
get
{
if (s_log == null)
{
s_log = new StringBuilder();
}
return s_log;
}
}
static StringBuilder s_log;
}
[TestClass]
public class TestClass1 : TestBase
{
[ClassInitialize]
public static void ClassInit(TestContext testContext)
{
Log.AppendLine("TestClass1.ClassInit()");
}
[TestInitialize]
public void TestInit()
{
Log.AppendLine("TestClass1.TestInit()");
}
[TestMethod]
public void TestMethod1()
{
Log.AppendLine("TestClass1.TestMethod1()");
}
}
Notice that the base class “TestBase” is also using the [TestClass] attribute, although we won’t be putting any test methods in this class. This allows the use of the [TestInitialize] and [TestCleanup] attributes within our base class. If you ran the tests in TestClass1 you would see the following output:
TestClass1.ClassInit()
TestBase.BaseTestInit()
TestClass1.TestInit()
TestClass1.TestMethod1()
The [ClassInitialize] will always run first, it’s static and will be invoked by the unit test engine before instantiating the test class. Next we see that the Test Initializer in the base class is called, followed by the Test Initializer in the test class itself, and lastly the test method is executed.
How can I create an initialization method that will run before any class initialization methods in my test project?
Building on the common base class approach described above you could simply add a static constructor to your base class and either perform the initialization there or call the method that will perform the desired initialization. The resulting base class might look like this:
[TestClass]
public class TestBase
{
static TestBase()
{
s_log = new StringBuilder();
Log.AppendLine("TestBase.ctor()");
}
[TestInitialize]
public void BaseTestInit()
{
Log.AppendLine("TestBase.BaseTestInit()");
}
[TestCleanup]
public void BaseTestCleanup()
{
Console.WriteLine(Log.ToString());
}
public static StringBuilder Log
{
get { return s_log; }
}
static StringBuilder s_log;
}
Will the same approach that was used for [TestInitialize] work with [ClassInitialize] in a base class?
Not exactly, if you create a [ClassInitialize] attributed method in the base class it won’t ever get called unless you explicitly call it at the beginning of your derived test classes ClassInitialize method; which of course is nowhere near as nice as the above approach.
If you really wanted this functionality you could hook the method calls using reflection and set things up that way, but that’s beyond the scope of this post.
Setting up the relationship in the reverse would be much easier, but is of questionable value. By reverse order I mean that it would be easier to create a method that resided in the base class and was called once per test class but it would be called after the derived classes ClassInitialize method.
The only viable option that I can come up for achieving an inheritable class initialization approach would be to ditch the ClassInitialize mechanism altogether and go back to good old fashioned class constructors.
Example:
[TestClass]
public class TestBase
{
static TestBase()
{
s_log = new StringBuilder();
Log.AppendLine("TestBase.ctor()");
}
public TestBase()
{
Log.AppendLine("TestBase.TestBase() <-- acts as ClassInitialize in base");
}
[TestInitialize]
public void BaseTestInit()
{
Log.AppendLine("TestBase.BaseTestInit()");
}
[TestCleanup]
public void BaseTestCleanup()
{
Console.WriteLine(Log.ToString());
}
public static StringBuilder Log
{
get { return s_log; }
}
static StringBuilder s_log;
}
[TestClass]
public class TestClass1 : TestBase
{
public TestClass1() //Replaces ClassInitialize method
{
Log.AppendLine("TestClass1.TestClass1() <-- acts as ClassInitialize in derived");
}
[TestInitialize]
public void TestInit()
{
Log.AppendLine("TestClass1.TestInit()");
}
[TestMethod]
public void TestMethod1()
{
Log.AppendLine("TestClass1.TestMethod1()");
}
}
Running TestMethod1 in the derived class produces the following output:
TestBase.ctor()
TestBase.TestBase() <-- acts as ClassInitialize in base
TestClass1.TestClass1() <-- acts as ClassInitialize in derived
TestBase.BaseTestInit()
TestClass1.TestInit()
TestClass1.TestMethod1()
