Dennis Stone's Blog

TCM.exe – A General Introduction

Dennis Stone — Mon, 09 Nov 2009 16:09:10 GMT

Along with the new tools we’ve added in VSTS 2010 for testers comes the TCM.exe command-line application. This command-line isn’t intended to be an alternative for everything you can do in MTLM but rather to fill some gaps and enable some of the less common scenarios that still may be important for some testers.

You can get a general overview of TCM.exe over on MSDN in this article.

Some examples of things you can do with TCM.exe include:

Create an automated test run that contains test points across multiple test suites.
Create an automated test run without specifying a Team Build (you can specify the build directory where automated test binaries are picked up directly).
Publish results from a trx (MSTest / VS run results file) against your test plan.
Import VSTT Unit Tests (or even other VSTT test types) from your test binary into your team project. Creates a Test Case work item from each Unit Test (given some user specified constraints on what to import) and creates the association with the automated unit test for you. You can also sync this set of imported tests with a specified Test Suite.

Test Impact post by Daryush

Dennis Stone — Mon, 09 Nov 2009 15:35:52 GMT

There’s a nice post on Test Impact from within Visual Studio (geared towards Developers rather than testers) here.

Check it out if you’re interested in learning about a different side of Test Impact!

Test Impact Walk-through

Dennis Stone — Mon, 29 Jun 2009 21:04:00 GMT

One of the key areas of focus for Visual Studio Team System 2010 is to enhance the experience for testers. There’s already lots of overview information on many of the new features out there so I’m going to dive right in to one of my favorite new features.

In short, the Test Impact Analysis feature gives you a list of recommended tests based on the changes that were made in a build. As you run your tests through Microsoft Test and Lab Manager (code named Camano) you will have the option to run the Test Impact Data Collector which is one of the many built-in Data Collectors available while running tests. This particular data collector will associate the code paths you execute in your application with the test you are running.

When Team Build generates a new build it will analyze what code has been changed since the last build was done and then provide you with a list of the impacted tests. These are the tests that touch the same pieces of code that were changed since the last build.

For a more thorough explanation of the Test Impact Data Collector read this blog post by Amit Chatterjee.

The first thing you need is an Application Under Test, for this walk-through I’ll use a simple calculator application. I’ll have a C# Class Library with some simple Add, Subtract, Multiply, and Divide functions and a C# Windows Forms project that will provide some input fields for the numbers and some buttons that call the library function. I’ll skip the creation of this sample application, you an use the attached sample application, mock up something similar or follow along with your own test application.

Add your test application to Source Control
Create a new Build Definition for your application
1. Make sure “Analyze Test Impacts” in the Process tab is set to “True”:
Queue a new build
Start “Microsoft Test and Lab Manager” (aka Camano)
1. If the dialog below comes up click “Set context”:
2. On the “Set Plan Context” page click “New”
3. Enter a name for your Test Plan and click Save & Close
4. Go to Testing Center –> Plan –> Contents
5. Click “New” in the toolbar to add a new Test case
6. Add a Title for the Test case, mine is “Add – Basic”, I’ll repeat these steps and add “Subtract – Basic”, “Multiply – Basic”, and “Divide – Basic” as well. Here’s what my Test Plan Contents will look like when I’ve done this.
7. Now it’s time to switch to the Lab Center so we can set up our Test Settings
8. Switch to the Test Settings tab and click “New”
9. Enter a Name and click “Next”
10. To keep things simple uncheck all collectors except the “Test Impact Collector” and click “Save and Close”
11. Go back to the Testing Center.
12. Switch to the Track tab and go to the “Assign Build” activity
13. The build you created earlier will be in the “Available builds” list, select it and click “Assign to plan”.
14. Now it’s time to run our tests. Switch to Testing Center –> Test –> Run Tests
15. Select all of your tests and click “Run with Options”
16. In the Run Options dialog select your Test Settings and verify that your newest build is selected. Click Run.
17. Test Runner will now appear
18. Click Start test and record, then launch your Application Under Test. This ordering is important, you must start the test before launching the test application or the impact data will not be collected.
19. Perform each test now. For my Add test I’ll set 2 numbers in my text boxes and click Add, for Subtract test I’ll set 2 numbers and click Subtract, etc. etc. Run your test then click “End Test”, then start the next test. There’s no need to re-launch your application between every test. When you’re done click “Save and Close” to return to Manual Test and Lab Manager (MTLM). It should look something like this:
20. To verify the Test Impact Collector worked correctly you can open up one of your test results and look at it’s attachments, there should be an impact xml file listed there. If there isn’t, re-check your steps above.
21. Now, switch to Visual Studio and make a change to your application under test. I’m going to make a change to my “Add” method and check it in.
22. Queue a new build and wait for it to complete.
23. Switch back to MTLM
24. Go to Testing Center –> Track –> Assign Build and assign your new build to your plan. (See Step #12 if you need help)
25. Now switch to Testing Center –> Track –> Recommended Tests
26. Notice the “Add – Basic” test is now listed.

That’s it for this post, I look forward to any questions or comments you may have. If you have any questions on some of the steps above try looking them up on the MSDN site. Topics such as creating builds, queuing builds, etc. should be well covered there.

The Test Management API - Getting Started

Dennis Stone — Fri, 12 Jun 2009 21:54:00 GMT

Overview

This post kicks off a series that will explore the Test Management API’s. Using these API’s you can accomplish many tasks that may not be possible via the UI (Microsoft Test and Lab Manager (MTLM) code named Camano). Perhaps you have test assets in another system for example, with the API you could create your own solution that fits your individual needs for migrating data between systems. Or maybe you have a custom tool in mind for your team that will need to be able to examine the various objects you’ve stored in the test case management component of TFS, such as your Test Plans, Test Cases, Test Runs, Results, etc.

Whatever your reason, using the Test Management API’s will almost always have a common starting point, you first need to establish a connection to the Team Foundation Server and that’s what I’ll cover here.

The Code

You’ll need to add references to the following assemblies to your project:

Microsoft.TeamFoundation.Client.dll
Microsoft.TeamFoundation.TestManagement.Client.dll
Microsoft.TeamFoundation.TestManagement.Common.dll
Microsoft.TeamFoundation.WorkItemTracking.Client.dll

These will all be GAC’d on a machine you’ve installed VSTS on.

Here’s a quick and easy sample C# console application that connects to the server and creates a very basic test case.

using System;

using System.Collections.Generic;

using System.Linq;

using System.Text;

using Microsoft.TeamFoundation.Client;

using Microsoft.TeamFoundation.TestManagement.Client;

using Microsoft.TeamFoundation.WorkItemTracking.Client;

namespace BlogSamples

{

class Program

{

static void Main(string[] args)

{

string serverurl = "http://localhost:8080/tfs";

string project = "Beta1";

ITestManagementTeamProject proj = GetProject(serverurl, project);

ITestCase tc = proj.TestCases.Create();

tc.Title = "Test";

tc.Save();

Console.WriteLine("TC: {0}", tc.Id);

}

static ITestManagementTeamProject GetProject(string serverUrl,

string project)

{

TeamFoundationServer tfs = new TeamFoundationServer(serverUrl);

ITestManagementService tms = tfs.GetService<ITestManagementService>();

return tms.GetTeamProject(project);

}

Next Time

In my next post I’ll start going over the core objects in the Test Management object model in more detail.

Using a Base Class for your Unit Test Classes

Dennis Stone — Fri, 16 May 2008 23:45:00 GMT

Overview:

This post will go over some simple but effective ways to setup common initialization and cleanup for Unit Tests on a larger scale than [ClassInitialize] and [TestInitialize] methods can provide for.

First, to establish a common starting point I’ll go over some of the basics, if you’re already familiar with using the ClassInitialize/Cleanup and TestInitialize/Cleanup attributes in unit tests you may wish to skip this next part.

Background:

When you have some setup and cleanup code that needs to run for several unit tests typically you would put them in the same Test Class and use a combination of Class Initialize/Cleanup and Test Initialize/Cleanup methods.

A simple start would be something like this:

[TestClass]

public class TestClass1

{

[TestInitialize]

public void TestInit()

{

Console.WriteLine("TestClass1.TestInit()");

}

[TestMethod]

public void TestMethod1()

{

Console.WriteLine("TestClass1.TestMethod1()");

}

[TestMethod]

public void TestMethod2()

{

Console.WriteLine("TestClass1.TestMethod2()");

}

[TestCleanup]

public void TestCleanup()

{

Console.WriteLine("TestClass1.TestCleanup()");

}

If you run both tests here the output will be:

TestClass1.TestInit()

TestClass1.TestMethod1()

TestClass1.TestCleanup()

and

TestClass1.TestInit()

TestClass1.TestMethod2()

TestClass1.TestCleanup()

The methods marked with the attributes [TestInitialize] and [TestCleanup] run before and after each test in that class. If instead you’d like to only run the initialization code once before all tests (not each individual test) you could use [ClassInitialize] and [ClassCleanup] instead, or you can also use them in combination.

Going Beyond Local ClassInitialize and TestInitialize

What can I do if I have a large project, with dozens or even hundreds of unit test methods spread across several classes and you want to share some setup or cleanup code between those tests?

One approach would be to create some initialize and cleanup helper methods in a separate class and call those methods from each of your individual test classes initialize and cleanup methods.

Another approach, the one I personally prefer, is to create a base class for your test classes. For example:

[TestClass]

public class TestBase

{

[TestInitialize]

public void BaseTestInit()

{

Log.AppendLine("TestBase.BaseTestInit()");

}

[TestCleanup]

public void BaseTestCleanup()

{

Console.WriteLine(Log.ToString());

}

public static StringBuilder Log

{

get

{

if (s_log == null)

{

s_log = new StringBuilder();

}

return s_log;

}

static StringBuilder s_log;

}

[TestClass]

public class TestClass1 : TestBase

{

[ClassInitialize]

public static void ClassInit(TestContext testContext)

{

Log.AppendLine("TestClass1.ClassInit()");

}

[TestInitialize]

public void TestInit()

{

Log.AppendLine("TestClass1.TestInit()");

}

[TestMethod]

public void TestMethod1()

{

Log.AppendLine("TestClass1.TestMethod1()");

}

Notice that the base class “TestBase” is also using the [TestClass] attribute, although we won’t be putting any test methods in this class. This allows the use of the [TestInitialize] and [TestCleanup] attributes within our base class. If you ran the tests in TestClass1 you would see the following output:

TestClass1.ClassInit()

TestBase.BaseTestInit()

TestClass1.TestInit()

TestClass1.TestMethod1()

The [ClassInitialize] will always run first, it’s static and will be invoked by the unit test engine before instantiating the test class. Next we see that the Test Initializer in the base class is called, followed by the Test Initializer in the test class itself, and lastly the test method is executed.

How can I create an initialization method that will run before any class initialization methods in my test project?

Building on the common base class approach described above you could simply add a static constructor to your base class and either perform the initialization there or call the method that will perform the desired initialization. The resulting base class might look like this:

[TestClass]

public class TestBase

{

static TestBase()

{

s_log = new StringBuilder();

Log.AppendLine("TestBase.ctor()");

}

[TestInitialize]

public void BaseTestInit()

{

Log.AppendLine("TestBase.BaseTestInit()");

}

[TestCleanup]

public void BaseTestCleanup()

{

Console.WriteLine(Log.ToString());

}

public static StringBuilder Log

{

get { return s_log; }

}

static StringBuilder s_log;

}

Will the same approach that was used for [TestInitialize] work with [ClassInitialize] in a base class?

Not exactly, if you create a [ClassInitialize] attributed method in the base class it won’t ever get called unless you explicitly call it at the beginning of your derived test classes ClassInitialize method; which of course is nowhere near as nice as the above approach.

If you really wanted this functionality you could hook the method calls using reflection and set things up that way, but that’s beyond the scope of this post.

Setting up the relationship in the reverse would be much easier, but is of questionable value. By reverse order I mean that it would be easier to create a method that resided in the base class and was called once per test class but it would be called after the derived classes ClassInitialize method.

The only viable option that I can come up for achieving an inheritable class initialization approach would be to ditch the ClassInitialize mechanism altogether and go back to good old fashioned class constructors.

Example:

[TestClass]

public class TestBase

{

static TestBase()

{

s_log = new StringBuilder();

Log.AppendLine("TestBase.ctor()");

}

public TestBase()

{

Log.AppendLine("TestBase.TestBase() <-- acts as ClassInitialize in base");

}

[TestInitialize]

public void BaseTestInit()

{

Log.AppendLine("TestBase.BaseTestInit()");

}

[TestCleanup]

public void BaseTestCleanup()

{

Console.WriteLine(Log.ToString());

}

public static StringBuilder Log

{

get { return s_log; }

}

static StringBuilder s_log;

}

[TestClass]

public class TestClass1 : TestBase

{

public TestClass1() //Replaces ClassInitialize method

{

Log.AppendLine("TestClass1.TestClass1() <-- acts as ClassInitialize in derived");

}

[TestInitialize]

public void TestInit()

{

Log.AppendLine("TestClass1.TestInit()");

}

[TestMethod]

public void TestMethod1()

{

Log.AppendLine("TestClass1.TestMethod1()");

}

Running TestMethod1 in the derived class produces the following output:

TestBase.ctor()

TestBase.TestBase() <-- acts as ClassInitialize in base

TestClass1.TestClass1() <-- acts as ClassInitialize in derived

TestBase.BaseTestInit()

TestClass1.TestInit()

TestClass1.TestMethod1()

Orcas WebTest API Enhancements

Dennis Stone — Wed, 27 Jun 2007 19:00:00 GMT

Several new API Enhancements have been made for WebTest's in the Orcas release of Visual Studio Team System. Here are some links to examples:

DeclarativeWebTest & DeclarativeWebTestSerializer: http://blogs.msdn.com/densto/pages/declarativewebtest-declarativewebtestserializer.aspx

WebTestRequest.DependentRequests Collection: http://blogs.msdn.com/densto/pages/webtestrequest-dependentrequests-collection.aspx

WebTest.Outcome & WebTestRequest.Outcome: http://blogs.msdn.com/densto/pages/webtest-outcome-webtestrequest-outcome.aspx

To find out about more of the new additions to Orcas check out the links in this post: http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=1177487&SiteID=1

How to Databind Web Test Properties that don't have the Databinding Dropdown UI in the Property Grid

Dennis Stone — Thu, 03 May 2007 19:30:00 GMT

Most of the time when you want to bind a web test, request, validation rule, or extraction rule property to a datasource you just click on the value in the properties grid and there is a dropdown that provides UI for selecting the datasource to bind to. Sometimes you'll find properties that don't provide this UI but you'll want to databind the value anyways. In most cases this it's possible, here are an easy set of steps to do this:

Add the datasource that contains the dynamic values you want to bind to the property.
Add a Querystring parameter to one of the requests in your Web Test, it doesn’t matter which one, we’re going to delete this when we’re done.
Bind the Querystring parameter to the datasource in the way you’d like to bind the property value that doesn’t have this binding UI.
Look at how the querystring value looks in the editor after binding, it’ll be something like: {{datasourceName.tableName.fieldName}}
Type that exact text, including the surrounding {{ }} into the value’s text field where you want the real binding.
Now you can delete the querystring parameter you added.

That’s it, you should be able to databind just about any property in the grid that doesn’t have the databinding UI by default using this method.

Dynamic QueryString Correlation: Custom Extraction Rules and WebTest Plug-in

Dennis Stone — Mon, 19 Mar 2007 16:29:00 GMT

Here is an article with some custom rules and plug-ins that you can use to apply dynamic querystring correlation to your web tests. These can be useful if you are testing a site that uses querystring parameters to track dynamic values (such as session id), for example, a SQL Reporting Services site.

http://blogs.msdn.com/densto/pages/dynamic-querystring-correlation-custom-extraction-rules-and-webtest-plug-in.aspx