Dennis Stone's Blog

Using Retention Policy to Delete Builds but Keep Test Results

2010-07-16T15:10:44Z

There’s a good post over at http://blog.nfocus.co.uk/2010/07/oi-what-happened-to-my-test-results.html by Danny Crone that goes over a scenario where a Retention Policy was specified on a Build Definition and testers were unpleasantly surprised to find their Test Results history disappearing as older builds got cleaned up. Check out the post to see a good walk-through on how to avoid this.

How to keep your Test Plans / Suites in sync with your Unit Test assemblies

2010-03-12T19:40:08Z

In my last post I gave an overview of what automated testing looks like in VS / TFS 2010 with the new testing features. In 2010 we’ve added Test Plans which contain Test Suites which you can think of as a folder hierarchy to separate and organize your tests within. Test Suites contain Test Cases (well, Test Points really, you can see my previous post here if you’re interested in the distinction) which you can run from Microsoft Test Manager (MTM.exe). Test Cases can either be Manual or Automated, for a Test Case to be automated you need to associate the Test Case work item with one of the VSTS 2010 automated test types which includes Unit Tests, Web Performance Tests, Load Tests, Coded UI Tests, etc. The most common scenario here is probably to associate your Unit Tests with Test Cases.

To manually create that association you perform the following steps:

Open your Test Project and Connect to your TFS 2010 Server
Create or open an existing Test Case
Switch to the Associated Automation tab
Click the … button next to the Automated test name textbox
Choose the test in the list you want to associate the work item Test Case with, in this simple example I only have one so I’ll go ahead and choose that and hit OK, then you’ll see the following data filled in.
Save the Test Case.

Now when you add this Test Case to a Test Suite in Camano, associate your Test Plan with a build, and then Run the Test MTM will go to the corresponding build share to find the test assembly and run the associated unit test.

As you can see there’s a point where things quickly become unmanageable if you’re forced to manually manage these associations. My teams Unit Test assembly contains approximately 650 Unit Tests that we run through Microsoft Test Manager every night. We’re pretty well locked down now of course but there was a long time during the product cycle during which we were adding new tests and removing deprecated tests quite regularly. It would have been very time consuming to keep our MTM Test Suites in sync with these changes to our Unit Test assembly (and that’s not even considering the amount of time it would have taken to manually create 650 Test Case work items and then associate each one with the corresponding Unit Test just to get started). We saved a lot of time using the TCM.exe command line tool that is included with the product to do this for us so I wanted to dive into that process here.

First, here’s a tip you may find useful if you are going to be using TCM.exe against the same server / project frequently.

Navigate to your VS Install Root\Common7\IDE
Open TCM.exe.config
Set the TeamFoundationServer and TeamProject app settings there, so you don’t have to specify them on every command you run.

Here’s a set of fake Unit Tests I’ll use to demonstrate some of the features of TCM.exe:

        [TestMethod]
        [TestCategory("BVT")]
        [Priority(1)]
        public void Apple()
        {
            //Test something
        }

        [TestMethod]
        [TestCategory("BVT")]
        [TestCategory("NotLocFriendly")]
        [Priority(1)]
        public void Banana()
        {
            //Test something
        }

        [TestMethod]
        [TestCategory("BVT")]
        [Priority(1)]
        public void Pear()
        {
            //Test something
        }

        [TestMethod]
        [TestCategory("NotLocFriendly")]
        [Priority(2)]
        public void Kiwi()
        {
            //Test something
        }

        [TestMethod]
        [Priority(1)]
        public void Lemon()
        {
            //Test something
        }

        [TestMethod]
        [TestCategory("BVT")]
        [Priority(1)]
        public void Orange()
        {
            //Test something
        }

This set of tests includes tests across 3 different Priority levels and 2 Test Categories (including some tests not in a Test Category). If you’re not familiar with the [TestCategory] attribute, I believe it’s new to this release. It’s a nice way to mark your unit tests in such a way that you can query them from MSTest.exe so you can easily run certain subsets of your tests without having to list them all out individually. We also support this in TCM.exe for the testcase import commands.

In my Test Plan I have a Suite Hierarchy like this:

Here goes first let’s check out the command we’ll use. As you can see in the help output below we can import our tests with as little input as simply pointing to the assembly with the “/storage” option. That would import all of the Test Methods in the entire assembly though and it would only create the associated work items rather than actually add them to our suites.
We’ll want to use the /syncsuite and /category options. That runs us into our first issue though, we don’t know our suite id’s, and they aren’t displayed in the MTM UI anywhere. We can get the Test Plan Id from the UI by going to the Test Plan Properties page. TCM.exe includes the ability to query this information from the server though, since I already know my Test Plans Id from the UI, I can do:
Now I have the Suite Id’s I need for the /syncsuite option and can proceed.
And we can see that Orange, Pear, Apple, and Banana were Added, here’s what our Test Suite now looks like:
What happens when I run the command again? We don’t create an additional set of Test Case work items, we just update the existing ones.

That’s it! I’ll skip going through populating the other Test Suites this way, hopefully you get the idea from the example above. Instead, let’s take a quick look at what the Test Case that the import command created looks like.

The “Assigned To” was set to the account I’m running the import as, if you specify the “Owner” attribute on your Unit Test it’ll use that value instead. Priority was set to the value on the Unit Test attribute also. You can also see that the Associated Automation info was set for us just as it was in the manual example at the beginning of this post.

That’s it for this post, hope you find it useful.

Automated Testing in VSTS / TFS 2010

2010-03-05T09:31:00Z

I thought I’d make a brief post giving an overview of what automated testing looks like in our 2010 release. While it’s true that in this release we focused primarily on nailing the experience for the manual tester there is quite a bit of carryover from those features that also apply to automated testing, and as we do quite a bit of automated testing ourselves and dogfood our own product we did end up working on the automated testing experience as well despite that heavy focus on the manual test side of things.

Since the 2005 release we’ve had several types of tests that you could create in a Visual Studio Test Project, they included Unit Tests, Web Tests, Load Tests, etc. In those versions you could run your tests locally through Visual Studio or MSTest, or you could run them against a remote controller and agent from either of those clients. When your tests were complete you could publish them against a TFS Server / Project / Build and then use the various reporting mechanisms in the product to view the test results. They would also show up on the Build Summary Report.

In the 2010 release we’ve added a ton of new test features. We now have first class support for Test Case work items, and tooling for creating Test Plans, organizing your test cases into test suites, tracking which test configurations you’ve run against, etc. There are tons of blog posts covering these new features so I won’t go into a lot of detail here on them, instead I’ll focus on how those automated test types fit into this new landscape of test objects.

Each Test Case can have associated automation, this creates a reference from the Test Case work item to the actual automated test, be it a Unit Test, Web Performance Test, Coded UI Test, or whatever. When you add that Test Case to one of the Test Suites in your Test Plan and run it, it will look for the referenced test on the build share for whatever build you’re running against. When you create this association between your Test Case work items and your VSTT tests you can then track your test progress within the context of your Test Plan and/or Test Suites and publishing / saving the results is all handled for you without the need for a separate publish operation.

If you aren’t using Team Build to generate your builds yet then you’ll need to use “Fake Builds” that your test plan can reference and use another process to copy your test assemblies / files to the drop location you specify on the fake build, that way when you run your automated tests through MTM it will know where to find the associated test files. See my earlier post Creating Fake Builds in TFS 2010 for more information on that. Alternatively you can use tcm.exe to run your tests and point to any share containing your test assemblies / files.

In the most basic scenario you would create an Environment (physical in this case, but there are also virtual environments) that specifies a Test Controller to run your tests against. When you kick off an automated test run from MTM you’ll have specified the environment to run tests on as well as the build you are running against. The test assemblies / files from the build you specified will be sent over to the controller & agent much like they were in the previous versions, they’ll be run there, and the results will be posted back to the new system. From within MTM you can then drill into failed tests and even open up the corresponding trx’s into Visual Studio.

All of this allows you to use all of the new test planning, organization, execution, and tracking features that we’ve added in 2010 with the automated test types that you’ve been producing throughout our earlier releases. TCM.exe provides some features that allow easier importing of these tests so that you don’t have to generate Test Case work items for each individual test case manually one by one. It also allows you to easily keep a Test Suite in sync with a unit test assembly for example without having to manually account for added or removed unit tests as you move through the product cycle.

For more details on this refer to this MSDN article.

The Test Management API – Part 2: Creating & Modifying Test Plans

2010-03-04T02:37:12Z

In the previous Getting Started post for this series I covered the basics of creating a connection to the server and getting an instance of ITestManagementTeamProject which is at the core of just about everything you may want to do with the Test Management objects.

This time I’d like to show how to create and modify a Test Plan, add a new Test Suite to it, and then add Test Cases to your new suite. Then I’ll query it all back and output some basic info about what we’ve created.

I start with the same basic framework I laid out in the Getting Started post, and also reference the post I previously did on Test Points so you may want to review those before proceeding. Note that in several places I used WIQL for querying test objects from the server, for more on that Duat Le has put together a great post here that goes into depth on that topic.

One last note, beyond the references mentioned in the Getting Started post you need to add a reference to WindowsBase. Now, on to the code.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

using Microsoft.TeamFoundation.Client;
using Microsoft.TeamFoundation.TestManagement.Client;
using Microsoft.TeamFoundation.WorkItemTracking.Client;

namespace BlogSamples
{
    class Program
    {
        static void Main(string[] args)
        {
            string serverurl = "http://localhost:8080/tfs";
            string project = "test";
            ITestManagementTeamProject proj = GetProject(serverurl, project);

            // List all Test Plans
            foreach (ITestPlan p in proj.TestPlans.Query("Select * From TestPlan"))
            {
                Console.WriteLine("Plan - {0} : {1}", p.Id, p.Name);
            }

            // Get a Test Plan by its Id
            int myPlansId = 1;
            
            ITestPlan foundPlan = proj.TestPlans.Find(myPlansId);
            Console.WriteLine("Got Plan {0} with Id {1}",
                foundPlan.Name, foundPlan.Id);

            // Let's create a new Test Plan from scratch
            // Only the Name property is required to be set before
            // you can save, but we'll set a few extras here as well.
            ITestPlan plan = proj.TestPlans.Create();
            plan.Name = "My New Test Plan";
            plan.StartDate = DateTime.Now;
            plan.EndDate = DateTime.Now.AddMonths(2);
            plan.Save();

            Console.WriteLine("Created new Plan \"{0}\" with Id: {1}",
                plan.Name, plan.Id);

            // Next let's add a new Test Suite to our plan
            // First, create a static suite (I'll cover Dynamic Suites
            // in a future post)
            IStaticTestSuite newSuite = proj.TestSuites.CreateStatic();
            newSuite.Title = "Acceptance Tests";
            
            plan.RootSuite.Entries.Add(newSuite);
            plan.Save();

            // Let's find the default configuration that was created when
            // we created our Team Project.  We'll need this to add some
            // tests to our new suite (or we could create a new one, but 
            // I'll keep things simple and just get the default one for now)
            ITestConfiguration defaultConfig = null;

            foreach (ITestConfiguration config in proj.TestConfigurations.Query(
                "Select * from TestConfiguration"))
            {
                defaultConfig = config;
                break;
            }

            // I would typically check that defaultConfig is not null here
            // but for brevity sake will skip that.

            // Next we'll create a Testcase to add to our Test Suite
            ITestCase tc = proj.TestCases.Create();
            tc.Title = "Verify X when doing Y";
            tc.Save();

            // Now let's add it to your suite, we still have our reference
            // to the static suite around so we'll use that
            // First we need an "IdAndName" object for our config
            IdAndName defaultConfigIdAndName = new IdAndName(
                defaultConfig.Id, defaultConfig.Name);
            
            // Now we'll set the default configs for our suite
            newSuite.SetDefaultConfigurations(new IdAndName[] {
                defaultConfigIdAndName});

            // Now let's add that testcase
            newSuite.Entries.Add(tc);

            // Now let's save the plan (don't need to explicitly
            // save test suites).
            plan.Save();
            
            // And finally, let's take a look at what we've created.
            Console.WriteLine("Test Plan: {0}", plan.Name);

            foreach (ITestSuiteBase suite in plan.RootSuite.SubSuites)
            {
                Console.WriteLine("\tTest Suite: {0}", suite.Title);

                IStaticTestSuite staticSuite = suite as IStaticTestSuite;

                // Let's Query the Test Points (see my previous post on what
                // Test Points are)
                foreach (ITestPoint point in plan.QueryTestPoints(
                    string.Format("SELECT * FROM TestPoint WHERE SuiteId = {0}",
                    suite.Id)))
                {
                    Console.WriteLine("\t\tTest Point for Testcase \"{0}\" against config \"{1}\"",
                        point.TestCaseWorkItem.Title, point.ConfigurationName);                        
                }
            }
        }


        static ITestManagementTeamProject GetProject(string serverUrl,
            string project)
        {
            TfsTeamProjectCollection tfs = new TfsTeamProjectCollection(TfsTeamProjectCollection.GetFullyQualifiedUriForName(serverUrl));
            ITestManagementService tms = tfs.GetService<ITestManagementService>();

            return tms.GetTeamProject(project);
        }
    }
}

Test Impact Demo Video posted by Danny Crone

2010-03-03T15:20:54Z

There’s a nice video demo featuring Test Impact you may find interesting here:

http://blog.nfocus.co.uk/2010/03/vs2010-test-impact-one-giant-leap-for.html

The demo is a little under 20 minutes and actually touches on quite a few of the integration features in VSTS / TFS 2010 including the Intellitrace Data Collector, Video Recording, and the experience of logging bugs directly from the Test Runner and all the automatically filled in bug data that you get with that.

Creating Fake Builds in TFS 2010

2010-03-02T20:26:18Z

Sometimes it can be useful to create “fake” builds in Team Foundation Server in order to successfully use other ALM features in the product. If you aren’t using Team Build to generate the build drops for your application but you’d like to have a way to track test runs against builds in TFS one way would be to generate a corresponding “Fake Build” in TFS and publish your tests against that. With all the added Test Management features in 2010 there are even more ways to associate your testing with specific builds so if you’re not using Team Build to generate your builds yet, the ability to create Fake Builds in TFS that correspond to your real builds may come in handy.

Here is a post by Jason Prickett that shows the updated code for 2010 (it also links to the old post which showed how to do it in 2008):

http://blogs.msdn.com/jpricket/archive/2010/02/23/creating-fake-builds-in-tfs-build-2010.aspx

The Test Management API – Getting Started (Updated for RC Release)

2010-03-02T20:09:02Z

Overview

This post kicks off a series that will explore the Test Management API’s. Using these API’s you can accomplish many tasks that may not be possible via the UI, Microsoft Test Manager (MTM). Perhaps you have test assets in another system for example, with the API you could create your own solution that fits your individual needs for migrating data between systems. Or maybe you have a custom tool in mind for your team that will need to be able to examine the various objects you’ve stored in the test case management component of TFS, such as your Test Plans, Test Cases, Test Runs, Results, etc.

Whatever your reason, using the Test Management API’s will almost always have a common starting point, you first need to establish a connection to the Team Foundation Server and that’s what I’ll cover here.

The Code

You’ll need to add references to the following assemblies to your project:

Microsoft.TeamFoundation.Client.dll
Microsoft.TeamFoundation.TestManagement.Client.dll
Microsoft.TeamFoundation.TestManagement.Common.dll
Microsoft.TeamFoundation.WorkItemTracking.Client.dll

These will all be GAC’d on a machine you’ve installed VSTS on.

Here’s a quick and easy sample C# console application that connects to the server and creates a very basic test case.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

using Microsoft.TeamFoundation.Client;
using Microsoft.TeamFoundation.TestManagement.Client;
using Microsoft.TeamFoundation.WorkItemTracking.Client;

namespace BlogSamples
{
    class Program
    {
        static void Main(string[] args)
        {
            string serverurl = "http://localhost:8080/tfs";
            string project = "Beta1";
            ITestManagementTeamProject proj = GetProject(serverurl, project);

            ITestCase tc = proj.TestCases.Create();
            tc.Title = "Test";
            tc.Save();

            Console.WriteLine("TC: {0}", tc.Id);
        }


        static ITestManagementTeamProject GetProject(string serverUrl,
            string project)
        {
            TfsTeamProjectCollection tfs = new TfsTeamProjectCollection(TfsTeamProjectCollection.GetFullQualifiedUriForName(serverUrl));
            ITestManagementService tms = tfs.GetService<ITestManagementService>();

            return tms.GetTeamProject(project);
        }
    }
}

Post on Querying Test Management Objects by Duat Le

2010-03-02T18:05:52Z

Duat Le, a Developer on our team, has put up a very nice post that goes over how to query Test Objects using WIQL (Work Item Query Language). This information will be very useful if you ever need to use our query syntax in any code you write against our Test Management API’s.

http://blogs.msdn.com/duat_le/archive/2010/02/25/wiql-for-test.aspx

Test Management Terminology: What is a Test Point?

2009-11-19T16:15:45Z

I was just getting started on what I intended to be my next post on this blog when I realized that the topic in that post would be much easier to discuss and understand if I first wrote about Test Points. A Test Point is an object in our TCM Object Model that isn’t really explained or directly surfaced in the UI anywhere, the reason is that the user doesn’t necessarily need to know they’re working with a “Test Point” when they are.

A Test Point represents a Test Case to be run against a Test Configuration in a Test Suite.

In MTLM, if you go to the Testing Center –> Plan tab –> Contents, you see the list of Test Case’s in each suite. One of the columns in the grid is “Configurations” which shows a count of how many Test Configurations that Test Case is tested against in that suite. When you go to Testing Center –> Test tab –> Run Tests and click on the same suite you will see an entry in the grid there for each Test Case / Test Configuration combination. These are Test Points.

Knowing this terminology shouldn’t fundamentally change the way you use the Test Management product, but it should help make future posts easier for me :)

TCM.exe – A General Introduction

2009-11-09T16:09:10Z

Along with the new tools we’ve added in VSTS 2010 for testers comes the TCM.exe command-line application. This command-line isn’t intended to be an alternative for everything you can do in Microsoft Test Manager (MTM) but rather to fill some gaps and enable some of the less common scenarios that still may be important for some testers.

You can get a general overview of TCM.exe over on MSDN in this article.

Some examples of things you can do with TCM.exe include:

Create an automated test run that contains test points across multiple test suites.
Create an automated test run without specifying a Team Build (you can specify the build directory where automated test assemblies are picked up directly).
Publish results from a trx (MSTest / VS run results file) against your Test Plan.
Import VSTT Unit Tests (or even other VSTT test types) from your test assembly into your team project. Creates a Test Case work item from each Unit Test (given some user specified constraints on what to import) and creates the association with the automated unit test for you. You can also sync this set of imported tests with a specified Test Suite.